We are currently integrating the Pyroscope Java agent into our infrastructure, and it works great. However, we have encountered a problem: In cases of a large fleet of pods (1000+) and aggressive sampling, we generate too much profiling data for the server to handle. Additionally, the 2-3% profiling overhead becomes a significant resource drain.
I propose adding an option (both in the SDK and as an environment variable) named PYROSCOPE_CLUSTER_RATIO with a value in percentage (0-100). This feature would enable the agent only if a randomly generated value in an inclusive range 0-100 is less than the provided percentage.
This feature will allow users to save resources on both the server and client sides while still maintaining statistically viable continuous profiling data for large deployments.
I would be glad to help implement this feature if you consider it appropriate.
We are currently integrating the Pyroscope Java agent into our infrastructure, and it works great. However, we have encountered a problem: In cases of a large fleet of pods (1000+) and aggressive sampling, we generate too much profiling data for the server to handle. Additionally, the 2-3% profiling overhead becomes a significant resource drain.
I propose adding an option (both in the SDK and as an environment variable) named PYROSCOPE_CLUSTER_RATIO with a value in percentage (0-100). This feature would enable the agent only if a randomly generated value in an inclusive range 0-100 is less than the provided percentage.
This feature will allow users to save resources on both the server and client sides while still maintaining statistically viable continuous profiling data for large deployments.
I would be glad to help implement this feature if you consider it appropriate.