We run perf in continuous mode, where it samples non-stop and aggregates samples in a single file. Every interval, gprofiler signals perf, to "switch output" to a new file, while gprofiler is consuming the old one.
This becomes a problem if gProfiler gets stuck, for example. We don't want perf to run wild. For this reason we need a timeout placed on perf, such that it disables itself / cleans its output files to refrain from writing more and more data.
We have similar protections for other components (for example, for async-profiler it was added here: https://github.com/Granulate/gprofiler/pull/265).
The solution I chose is to force perf to write output files every set interval, even if it is not signaled by gprofiler. perf will write the files and rotate a single copy. I chose this solution because it was the simplest one to apply on perf's code; ideally we'd like perf to "pause" until it gets signaled again, but I couldn't think of a simple implementation for that. It is not possible to simply "pause". See the perf change here: https://github.com/Granulate/linux/commit/aa689e9b55c7b5ba8d399bc2560d36ef98436150
I ran gProfiler from master, sent SIGSTOP to the main Python process. The perf files in /tmp/gprofiler_tmp/tmpXXX/perf.* continue to grow forever. Once SIGCONT is sent, gProfiler recovers and continues. The next file being read has all the "missed" samples.
I ran gProfiler from this branch, sent SIGSTOP to the main Python process. The perf files in /tmp/gprofiler_tmp/tmpXXX/perf.* no longer grow forever (they are replaced every timeout). Once SIGCONT is sent, gProfiler recovers and continues. The next file being read is small (does not contain all missed samples).
We run perf in continuous mode, where it samples non-stop and aggregates samples in a single file. Every interval, gprofiler signals perf, to "switch output" to a new file, while gprofiler is consuming the old one.
This becomes a problem if gProfiler gets stuck, for example. We don't want perf to run wild. For this reason we need a timeout placed on perf, such that it disables itself / cleans its output files to refrain from writing more and more data. We have similar protections for other components (for example, for async-profiler it was added here: https://github.com/Granulate/gprofiler/pull/265).
The solution I chose is to force perf to write output files every set interval, even if it is not signaled by gprofiler. perf will write the files and rotate a single copy. I chose this solution because it was the simplest one to apply on perf's code; ideally we'd like perf to "pause" until it gets signaled again, but I couldn't think of a simple implementation for that. It is not possible to simply "pause". See the perf change here: https://github.com/Granulate/linux/commit/aa689e9b55c7b5ba8d399bc2560d36ef98436150
Closes: https://github.com/Granulate/gprofiler/issues/782
Testing done
/tmp/gprofiler_tmp/tmpXXX/perf.*continue to grow forever. Once SIGCONT is sent, gProfiler recovers and continues. The next file being read has all the "missed" samples./tmp/gprofiler_tmp/tmpXXX/perf.*no longer grow forever (they are replaced every timeout). Once SIGCONT is sent, gProfiler recovers and continues. The next file being read is small (does not contain all missed samples).