brendangregg / perf-tools

Performance analysis tools based on Linux perf_events (aka perf) and ftrace
GNU General Public License v2.0
9.9k stars 1.64k forks source link

Will CONFIG_FUNCTION_PROFILER=y cause poor performance? #77

Closed AirbornWdd closed 4 years ago

AirbornWdd commented 6 years ago

Hi Brendan Gregg:

AirbornWdd commented 6 years ago

Debian OS has CONFIG_FUNCTION_PROFILER not set as default configuration. I am not sure setting CONFIG_FUNCTION_PROFILER=y will cause poor performance or not. Especially in the production environment.

brendangregg commented 4 years ago

We have CONFIG_FUNCTION_PROFILER=y set everywhere at Netflix in production. My understanding is that the overhead is very low; see Steven's recent talk for an explanation of how Ftrace lowers overhead[0]. But I haven't benchmarked it. You can do with and without kernel builds and then a stress test, e.g., perf bench futex, to see the synthetic-worst-case difference. You'd then need to map that back to realistic worst case. E.g., you could look at the %sys time during the stress test vs production workloads.

[0] http://www.brendangregg.com/blog/2019-10-15/kernelrecipes-kernel-ftrace-internals.html

AirbornWdd commented 4 years ago

We have CONFIG_FUNCTION_PROFILER=y set everywhere at Netflix in production. My understanding is that the overhead is very low; see Steven's recent talk for an explanation of how Ftrace lowers overhead[0]. But I haven't benchmarked it. You can do with and without kernel builds and then a stress test, e.g., perf bench futex, to see the synthetic-worst-case difference. You'd then need to map that back to realistic worst case. E.g., you could look at the %sys time during the stress test vs production workloads.

[0] http://www.brendangregg.com/blog/2019-10-15/kernelrecipes-kernel-ftrace-internals.html

Thanks very much. I will do that test later.