search

facebook / buck2

Build system, successor to Buck
https://buck2.build/
Apache License 2.0
3.39k stars 201 forks source link

Profiling dynamic analysis #707

Open aherrmann opened 2 days ago

aherrmann commented 2 days ago

Buck2 supports profiling of Starlark as documented under Observability and Optimization. In particular, it supports profiling of analysis via buck2 profile analysis. However, this feature only captures the static analysis. Any Starlark code that is executed as part of dynamic output actions is not included in this profile.

I would like to profile Starlark code that is executed as part of dynamic output actions, i.e. the implementation of f in ctx.actions.dynamic_output, or the equivalent under the upcoming new API. The motivation is that Buck2 reports a noticeable amount of time spent in "dynamic analysis" on the TUI for some of our targets, and I would like to better understand where this time is spent and optimize the relevant code. (Anecdotally, we had an instance where dynamic analysis took 20s. Luckily we were able to reduce it to 2s simply by refactoring, but we would like to continue optimizing it based on actual data.)

Alternative profiling methods, in particular native profiling did not yield meaningful information about the Starlark code that we need to optimize.

aherrmann commented 2 days ago

To my understand the problem is that buck2 profile analysis is implemented as a separate command to the Buck2 server and does not proceed into the build, which is required to execute dynamic analysis.

A possible approach that was discussed on Discord is to add a profiling mode to the regular build command that can produce an analysis profile, including dynamic analysis, during a regular build.

stepancheg commented 2 days ago

@aherrmann right. Internally we agreed to go with this approach — add --profile-mode= to many commands and output profile of whatever is evaluated. So far I have implemented this flag for cquery command (which is useful to do recursive "loading" profiler), but it is not committed yet for cquery command https://www.internalfb.com/diff/D59088660.

aherrmann commented 1 day ago

@stepancheg Thanks for the update that's good to hear. Thanks for implementing the feature for cquery. Unfortunately, I don't have access to the internal diff. Is it also visible on GitHub somewhere?