Performance regression between 2.4.4 and 3.0.20

[guoxe]

Describe the bug Generating compile_commands.json is slower when using version 3 of Bear compared to version 2. On my machine (i7 8700K, 64gb ram and nvme ssd), compiling curl is about 50% slower with version 3.0.20 compared to 2.4.4.

To Reproduce Download the attached Makefile and Dockerfile somewhere and run make test-2.4.4 and make test-3.0.20 and note the times. make and a docker version that supports buildkit is required. I used GNU Make 4.3 and Docker 20.10.18. The included files read proxy settings from the environment, delete the relevant lines if they are not required. Had to upload them as .txt files so they need to have that suffix removed before use.

The times are measured using the time command, more specifically: time bear make or time bear -- make. The ./configure step is not included in the measurement.

Expected behavior Expecting similar overhead for bear between versions.

Environment:

• OS name: Linux
• OS version: Debian GNU/Linux 11 (bullseye)
• OS architecture: x86_64
• Bear version: 3.0.20
• Bear install method: built from source

Additional context

• Can you give us a reference to the project that you are running against this tool? https://github.com/curl/curl . Originally discovered this issue when compiling an internal project, in that case the build times went from around 2 minutes to 17 minutes when upgrading from 2.4.4 to 3.0.8
• What build tools this project is using? GNU make with configure script for curl.
• What architecture you are trying to compile for? x86_64
• Could you attach build logs? N/A

Dockerfile.txt Makefile.txt

[rizsotto]

Thanks @guoxe , it's a known slowdown between the 2.4 vs 3.0 versions. (Probably the 3.0.8 is the worst from performance perspective.)

In the 3.x versions:

• The processes are spawing a new supervisor process for each exec calls.
• Plus the supervisor processes are doing gRPC talk to the root proces.
• Plus the final report generation is not done parallel.

These are all contributing the slowness, which can be extremely high as you've been measured.

The main driver for this was to implement new features (like record the compiler exit codes, or support environments where the dynamic loader is not working, etc...) That implied the supervisor processes... The gRPC for IPC was probably a bad choice... The sequential processing of the event file is probably the easiest thing to fix.