Regression processing medium to large sized project

rizsotto / Bear

Bear is a tool that generates a compilation database for clang tooling.

GNU General Public License v3.0

4.97k stars 321 forks source link

Regression processing medium to large sized project #350

Closed rizsotto closed 3 years ago

rizsotto commented 3 years ago

Describe the bug Bear 3.x is slower and consume more memory on large projects.

To Reproduce LLVM, linux kernel, wine are examples for large code bases.

Expected behavior Similar or better performance metrics as Bear 2.x

execution time overhead shall be relative small
memory consumption shall be relative small

Environment:

OS name: n/a
OS version: n/a
OS architecture: n/a
Bear version: 3.x
Bear install method: n/a

phush0 commented 3 years ago

for me bear 3.0.5 is significantly faster than bear 3.0.7. Last one is using only part of CPU, as 3.0.5 is using all cores and it is almost same as real compile without cache

edit: I am using it to generate compile database for ESP32 project, so if you need some help debugging or logs just call

rizsotto commented 3 years ago

Thanks @phush0 for the feedback. The 3.0.7 and 3.0.8 are writing the execution reports into SQLite synchronously, which blocks the build to proceed. Before all event was stored in memory, but cause problem for big projects.

I've tried to make the report writing function to be async, but failed on thread syncronization. Will give another try next week.
I'm planing to eliminate the SQLite file writing (and calculate the final entries directly from the execution reports). That should fix the performance problem I think.

rizsotto commented 3 years ago

Release 3.0.10 has fix for the citnames (now the execution time is linear to the input, before it was polynomial). Run against the linux kernel compilation, it is still allocates more memory than it is desired. But valgrind does not report leaks. (Will investigate it further.)

==2300522== LEAK SUMMARY:
==2300522==    definitely lost: 0 bytes in 0 blocks
==2300522==    indirectly lost: 0 bytes in 0 blocks
==2300522==      possibly lost: 0 bytes in 0 blocks
==2300522==    still reachable: 143,671 bytes in 2,179 blocks
==2300522==         suppressed: 0 bytes in 0 blocks
==2300522== 
==2300522== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

rizsotto commented 3 years ago

Release 3.0.11 has fix for intercept memory allocation problem. Now it is consuming a modest 20MB heap during the Linux kernel compilation. (Before and after graphs)

intercept linux 3 10 intercept linux noasync

smhc commented 3 years ago

Thank you for all your efforts on this @rizsotto , those stats look much better. I will try and give it a test run soon and provide any feedback if I notice any problems. Cheers!