search

seddonym / grimp

Builds a graph of a Python project's internal dependencies.
https://grimp.readthedocs.io
BSD 2-Clause "Simplified" License
51 stars 10 forks source link

Speed up import scanning by 400-500% #142

Open duzumaki opened 4 months ago

duzumaki commented 4 months ago

Tested on a large mono repo (35,000 modules) using a mac m1 (8 cores). The modules are divided amongst the cores on the system it runs. so each core is doing an even share of ast walking

profiled using Cprofile, pstats(to read the profile dump) and snakeviz(for the visualisation)

Current(~37s):

Screenshot 2024-02-12 at 00 50 30

after parallelisation(~8s):

Screenshot 2024-02-12 at 00 52 35
seddonym commented 4 months ago

Very exciting!

Have left a couple of initial comments, also the tests are failing which would be good to sort out.