High memory consumption when loading `routes-large.kite`

assetnote / kiterunner

Contextual Content Discovery Tool

GNU Affero General Public License v3.0

2.59k stars 288 forks source link

High memory consumption when loading `routes-large.kite` #11

Open minight opened 3 years ago

minight commented 3 years ago

running routes-large.kite consumes an inordinate amount of memory, causing the program to start thrashing on machines with limited ram

The route compilation should be optimised to avoid this if possible

minight commented 3 years ago

Profiling taken by curl localhost:6060/debug/pprof/heap > heap.pprof2 upon program startup with ./kr scan -w ~/routes-large.kite -A=apiroutes-210328:20000 -x 20 -j 1 --fail-status-codes 400,401,404,403,501,502,426,411 https://target.com seems to indicate that unmarshalling the kite file is consuming a substantial amount of memory.

I suspect using proto.Unmarshal is less optimal than using the inbuilt proute.ProtoAPI.Unmarshal function generated by gogoproto.

heap.pprof2.zip

minight commented 3 years ago

Later profiling indicates that converting to proto routes and then to http routes is causing 3x the memory usage necessary.

It may warrant rethinking how we load the dataset into memory and whether a non-copy version is possible

heap.pprof7.zip

0xCGonzalo commented 3 years ago

I have the same problem here.

0xCGonzalo commented 3 years ago