rvagg
commented
1 year ago yeah, the pruner is a beast, I've looked at those loops a few times and sweated at the frequency and size
so the theory about memory usage is simply that graphsync has a massive queue backed up to put into the store
if true then this might also help explain why I able to observe relatively few bitswap fulfilments with https://github.com/application-research/autoretrieve/pull/157 - in fact, iirc the fulfiments are early on in an autoretrieve's lifecycle and then they just stop happening at all
What
So in addition to memory, our CPU is pegged, and, moreover, given the way that GraphSync works, this is also likely the source of both SP bandwidth problems and memory usage. (if block putting and processing is very slow, graphsync will accumulate them in memory)
Profiling names a clear hotspot: the Random Pruner and specifically the inner loop in clearPins, which if I'm reading right runs on every block put.
Let's try to improve this to solve MANY problems.