search

ssbc / ssb-db2

A new database for secure-scuttlebutt
47 stars 8 forks source link

OOM when deleting huge feeds #402

Closed arj03 closed 2 years ago

arj03 commented 2 years ago

My manyverse on desktop crashes after < 1 minute. I finally had some time to debug what is going on and found out that the problem is ssb-friends-purge. This module seems to try and delete around 1250 feeds when it starts up. One of the first feeds it finds is: @/iGURN4KIyJWMUDdbpjaZQHyCL9fjpFWGObGZ8zhzts=.ed25519. This feed has 121863 messages so when it tries to delete the feed, the asOffsets call here never finishes. It simply dies from out of memory. Is there any reason we can't just stream the offsets directly into log.del?

staltz commented 2 years ago

Can you confirm what version of manyverse it is?

Is there any reason we can't just stream the offsets directly into log.del?

The push-stream pipeline is synchronous, so the whole backend gets frozen until the whole feed is deleted.

If you want to play around with this (in production heheh), you can git clone the manyverse repo then:

It'll use the same database as production uses. But you can edit the file in the repo ./desktop/index.js it's one huge noderified js file, and you can modify ssb-db2 code until you find something that works for you. (Just trying to get you unstuck on this issue, but I'm also curious what you learn about the bug) But I can also look at it later.

arj03 commented 2 years ago

Oh I should have provided some more information :) I tracked this down by running manyverse from source, master branch. I was guessing what was the cause since it was hard to get a proper memory dump before the whole thing crashed. Trial and error led me to the root cause, which is this issue :)

I created a PR, can confirm that it fixes the problem for me.

arj03 commented 2 years ago

I see that you hacked this in manyverse with a setTimeout, you should also be able to do that in the PR version.