Luminarys
commented
4 years ago Do you know if it OOM'd before this happened? I'll do some thinking about the startup process to see if this can be avoided.
Closed kpcyrd closed 4 years ago
Luminarys
commented
4 years ago Do you know if it OOM'd before this happened? I'll do some thinking about the startup process to see if this can be avoided.
Luminarys
commented
4 years ago Also, how large is your session folder(~/.local/share/synapse by default)
kpcyrd
commented
4 years ago As far as I know it never happened before, my session folder is 184M. I'm wondering if an option that prevents torrents from starting might help to bring synapse back up, so I can then start torrents one by one. Also let me know if you have a patch with addition logging that you want me to try.
Luminarys
commented
4 years ago First thing to figure out is if the OOM is caused by startup serialization, can you try commenting out https://github.com/Luminarys/synapse/blob/master/src/torrent/mod.rs#L1396 and seeing if OOM occurs?
Also I'm unsure if it'll actually help but you can enable even more verbose logging if you tweak https://github.com/Luminarys/synapse/blob/master/src/args.rs#L37 to LogLevel::Trace
kpcyrd
commented
4 years ago Commenting the line seems to fix the issue. I also remember that I'm not sure if I updated the binary without restarting the daemon, in that case it's not necessarily related to the out-of-disk.
btw, it seems synapse is working correctly without that line, are there any implications for my state folder? I made a copy and I'm not sure if I should revert afterwards or if this is introducing issues with the files downloading.
Luminarys
commented
4 years ago The only implication of the change is that if synapse crashes immediately after adding a new torrent it won't persist the state, so I think it's relatively safe to use for now.
How much RAM does synapse use while operating normally? While it makes sense that those serialize calls consume some RAM I feel that they shouldn't take more than the size of your session directory ever. I'll have to look more carefully into how that could happen.
Luminarys
commented
4 years ago I've updated this in master. I'm still not fully sure why it occurs though, since if your session folder is only 180~ MBs it shouldn't create allocations more than that. If possible maybe try running something like heaptrack and reporting the results? I think that would help a lot with debugging this. I'll also try this myself on a large session to see if there's any notable results.
kpcyrd
commented
4 years ago It seems the new commits fixed the issue, thanks!
I ran out of disk, freed up some space but have trouble starting synapse since then.
Synapse starts allocating memory in steps of ~100MB/s, starts to swap after 16GB and eventually OOMs.
I'm aware that this is a very vague description. I've tried to pause all torrents to recover but with no success:
I tried to run it with
-dand got a lot of text, I've attached the start and the last second (not including more due to the volume). There didn't seem any interesting in the rest of the log, let me know if you need more.Let me know if there's something I can try to troubleshoot this further.