search

ueberdosis / hocuspocus

The CRDT Yjs WebSocket backend for conflict-free real-time collaboration in your app.
https://tiptap.dev/docs/hocuspocus/introduction
MIT License
1.34k stars 128 forks source link

extension-redis memory leak with frequent doc updates #854

Closed entropi closed 1 month ago

entropi commented 2 months ago

Description When performing a large number of concurrent DirectConnection updates to a handful of documents, a rapidly escalating memory leak was observed. Waiting and forcing garbage collection had no impact on memory growth.

This appears to be separate from https://github.com/ueberdosis/hocuspocus/issues/846 as the behavior was observed even when executing rapid updates on only a small number of distinct documents, so it doesn't just appear to be due to not unloading docs.

Steps to reproduce the bug Steps to reproduce the behavior:

Perform many (hundreds to thousands) of updates (in my case via DirectConnection) to a finite set of documents in a very small window. Observe rapid memory growth, which calls to GC do not clear.

Expected behavior Memory stays at a reasonable level. setTimeout delays to account for sync remain at one per documentName.

Environment?

Additional context Lowering the disconnectDelay to 1ms removed the leak (though obviously at the cost of the benefit of the delay for syncs), and raising it to 5000ms during rapid updates escalated the memory growth curve. I suspect that it's related to the fact that we're potentially stacking hundreds or thousands of setTImeout operations on the same document in afterStoreDocument within the disconnectDelay window during rapid edits.

~A forthcoming~ The Linked PR appears to mitigate the issue by clearing any pending timeout operation on a document when a new afterStoreDocument or onDisconnect hook fires for the same documentName, instead of always adding new ones on top.

janthurau commented 1 month ago

Thanks for your work! I've released this update as 2.13.7 :-)