Half-Shot commented 1 year ago

We've noticed that the RSS reader logic now has a memory leak, and will eat up memory upon every poll. We've got suspicions that it might be due to the Rust native code leaving some things behind, but haven't conclusively proven it yet.

From heap analysis, Node thinks it's using a consistent 100-200MB however the residential memory increases dramatically. We've tried to isolate the rust code and benchmark it with generated and real RSS feed data from production, but nothing stood out. We're continuing to investigate.

More facts:

This got worse with the 4.3.0 release. The concurrency default is 4, which might have caused some issues.
786 made this worse as the heap now seems to be consumed as well as RSS, which probably means we have both problems.

Half-Shot commented 1 year ago

Reducing the concurrency to 1 on the performance branch didn't achieve much in the end.

Half-Shot commented 1 year ago

786 has fixed this by having the feeds themselves polled in Rust too, so we no longer share the whole XML buffer over the Node<->Rust layer. This seems to have in practice stopped the bleed of memory.

I'll tidy up that PR.

matrix-org / matrix-hookshot

Feed reader has unaccounted memory leak #789

786 made this worse as the heap now seems to be consumed as well as RSS, which probably means we have both problems.

786 has fixed this by having the feeds themselves polled in Rust too, so we no longer share the whole XML buffer over the Node<->Rust layer. This seems to have in practice stopped the bleed of memory.