search

hydradatabase / hydra

Hydra: Column-oriented Postgres. Add scalable analytics to your project in minutes.
https://www.hydra.so
Apache License 2.0
2.83k stars 76 forks source link

[columnar] delete the scanContext whenever possible after creation #226

Closed JerrySievert closed 9 months ago

JerrySievert commented 9 months ago

during logical replication of large tables, multiple MemoryContext's are created via CreateColumnarScanMemoryContext() and do not get removed, eventually resulting in the OOM killer killing off the replication process and forcibly restarting the database.

since a scan context can get used outside of an actual read, it did not make sense to delete it in ColumnarEndRead(), so individual scan contexts are deleted when the function that created them are about to end execution.

this eliminates the ever-growing of scan contexts that appear under TopMemoryContext, and fixes the leak.

JerrySievert commented 9 months ago

I was not able to find a way to create a test to verify the results, verification was done by adding a memory walking debug output (which I removed) that showed the existence of thousands of memory contexts (all 8k) that went away after my fix.