search

wasilibs / go-pgquery

MIT License
28 stars 2 forks source link

Large memory growth in long running processes #29

Open nickzelei opened 4 weeks ago

nickzelei commented 4 weeks ago

Heyo!

First off, great project! We are parsing postgres queries in Go and came across this via sqlc due to not wanting to add a CGO dependency. This for the most part works great, however, it seems to have a large amount of memory growth when running in a long running process.

Below is a heap snapshot from calling it four times.

heap

Not doing anything crazy here, just passing in a sql statement so that we can qualify where conditions with their respective schema and table. Maybe we're doing something wrong, but I am seeing massive memory growth correlated to the amount of calls we make to this library.

What makes it even more difficult is that it doesn't seem to free the memory for 10 minutes. Here's an snapshot of some memory consumption I clocked when running it.

image

You can see how we're calling it here: https://github.com/nucleuscloud/neosync/blob/main/worker/pkg/query-builder2/querybuilder.go#L263

anuraaga commented 4 weeks ago

Hi @nickzelei - thanks for the report and sorry for the issut. While performance isn't a focus for this library (yet) this still looks worth checking. Actually our build is probably showing the same behavior in the benchmark workflow of this repo and I've been putting it off since the common usage with the sqlc binary isn't long living. But time to stop putting it off 😅

anuraaga commented 3 weeks ago

Hi @nickzelei - I have pushed a change that seems to have resolved the memory issues we were seeing in the benchmark workflow here, so I am hopeful it helps with your issue too (basically, sync.Map doesn't interact well when wazero uses mmap for memory allocation as it is off-GC). Would you be able to try it and see if it helps?

nickzelei commented 1 week ago

Hi @anuraaga appreciate the quick response and apologies I haven't had much time to look into this at all. But I did want to atleast ack your message and let you know that I appreciate your quick response. I'll check into it when I am able to.