search

wasilibs / go-pgquery

MIT License
32 stars 3 forks source link

Out of memory error in low memory environment. #31

Open fukasawah opened 1 week ago

fukasawah commented 1 week ago

We trying to use go-pgquery for a PostgreSQL implementation of sqldef. Thanks for a great library!

Problem

As I was working on it, there seemed to be a problem in environments where 4GB is not available in virtual memory.

The way I see it, wazero requires 4GB of virtual memory by default, which seems to be a problem when there is not enough memory and swap space. The error occurs when there is actually 2GB of memory and no swap space.

So I prepared a go-pgquery with WithMemoryLimitPages, and I also confirmed that this problem can be avoided.

https://github.com/fukasawah/go-pgquery/commit/9a5c9d44cb9573b1bb39934ca2791a1026876e23

Perhaps sqlc has the same problem. https://github.com/sqlc-dev/sqlc/blob/v1.27.0/internal/ext/wasm/wasm.go#L150

I also suspect that the large amount of memory consumption in the following issue #29 may be related to this issue.

Suggestion.

We believe that by specifying WithMemoryLimitPages in the WASM runtime (wazero) or making it optional, the library will be able to run in more environments.

For sqldef purposes, we think 4096 pages is sufficient, but it may depend on the use case.

anuraaga commented 1 week ago

Hi @fukasawah - sorry for the issue and the long debugging. We have a custom allocator to avoid the issue with wazero you found, but I had a very silly bug causing it to not be used :( #32. I think that should fix your issue without explicitly limiting pages, would you be able to try it?

fukasawah commented 1 week ago

Thank you.

I followed the steps below to use the latest release, and I observed improvements in my environment!!

$ go get github.com/wasilibs/go-pgquery@latest
go: upgraded github.com/google/go-cmp v0.5.9 => v0.6.0
go: upgraded github.com/wasilibs/go-pgquery v0.0.0-20240826014338-9ea9e19d01fd => v0.0.0-20241011013927-817756c5aae4

I still don’t fully understand why this fix worked..., but I’ll try to get it integrated into sqldef. I’ll update you once it’s successfully applied.

anuraaga commented 1 week ago

Thanks @fukasawah - in case you're interested, the custom allocator requests pages as they are needed as opposed to wazero's default behavior of requesting them all during module initialization, so it works better on low RAM environments.

fukasawah commented 1 week ago

That's indeed a more ideal behavior! If that's the case, I can understand how it would resolve the issue. Thank you very much!