search

pganalyze / pg_query_go

Go library to parse and normalize SQL queries using the PostgreSQL query parser
BSD 3-Clause "New" or "Revised" License
663 stars 80 forks source link

Wasm-based codepath to run without cgo #103

Open anuraaga opened 11 months ago

anuraaga commented 11 months ago

I have developed a version of pgquery for Go that uses WebAssembly and wazero

https://github.com/wasilibs/go-pgquery

This means that it enable the library to be used without cgo, notably meaning no warnings when building after a MacOS upgrade and full windows support. I have been able to use it to remove platform constraints from sqlc

https://github.com/sqlc-dev/sqlc/pull/3027

I wanted to check if there is any interest in upstreaming that technique into this repository, in which case I would be happy to archive that project. WebAssembly is quite gnarly, so I don't know if I particularly recommend it and run into possible maintenance issues, though I'd be happy to help when needed. More practical may be to keep that separate and make sure the types are compatible to allow easy switching. I have verified that with #102 I can switch go-pgquery to return the proto from this package in its public API without issue.

I wanted to share the idea and provide context on recent changes and happy to hear any thoughts on what could work well for this project.

Thank you.

lfittl commented 11 months ago

I have developed a version of pgquery for Go that uses WebAssembly and wazero

https://github.com/wasilibs/go-pgquery

This means that it enable the library to be used without cgo, notably meaning no warnings when building after a MacOS upgrade and full windows support. I have been able to use it to remove platform constraints from sqlc

sqlc-dev/sqlc#3027

I wanted to check if there is any interest in upstreaming that technique into this repository, in which case I would be happy to archive that project. WebAssembly is quite gnarly, so I don't know if I particularly recommend it and run into possible maintenance issues, though I'd be happy to help when needed.

Thanks for sharing!

I think its potentially interesting to offer this upstream as part of libpg_query, but we'd need to understand better what the right runtime would be across different languages (ideally this could also be used to solve e.g. Node.js porting issues, as described in https://github.com/pganalyze/libpg_query/issues/44).

However we are also revisiting whether we can support Windows directly in libpg_query, which then may not make the effort of maintaining a wasm port worthwhile.

All that said, I'd suggest we keep this separate in the short term (and hopefully with #102 merged it should work better I assume), and medium term (i.e. some time next year) revisit and see which direction makes sense.

anuraaga commented 11 months ago

Thanks @lfittl - yeah, the runtime aspect is definitely tricky, I found postgres requires threads and exceptions to compile, neither of which supported by officially released Wasm yet so runtime support is poor.

Agree that revisiting later after getting more clarity on those other options sounds great. The options to look at then would probably be

1) Maintain a wasm that can be shared among languages 2) Only use wasm for pg_query_go 3) Don't use wasm

The reason I separate out 2) is that my impression is that out of all languages, Go tends to have the most issues and least appetite toward C bindings so it could be worth supporting CGO_ENABLED=0 apps which are common. Just something to keep in mind, looking at the PR links to this issue, my library got used outside of sqlc much sooner than I was expecting 😂

karlhorky commented 11 months ago

cc @gregnr, since you have been working on a WebAssembly version of node-libpg-query over here:

Wonder if the WebAssembly efforts can be combined to have Windows support before full libpg_query native Windows support happens...