search

exyi / pg2parquet

Export PostgreSQL table or query into Parquet file
Apache License 2.0
57 stars 11 forks source link

Publish as WASM package? #9

Closed mhkeller closed 1 year ago

mhkeller commented 1 year ago

I have an Electron app this would be really useful in. Would it be possible to create a web assembly version of this package?

exyi commented 1 year ago

I haven't tried, but I think it can't work from the browser since the we need to use native TCP to connect to postgres.

mhkeller commented 1 year ago

That makes sense. I wonder if it changes anything if the target is a node.js environment and it could reference that as an external dependency of some kind...

exyi commented 1 year ago

Well, then you can simply launch the native binary as a subprocess. I'm not familiar with electron, but I suppose you have separate binaries for each platform - then you just have to make sure to bundle the correct build of pg2parquet.

mhkeller commented 1 year ago

That's my current setup – I was thinking there might be some cost of a few hundred milliseconds or something if I'm launching a subprocess for every query versus if it were loaded into Node. It does work for now, though, so can figure out optimizations down the road if it seems slow. Thanks for the helpful package!

exyi commented 1 year ago

I hope the process starts up faster than 100ms 😅.

If it is a problem, we can publish pg2parquet as a native library which you could use from a Node native module, which runs in-process.

mhkeller commented 1 year ago

Thanks! I'll do some testing once I get everything all set up properly.