search

duckdb / duckdb-node

MIT License
56 stars 27 forks source link

Request for Prebuilt Packages for duckdb-node #70

Open azmy60 opened 7 months ago

azmy60 commented 7 months ago

Hi,

First of all, thanks for the efforts in maintaining duckdb-node! We've noticed the yarn install/add process is quite time-consuming. I'm running on WSL, AMD Ryzen 5 5600H, laptop and the time it takes to run yarn after adding duckdb-node is > 20 minutes, and the number grows even more each time I run yarn add. This also happens during test automation on Github Actions.

Could prebuilt packages be made available to cover this?

Thanks!

rathboma commented 7 months ago

For context, Azmi and I work on Beekeeper Studio - github.com/beekeeper-studio/beekeeper-studio

The WASM build is great for front end, but because of electron we can use the 'full' duckdb.

Hoping something similar can be provided to how better-sqlite3 does their pre builds?

No idea how hard this is.

Love DuckDB, would be thrilled to add it to BKS ❤️

maxzhuo commented 7 months ago

Execute commands in the following order:

D:\app> npm i duckdb

D:\app> npm i @electron-forge/cli

D:\app> npx electron-forge import

D:\app> npm i electron-rebuild

D:\app> electron-rebuild -f -w duckdb

then... some bug errors, is my method incorrect? I am using the Win11 system.

carlopi commented 7 months ago

We distribute precompiled packages for a bunch of platform/arch/version combinations.

For details, this is the CI job that generated the duckdb-node npm package: https://github.com/duckdb/duckdb-node/actions/runs/8388373879.

Can you share the result of:

$ node
> process.arch
'arm64'
> process.platform
'darwin'
> process.version
'v21.7.1'

or directly look up whether your platform is currently being built?

rathboma commented 7 months ago

Ah, yes it's because we use Electron. Is there any chance you could add Electron to your prebuilds, in a similar way to how better-sqlite3 does it?

https://github.com/WiseLibs/better-sqlite3/blob/master/.github/workflows/build.yml#L17

Once it's added once we can help maintain it when new electron versions come out.

rathboma commented 7 months ago

@carlopi if we make a PR that adds electron pre builds would that be something you'd consider merging?

RyanChapman2x commented 7 months ago

@rathboma: perhaps try using electron-builder over electron-forge.

I was having trouble getting electron-forge to package my project with duckdb-node in any kind of reasonable timeframe (I waited for hours and never saw any progress). I searched around and found that a lot of people were having trouble with electron-forge for some packages like duckdb-node, so I made the switch. My build times now clock in at about 1-2 minutes.

carlopi commented 7 months ago

@carlopi if we make a PR that adds electron pre builds would that be something you'd consider merging?

It would be welcome a blueprint on how to do so / what changes are required.

rathboma commented 7 months ago

@RyanChapman2x we use Electron builder already, but the build times are very slow still. It's because there are no precompiled packages. Electron 18 is node 16 basically, so it shouldn't be too hard to figure out how to add this to the existing pipelines.

rathboma commented 4 months ago

@carlopi We've made a PR for this. Let us know what you think. Right now it's just for a couple of Electron versions, but it could be easily extended to automatically build for all published electron versions.