Having problems trying to use duckdb-async in Next.js

Kennyac commented 1 year ago

When running on next.js I recieve the following error

error ./node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html Module parse failed: Unexpected token (1:0) You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders

<!doctype html> | |

Import trace for requested module: ./node_modules/@mapbox/node-pre-gyp/lib/util/nw-pre-gyp/index.html ./node_modules/@mapbox/node-pre-gyp/lib/ sync ^.\/.*$ ./node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js ./node_modules/duckdb/lib/duckdb-binding.js ./node_modules/duckdb/lib/duckdb.js ./node_modules/duckdb-async/dist/duckdb-async.js ./src/app/api/duckdb/route.ts

I have a simple example project showing the problem https://drive.google.com/file/d/1dceXEXiaH01-SJKxu2ckgoWCTf8f79Rm/view?usp=sharing

unzip form the above link to reproduce yarn install yarn dev browse to http://localhost:3001/api/duckdb

aaronjbecker commented 1 year ago

I tried following the error messages and installing a webpack loader for the HTML dependency and wiring that up in my next.config.js, but that just led to more errors related to some dependencies on mock-aws-s3, aws-sdk, and finally another webpack error related to a C# file: error ./node_modules/node-gyp/lib/Find-VisualStudio.cs Module parse failed: Unexpected token (9:6) You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders | // This script needs to be compatible with PowerShell v2 to run on Windows 2008R2 and Windows 7. | > using System; | using System.Text; | using System.Runtime.InteropServices; Import trace for requested module: ./node_modules/node-gyp/lib/Find-VisualStudio.cs ./node_modules/node-gyp/lib/ sync ^\.\/.*$ ./node_modules/node-gyp/lib/node-gyp.js ./node_modules/@mapbox/node-pre-gyp/lib/util/compile.js ./node_modules/@mapbox/node-pre-gyp/lib/ sync ^\.\/.*$ ./node_modules/@mapbox/node-pre-gyp/lib/node-pre-gyp.js ./node_modules/duckdb/lib/duckdb-binding.js ./node_modules/duckdb/lib/duckdb.js ./node_modules/duckdb-async/dist/duckdb-async.js

At this point I gave up because I don't understand why webpack would need to load a C# file.

All the cascading dependency issues seem related to mapbox integration. Is there a way to isolate that functionality? For the time being I'm just using DuckDB from Python with a private API on the backend...

antonycourtney commented 1 year ago

I have only done small experiments with Next.js, so forgive any lack of understanding, but...what are you trying to do?

duckdb-async is just a thin wrapper around the duckdb node.js bindings (duckdb npm package). The duckdb npm package contains a build of the DuckDb library (C++ code) wrapped with node.js addins.

So the duckdb npm by module (and, by extension, duckdb-async) can only ever run server-side, in a node.js process. I'm not entirely sure what Next.js is trying to do here, but the fact that webpack involves typically suggests bundling for the browser. That's not going to work for duckdb or duckd-async. There is an adjacent project to duckdb, duckdb-wasm, for running DuckDb in a browser using WebAssembly if that's what you are looking for.

Hope that helps.

mattf96s commented 1 year ago

You won't be able to get this working 😅—the original DuckDB node package is the issue. I gave up and instead make a post request to a custom lambda layer as explained in this repo. It's not ideal—there's a wasm version too (if it appropriate for your use case).

See the node issues here:

Kennyac commented 1 year ago

Thanks for your help

My main aim here was use duckdb for a database access layer at the server side

I endded up writing a separate fastify service to handle my requests works but would have been nice for this to be used in directly in next.js

tobilg commented 1 year ago

I maintain https://www.npmjs.com/package/duckdb-lambda-x86 for exactly the reason @mattf96s explained. I uses the correct GLIBC versions for Vercel (which runs on Lambda).

pencilcheck commented 1 month ago

https://github.com/vercel/next.js/discussions/49709

but somehow the poster said it doesn't work for duckdb-async, which is weird if this package only is a thin wrapper around duckdb

pencilcheck commented 1 month ago

Btw, copying my answer from the vercel issue here:

If you are on nextjs 14, try this

  experimental: {
    serverComponentsExternalPackages: ["duckdb", "duckdb-async"],
  },

duckdb-async works with this configuration above on nextjs 14

motherduckdb / duckdb-async

Having problems trying to use duckdb-async in Next.js #6