search

denoland / deno

A modern runtime for JavaScript and TypeScript.
https://deno.com
MIT License
93.17k stars 5.16k forks source link

Using DuckDB with Deno #23656

Open nshiab opened 1 month ago

nshiab commented 1 month ago

Version: Deno 1.41.3

Hi!

I tried to use duckdb-node with Deno by running this index.ts.

import duckdb from "npm:duckdb";

const db = new duckdb.Database(":memory:");

But when running deno run -A index.ts, I get this error:

naelshiab test  $ deno run -A index.ts   
error: Uncaught (in promise) Error: Cannot find module '/Users/naelshiab/Library/Caches/deno/npm/registry.npmjs.org/duckdb/0.10.1/lib/binding/duckdb.node'
Require stack:
- /Users/naelshiab/Library/Caches/deno/npm/registry.npmjs.org/duckdb/0.10.1/lib/duckdb-binding.js
- /Users/naelshiab/Library/Caches/deno/npm/registry.npmjs.org/duckdb/0.10.1/lib/duckdb.js
- /Users/naelshiab/Library/Caches/deno/npm/registry.npmjs.org/duckdb/0.10.1/lib/duckdb.js
    at Function.Module._resolveFilename (node:module:609:15)
    at Function.Module._load (node:module:487:27)
    at Module.require (node:module:662:19)
    at require (node:module:776:16)
    at Object.<anonymous> (file:///Users/naelshiab/Library/Caches/deno/npm/registry.npmjs.org/duckdb/0.10.1/lib/duckdb-binding.js:4:15)
    at Object.<anonymous> (file:///Users/naelshiab/Library/Caches/deno/npm/registry.npmjs.org/duckdb/0.10.1/lib/duckdb-binding.js:7:4)
    at Module._compile (node:module:718:34)
    at Object.Module._extensions..js (node:module:732:10)
    at Module.load (node:module:643:32)
    at Function.Module._load (node:module:524:12)

Is it possible to use DuckDB with Deno?

Thank you very much for your great work.

bartlomieju commented 1 month ago

This is because we currently don't support "lifecycle hooks" when installing npm dependencies (eg. postinstall script). We have a PR that implements that that's WIP: https://github.com/denoland/deno/pull/23558.

You can work around this for now if you use "bring your own node modules" - by installing using npm/yarn/pnpm.

nshiab commented 1 month ago

Thank you!

MaximePawlakFr commented 3 weeks ago

Hi @nshiab , have you succeeded?

I tried but I got this error: 

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: macos aarch64
Version: 1.40.4
Args: ["deno", "run", "--allow-all", "data/index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2508:56:
called `Result::unwrap()` on an `Err` value: BadType { actual: "v8::data::Value", expected: "v8::data::Array" }
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

My data/index.ts contains:

import duckdb from "duckdb";
console.log(duckdb)
const db = new duckdb.Database(":memory:");
console.log(db)

The steps are:

============================================================ Deno has panicked. This is a bug in Deno. Please report this at https://github.com/denoland/deno/issues/new. If you can reliably reproduce this panic, include the reproduction steps and re-run with the RUST_BACKTRACE=1 env var set and include the backtrace in your report.

Platform: macos aarch64 Version: 1.40.4 Args: ["deno", "run", "--allow-all", "./data/index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2508:56: called Result::unwrap() on an Err value: BadType { actual: "v8::data::Value", expected: "v8::data::Array" } note: run with RUST_BACKTRACE=1 environment variable to display a backtrace


By adding `RUST_BACKTRACE=1`, I get:

============================================================ Deno has panicked. This is a bug in Deno. Please report this at https://github.com/denoland/deno/issues/new. If you can reliably reproduce this panic, include the reproduction steps and re-run with the RUST_BACKTRACE=1 env var set and include the backtrace in your report.

Platform: macos aarch64 Version: 1.40.4 Args: ["deno", "run", "--allow-all", "./data/index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2508:56: called Result::unwrap() on an Err value: BadType { actual: "v8::data::Value", expected: "v8::data::Array" } stack backtrace: 0: _rust_begin_unwind 1: core::panicking::panic_fmt 2: core::result::unwrap_failed 3: core::result::Result<T,E>::unwrap 4: _napi_set_element 5: ZNK4Napi6Object3SetIA11cEEbjRKT 6: ZN10NodeDuckDBC2EN4Napi3EnvENS0_6ObjectE 7: __ZN4Napi5AddonI10NodeDuckDBE4InitENS_3EnvENS_6ObjectE 8: _napi_register_module_v1 9: deno_napi::op_napi_open::v8_fn_ptr note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.


And by adding `RUST_BACKTRACE=full`, I get:

============================================================ Deno has panicked. This is a bug in Deno. Please report this at https://github.com/denoland/deno/issues/new. If you can reliably reproduce this panic, include the reproduction steps and re-run with the RUST_BACKTRACE=1 env var set and include the backtrace in your report.

Platform: macos aarch64 Version: 1.40.4 Args: ["deno", "run", "--allow-all", "./data/index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2508:56: called Result::unwrap() on an Err value: BadType { actual: "v8::data::Value", expected: "v8::data::Array" } stack backtrace: 0: 0x1018f46b0 - ::fmt::hade97c44b56fc870 1: 0x100bcd6b0 - core::fmt::write::h81cbefbffc581dab 2: 0x101901918 - std::io::Write::write_fmt::h125c60058ebfe43c 3: 0x1018f4508 - std::sys_common::backtrace::print::hfa54be0dd0cf5860 4: 0x10190838c - std::panicking::default_hook::{{closure}}::h4235e0929057f079 5: 0x1019080d0 - std::panicking::default_hook::hcf67171e7c25be94 6: 0x100b05c18 - deno::setup_panic_hook::{{closure}}::h71a800dcbf73da30 7: 0x101908a2c - std::panicking::rust_panic_with_hook::h1767d40d669aa9fe 8: 0x1018f49e8 - std::panicking::begin_panic_handler::{{closure}}::h83ff281d56dc913c 9: 0x1018f4954 - std::sys_common::backtrace::rust_end_short_backtrace::h2f399e8aa761a4f1 10: 0x1019086b4 - _rust_begin_unwind 11: 0x100bd78cc - core::panicking::panic_fmt::hc32404f2b732859f 12: 0x100bd16f0 - core::result::unwrap_failed::h2ea3b6e22f1f6a7c 13: 0x1007888a0 - core::result::Result<T,E>::unwrap::h2d955a3171e3fa0a 14: 0x10078f060 - _napi_set_element 15: 0x11a7719c0 - ZNK4Napi6Object3SetIA11cEEbjRKT 16: 0x11a7710a4 - ZN10NodeDuckDBC2EN4Napi3EnvENS0_6ObjectE 17: 0x11a7728a0 - ZN4Napi5AddonI10NodeDuckDBE4InitENS_3EnvENS_6ObjectE 18: 0x11a771f38 - _napi_register_module_v1 19: 0x101408960 - deno_napi::op_napi_open::v8_fn_ptr::h315a352275b5cff8



Have you managed to use duckdb in Deno? 
What could I do?

Best
MaximePawlakFr commented 3 weeks ago

--Update--

I've upgraded to deno 1.43.1 (was 1.40.4), and now I get this error:

error: Uncaught (in promise) Error: Error in native callback
    at Object.Module._extensions..node (node:module:704:20)
    at Module.load (node:module:597:32)
    at Function.Module._load (node:module:484:12)
    at Module.require (node:module:609:19)
    at require (node:module:715:16)
    at Object.<anonymous> (file:MYPATH/node_modules/.deno/duckdb@0.10.2/node_modules/duckdb/lib/duckdb-binding.js:4:15)
    at Object.<anonymous> (file:MYPATH/node_modules/.deno/duckdb@0.10.2/node_modules/duckdb/lib/duckdb-binding.js:7:4)
    at Module._compile (node:module:659:34)
    at Object.Module._extensions..js (node:module:673:10)
    at Module.load (node:module:597:32)

Do you have any idea?

nshiab commented 3 weeks ago

Hi @MaximePawlakFr!

For now, I am using node/npm. But @bartlomieju is working on it https://github.com/denoland/deno/pull/23558.

bartlomieju commented 3 weeks ago

It's an internal bug in Deno, but it seems easy enough to fix it to make example from DuckDB's page work. I'll try to have it done tonight.

bartlomieju commented 3 weeks ago

I merged a fix that makes DuckDB work. It should be available in canary in ~2h - you can then use deno upgrade --canary to try it out.

MaximePawlakFr commented 3 weeks ago

Thank you Bartek.

I've just updated deno to deno 1.43.1+cbb78e1.

I tried again. I can load duckdb without the previous error.

But I can not run any request to duckdb. I get a segmentation fault.

Here is my code: 


import duckdb from "duckdb";
const db = new duckdb.Database(":memory:"); 
console.log(db);

Deno.serve(async (req) => {

    const request =
      "SELECT * FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parquet' LIMIT 1000;";
    console.log(request);

    const result = await new Promise((resolve, reject) => {
      db.all(request, function (err, res) {
        console.log(res);
        if (err) {
          console.warn(err);
          reject(err);
        }
        resolve(res);
      });
    });

    const res = JSON.stringify(
      result,
      (key, value) => typeof value === "bigint" ? value.toString() : value, // return everything else unchanged
    );

    return new Response(
      res,
      {
        headers: { "Content-Type": "application/json" },
      },
    );
});

I run deno run --allow-all index.ts

I see in my terminal:

Database {}
Listening on http://localhost:8000/
SELECT * FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parquet' LIMIT 1000;
zsh: segmentation fault  deno run --allow-all index.ts

It looks like something went wrong with duckdb. Any idea what?

Regards

bartlomieju commented 3 weeks ago

So far I'm not able to reproduce this segmentation fault on my end - I get results as expected. Can you update once again to latest canary and try again?

nshiab commented 3 weeks ago

I merged a fix that makes DuckDB work. It should be available in canary in ~2h - you can then use deno upgrade --canary to try it out.

Unfortunately, I am on a M1 Mac, so I can't install canary versions, from what I gathered.

bartlomieju commented 3 weeks ago

@nshiab you can, we've been shipping canary version for ARM Macs for a couple months now :)

nshiab commented 3 weeks ago

@bartlomieju What am I doing wrong? 😢

Screenshot 2024-05-07 at 9 04 47 AM

Thank you so much for your help!

bartlomieju commented 3 weeks ago

It appears that deno binary you have locally on your system was installed via homebrew. You should remove it using homebrew and then install it again using the script, then add it to you PATH.

nshiab commented 3 weeks ago

You are right. Stupid mistake on my end. Sorry.

Unfortunately, it doesn't seem to work on my laptop. Should I do something differently?

Screenshot 2024-05-07 at 10 32 33 AM
bartlomieju commented 3 weeks ago

@nshiab the problem with postinstall scripts is not solved - you still need to use npm install and use DENO_FUTURE=1 env var, I fixed the problem related to actual execution of DuckDB.

nshiab commented 3 weeks ago

Hmmm. I tried again but I am still doing something wrong I guess. It's probably best if I wait for postinstall scripts to work.

Screenshot 2024-05-07 at 4 00 46 PM
bartlomieju commented 3 weeks ago

@nshiab you need to specify DENO_FUTURE env var in your terminal when you run the deno command:

# bash 
DENO_FUTURE=1 deno run -A index.ts
# fish
env DENO_DUTURE=1 deno run -A index.ts
nshiab commented 3 weeks ago

IT WORKS! Amazing! Thanks for your patience @bartlomieju!

But... if I actually run a query, it fails. This is an example from the duckdb repo.

Screenshot 2024-05-07 at 4 33 24 PM
bartlomieju commented 3 weeks ago

Thanks! I don't get a segfault, but I'm getting a panic like so:

import duckdb from "duckdb";
const db = new duckdb.Database(":memory:"); 

const con = db.connect();

con.all("SELECT 42 AS fortytwo", function (err, res) {
  if (err) {
    console.warn(err);
  }
  console.log(res[0].fortytwo);
})

env DENO_FUTURE=1 deno run -A index.ts

============================================================
Deno has panicked. This is a bug in Deno. Please report this
at https://github.com/denoland/deno/issues/new.
If you can reliably reproduce this panic, include the
reproduction steps and re-run with the RUST_BACKTRACE=1 env
var set and include the backtrace in your report.

Platform: macos aarch64
Version: 1.43.1+998036b
Args: ["deno", "run", "-A", "index.ts"]

thread 'main' panicked at cli/napi/js_native_api.rs:2348:63:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I'll try to debug this one as well.

MaximePawlakFr commented 3 weeks ago

So far I'm not able to reproduce this segmentation fault on my end - I get results as expected. Can you update once again to latest canary and try again?

I've just update to the last canary: deno 1.43.1+998036b I still get the same segmentation fault.

Keep me in the loop if you make any advance.

Thanks a lot for what you tried.