Better docs for examples/hello-ext

[theelderbeever]

I am trying to create an extension in Rust but I am not sure how to get started with the example mentioned above. How do you build it or load it into duckdb at runtime?

I got the example to build with cargo build --example hello-ext --features="vtab bundled,vtab-loadable" but how do I load it? I found ~/.duckdb/extensions/<version>/libhello_ext.duckdb_extension but I receive the following...

D INSTALL '/Users/taylorbeever/.duckdb/extensions/v0.8.1/osx_arm64/libhello_ext.duckdb_extension';
D LOAD libhello_ext;
Error: IO Error: Extension "/Users/taylorbeever/.duckdb/extensions/v0.8.1/osx_arm64/libhello_ext.duckdb_extension" could not be loaded: dlopen(/Users/taylorbeever/.duckdb/extensions/v0.8.1/osx_arm64/libhello_ext.duckdb_extension, 0x0006): tried: '/Users/taylorbeever/.duckdb/extensions/v0.8.1/osx_arm64/libhello_ext.duckdb_extension' (not a mach-o file), '/System/Volumes/Preboot/Cryptexes/OS/Users/taylorbeever/.duckdb/extensions/v0.8.1/osx_arm64/libhello_ext.duckdb_extension' (no such file), '/Users/taylorbeever/.duckdb/extensions/v0.8.1/osx_arm64/libhello_ext.duckdb_extension' (not a mach-o file)

[rustyconover]

Hello,

To demonstrate the example extension function follow these steps.

0. Use the cargo build command you previously ran. It won't place the output in ~/.duckdb it will produce output under the source directory of duckdb-rs.

1. Start duckdb so it allows unsigned extensions with:

duckdb -unsigned

3. Install and load the extension in the duckdb session.

> install '/Users/rusty/Development/duckdb-rs/target/debug/examples/libhello_ext.dylib';
> load '/Users/rusty/Development/duckdb-rs/target/debug/examples/libhello_ext.dylib';

Since I'm using Mac OS X, rust produced a .dylib file. If I was using linux it would produce a .so file.

> select * from hello('rusty');
┌─────────────┐
│   column0   │
│   varchar   │
├─────────────┤
│ Hello rusty │
└─────────────┘

[theelderbeever]

@rustyconover Thanks a ton!

[theelderbeever]

@wangfenjin @rustyconover it looks like the hello_ext example registers a table function... Is there any way to register a column function that lets you access the underlying arrow arrays of a given column?

I am trying to implement a BYTES_SUM function to handle summation of numeric values that don't fit in arrow and are stored as binary data such as little endian bytes. Over in the pola-rs world I essentially did the following where s is a polars::Series using arrow and BigInt is from the num_bigint crate.

    let sum = s.iter().fold(BigInt::from(0), |acc, x| {
        if let AnyValue::Binary(b) = x {
            acc + BigInt::from_bytes_le(Sign::Plus, b)
        } else {
            acc
        }
    });

With the goal being

SELECT BYTE_SUM(column) FROM 'some-file.parquet'
-- 19997900000000000000000000000

[rustyconover]

@theelderbeever from what I've read over the last few weeks the problem right now is it is not yet possible to implement UDFs for DuckDB in any other language that C++. While table returning functions have a C based API, there isn't a C API for UDFs. If there was a C based API, duckdb-rs could bind to those functions and expose a method for UDFs.

[theelderbeever]

@rustyconover Ah yeah, I had seen that in their docs for the Python API. Makes sense it would extend to the other languages as well.