duckdb_from_source option disappeared in the v0.9.0 upgrade

[michaelmdresser]

I noticed that in https://github.com/marcboeker/go-duckdb/pull/125 (the v0.9.0 upgrade) that the duckdb_from_source option is no longer available. What is the recommended path to achieve the same behavior after that change?

[marcboeker]

The amalgamation file became so large that it caused memory issues on some machines, including the free GitHub Actions runners, preventing compilation. The DuckDB team advised against using this file.

To build your own static libraries, either execute make deps.linux.amd64 beforehand or follow the Makefile instructions to build the libraries yourself. You can then link against these libraries.

git clone -b v${DUCKDB_VERSION} --depth 1 https://github.com/duckdb/duckdb.git
cd duckdb && \
CFLAGS="-O3" CXXFLAGS="-O3" make -j 2 && \
BUILD_SHELL=0 BUILD_UNITTESTS=0 make -j 2 && \
mkdir -p lib && \
for f in `find . -name '*.o'`; do cp $$f lib; done && \
cd lib && \
ar rvs ../libduckdb.a *.o && \
cd .. && \
mv libduckdb.a ../deps/linux_amd64/libduckdb.a

Is there another reason you must build it from source?

[michaelmdresser]

The amalgamation file became so large that it caused memory issues on some machines, including the free GitHub Actions runners, preventing compilation. The DuckDB team advised against using this file.

That absolutely makes sense, thank you for the explanation and for the point towards the makefile.

Is there another reason you must build it from source?

We have to build a variety of Docker images from different bases, and we've had trouble using go-duckdb because the libstdc++ version used to build libduckdb.a doesn't match the version of libstdc++ available for the container base. Also, in the case of Alpine, the inevitable musl vs. glibc situation. We can build around a need to compile manually in those situations.

Thanks for maintaining this library!