install

[jhartman86]

Build fails with Apache Arrow *.a file missing.

Tried doing install w/ just go modules in package, and getting a copy of libduckdb and following the linking instructions - neither seemed to work.

[marcboeker]

This seems to be an Apache Arrow related problem. Do you have the required apache-arrow library installed?

On macOS: brew install apache-arrow

[jhartman86]

Hey @marcboeker - thx for the fast reply. Did not have apache-arrow installed but will try that; had been trying w/ go modules only so far. The error message lead me to believe the include was looking for arrow/c/abi.h from the arrow package in the vendor directory (and vendor/github.com/apache/arrow/go/v14/arrow/cdata is present), its just the abi.h file that isn't.

Will try installing apache-arrow and follow back up here.

Would you expect that issue to occur when dynamically linking libduckdb too?

[marcboeker]

Has it worked for you installing the Apache Arrow lib?

The missing abi.h file has nothing to do with DuckDB or libduckdb as this is a dependency from the Apache Arrow Go package.

[jhartman86]

@marcboeker it hasn't yet but I've still been mucking around trying to get it to work. Disclaimer: I'm not particularly well versed in tracking down problems getting C/C++ dependencies to compile.

Going to spend some more time on it tonight and will post back here w/ any updates.

[grounded042]

I'm also hitting this issue. I was not hitting it with v1.5.6. If I change back to 1.5.6 the error does not happen.

[jhartman86]

@grounded042 I still haven't found a way around it either. If I do I'll let you know back here

[marcboeker]

@jhartman86 Could you please remove the go-duckdb module completely and list the steps to reproduce the problem. I can than dig into it. Thanks!

[jhartman86]

Hey @marcboeker thx for the help w/ this. Here's what I've done so far:

Using docker for an isolated environment; related dockerfile:

FROM golang:1.22.0-bookworm

# Defaults to local, unless passed in
ARG BUILD_VERSION=local

ENV CGO_ENABLED 1
ENV TZ=UTC

RUN apt update && apt install -y \
  ca-certificates \
  tzdata \
  bash \
  gcc \
  cmake \
  git \
  openssl \
  curl \
  unzip \
  wget \
  build-essential \
  dpkg \
  inotify-tools \
  libc6 \
  wget \
  lsb-release

RUN wget https://github.com/duckdb/duckdb/releases/download/v0.10.0/libduckdb-linux-aarch64.zip -P /opt && \
  unzip /opt/libduckdb-linux-aarch64.zip -d /opt && \
  rm /opt/libduckdb-linux-aarch64.zip

RUN apt update && \
  wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb && \
  apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb && \
  apt update && \
  apt install -y -V libarrow-dev && \
  apt install -y -V libarrow-glib-dev && \
  apt install -y -V libarrow-dataset-dev && \
  apt install -y -V libarrow-dataset-glib-dev && \
  apt install -y -V libarrow-acero-dev && \
  apt install -y -V libarrow-flight-dev && \
  apt install -y -V libarrow-flight-glib-dev && \
  apt install -y -V libarrow-flight-sql-dev && \
  apt install -y -V libarrow-flight-sql-glib-dev && \
  apt install -y -V libgandiva-dev && \
  apt install -y -V libgandiva-glib-dev && \
  apt install -y -V libparquet-dev && \
  apt install -y -V libparquet-glib-dev

Tried builds with both of the following:

# linking
CGO_ENABLED=1 CGO_LDFLAGS="-L/opt" GOOS=linux go build \
  -o /go/bin/fractools \
  -mod=vendor \
  -tags netgo \
  -gcflags "all=-N -l" \
  -ldflags "$BUILD_VARS" \
  -tags=duckdb_use_lib \
  "$ROOTPATH/main.go"

# bundled duckdb lib
CGO_ENABLED=1 GOOS=linux go build \
  -o /go/bin/fractools \
  -mod=vendor \
  -tags netgo \
  -gcflags "all=-N -l" \
  -ldflags "$BUILD_VARS" \
  "$ROOTPATH/main.go"

All attempts result in the same thing:

# command-line-arguments
/usr/local/go/pkg/tool/linux_arm64/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-2376170843/000031.o: in function `ArrowArrayMove':
/usr/include/arrow/c/helpers.h:81: undefined reference to `assert'
/usr/bin/ld: /usr/include/arrow/c/helpers.h:82: undefined reference to `assert'
/usr/bin/ld: /tmp/go-link-2376170843/000031.o: in function `ArrowArrayStreamMove':
/usr/include/arrow/c/helpers.h:112: undefined reference to `assert'
/usr/bin/ld: /usr/include/arrow/c/helpers.h:113: undefined reference to `assert'
collect2: error: ld returned 1 exit status

---

I deleted the repo per your request then re-added via normal go get (using go modules), then also did a direct download from github and copied into vendors (to ensure the bundled dependencies that go modules installations misses were present).

Did all of these ^ steps with both v1.6.1 tag, then with master @ commit ID 8b5c861527b720d36e62809e58dbcf40fcfada5f. All resultted in the same build error.

---

Of note, this was all working on version 1.5.4 previously.

Let me know if there's anything I can provide or try to help. I'm still trying more things and if I find anything will certainly reply back on this thread. Thanks again @marcboeker

[marcboeker]

Thanks for the detailed description @jhartman86!

I was able to reproduce this with the mod vendor approach. When not using mod vendoring, everything works fine. Is there a specific reason why you need to use vendoring?

Maybe remove the vendor directory and -mod=vendor from the build command. This should work. Could you confirm this?

[grounded042]

I've been able to work around this issue and it doesn't seem to track with everything mentioned here. For me all I had to do was run modvendor -copy="**/*.a **/*.h" -v in order to copy files like https://github.com/apache/arrow/blob/main/go/arrow/cdata/arrow/c/abi.h. The tool is from here: https://github.com/goware/modvendor.

The Dockerfile looks like this:

FROM golang:1.21.1
WORKDIR /workspace
COPY . .

RUN CGO_ENABLED=1 GOOS=linux go build -o /go/bin -a ./cmd/...

before running that modvendor command I would get this error:

after running that modvendor command I would get a successful build:

[taniabogatsch]

Linking to https://github.com/marcboeker/go-duckdb/issues/77.

[jhartman86]

@grounded042 awesome, that seems a likely candidate for solving this (going to try tonight and will comment back here).

@marcboeker - sorry, got wrapped up with other stuff and haven't tried skipping mod vendor yet; I'll also give that a try tonight and comment back here.

Thanks for all the help guys.

[marcboeker]

@grounded042 Thanks for figuring this out. I was able to reproduce and fix this with your approach and modvendor. To make it easier for others, I have added a section about vendoring to the readme. See https://github.com/marcboeker/go-duckdb/?tab=readme-ov-file#vendoring