Closed jhartman86 closed 5 months ago
This seems to be an Apache Arrow related problem. Do you have the required apache-arrow library installed?
On macOS: brew install apache-arrow
Hey @marcboeker - thx for the fast reply. Did not have apache-arrow installed but will try that; had been trying w/ go modules only so far. The error message lead me to believe the include was looking for arrow/c/abi.h
from the arrow
package in the vendor directory (and vendor/github.com/apache/arrow/go/v14/arrow/cdata
is present), its just the abi.h
file that isn't.
Will try installing apache-arrow and follow back up here.
Would you expect that issue to occur when dynamically linking libduckdb
too?
Has it worked for you installing the Apache Arrow lib?
The missing abi.h
file has nothing to do with DuckDB or libduckdb
as this is a dependency from the Apache Arrow Go package.
@marcboeker it hasn't yet but I've still been mucking around trying to get it to work. Disclaimer: I'm not particularly well versed in tracking down problems getting C/C++ dependencies to compile.
Going to spend some more time on it tonight and will post back here w/ any updates.
I'm also hitting this issue. I was not hitting it with v1.5.6. If I change back to 1.5.6 the error does not happen.
@grounded042 I still haven't found a way around it either. If I do I'll let you know back here
@jhartman86 Could you please remove the go-duckdb module completely and list the steps to reproduce the problem. I can than dig into it. Thanks!
Hey @marcboeker thx for the help w/ this. Here's what I've done so far:
Using docker for an isolated environment; related dockerfile:
FROM golang:1.22.0-bookworm
# Defaults to local, unless passed in
ARG BUILD_VERSION=local
ENV CGO_ENABLED 1
ENV TZ=UTC
RUN apt update && apt install -y \
ca-certificates \
tzdata \
bash \
gcc \
cmake \
git \
openssl \
curl \
unzip \
wget \
build-essential \
dpkg \
inotify-tools \
libc6 \
wget \
lsb-release
RUN wget https://github.com/duckdb/duckdb/releases/download/v0.10.0/libduckdb-linux-aarch64.zip -P /opt && \
unzip /opt/libduckdb-linux-aarch64.zip -d /opt && \
rm /opt/libduckdb-linux-aarch64.zip
RUN apt update && \
wget https://apache.jfrog.io/artifactory/arrow/$(lsb_release --id --short | tr 'A-Z' 'a-z')/apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb && \
apt install -y -V ./apache-arrow-apt-source-latest-$(lsb_release --codename --short).deb && \
apt update && \
apt install -y -V libarrow-dev && \
apt install -y -V libarrow-glib-dev && \
apt install -y -V libarrow-dataset-dev && \
apt install -y -V libarrow-dataset-glib-dev && \
apt install -y -V libarrow-acero-dev && \
apt install -y -V libarrow-flight-dev && \
apt install -y -V libarrow-flight-glib-dev && \
apt install -y -V libarrow-flight-sql-dev && \
apt install -y -V libarrow-flight-sql-glib-dev && \
apt install -y -V libgandiva-dev && \
apt install -y -V libgandiva-glib-dev && \
apt install -y -V libparquet-dev && \
apt install -y -V libparquet-glib-dev
Tried builds with both of the following:
# linking
CGO_ENABLED=1 CGO_LDFLAGS="-L/opt" GOOS=linux go build \
-o /go/bin/fractools \
-mod=vendor \
-tags netgo \
-gcflags "all=-N -l" \
-ldflags "$BUILD_VARS" \
-tags=duckdb_use_lib \
"$ROOTPATH/main.go"
# bundled duckdb lib
CGO_ENABLED=1 GOOS=linux go build \
-o /go/bin/fractools \
-mod=vendor \
-tags netgo \
-gcflags "all=-N -l" \
-ldflags "$BUILD_VARS" \
"$ROOTPATH/main.go"
All attempts result in the same thing:
# command-line-arguments
/usr/local/go/pkg/tool/linux_arm64/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-2376170843/000031.o: in function `ArrowArrayMove':
/usr/include/arrow/c/helpers.h:81: undefined reference to `assert'
/usr/bin/ld: /usr/include/arrow/c/helpers.h:82: undefined reference to `assert'
/usr/bin/ld: /tmp/go-link-2376170843/000031.o: in function `ArrowArrayStreamMove':
/usr/include/arrow/c/helpers.h:112: undefined reference to `assert'
/usr/bin/ld: /usr/include/arrow/c/helpers.h:113: undefined reference to `assert'
collect2: error: ld returned 1 exit status
I deleted the repo per your request then re-added via normal go get
(using go modules), then also did a direct download from github and copied into vendors
(to ensure the bundled dependencies that go modules installations misses were present).
Did all of these ^ steps with both v1.6.1
tag, then with master @ commit ID 8b5c861527b720d36e62809e58dbcf40fcfada5f
. All resultted in the same build error.
Of note, this was all working on version 1.5.4
previously.
Let me know if there's anything I can provide or try to help. I'm still trying more things and if I find anything will certainly reply back on this thread. Thanks again @marcboeker
Thanks for the detailed description @jhartman86!
I was able to reproduce this with the mod vendor approach. When not using mod vendoring, everything works fine. Is there a specific reason why you need to use vendoring?
Maybe remove the vendor
directory and -mod=vendor
from the build command. This should work. Could you confirm this?
I've been able to work around this issue and it doesn't seem to track with everything mentioned here. For me all I had to do was run modvendor -copy="**/*.a **/*.h" -v
in order to copy files like https://github.com/apache/arrow/blob/main/go/arrow/cdata/arrow/c/abi.h. The tool is from here: https://github.com/goware/modvendor.
The Dockerfile looks like this:
FROM golang:1.21.1
WORKDIR /workspace
COPY . .
RUN CGO_ENABLED=1 GOOS=linux go build -o /go/bin -a ./cmd/...
before running that modvendor
command I would get this error:
after running that modvendor
command I would get a successful build:
@grounded042 awesome, that seems a likely candidate for solving this (going to try tonight and will comment back here).
@marcboeker - sorry, got wrapped up with other stuff and haven't tried skipping mod vendor
yet; I'll also give that a try tonight and comment back here.
Thanks for all the help guys.
@grounded042 Thanks for figuring this out. I was able to reproduce and fix this with your approach and modvendor
. To make it easier for others, I have added a section about vendoring to the readme. See https://github.com/marcboeker/go-duckdb/?tab=readme-ov-file#vendoring
Build fails with Apache Arrow
*.a
file missing.Tried doing install w/ just go modules in package, and getting a copy of libduckdb and following the linking instructions - neither seemed to work.