search

apache / arrow-adbc

Database connectivity API standard and libraries for Apache Arrow
https://arrow.apache.org/adbc/
Apache License 2.0
364 stars 89 forks source link

Installation of adbcflightsql fails #1703

Open krlmlr opened 6 months ago

krlmlr commented 6 months ago

What happened?

Can't clone and install adbcflightsql from e9f2122b449e6a5a5a22b3c8365cedb4b5e561fc.

How can we reproduce the bug?

R CMD INSTALL r/adbcflightsql emits the following output:

* installing to library ‘/Users/kirill/Library/R/arm64/4.3/library’
* installing *source* package ‘adbcflightsql’ ...
** using staged installation
Vendoring files from arrow-adbc/go/adbc to src/go/adbc:
- ../../go/adbc/LICENSE.txt  ->  src/go/adbc/LICENSE.txt
- ../../go/adbc/adbc.go  ->  src/go/adbc/adbc.go
- ../../go/adbc/driver/flightsql/cmd/testserver/main.go  ->  src/go/adbc/driver/flightsql/cmd/testserver/main.go
- ../../go/adbc/driver/flightsql/flightsql_adbc_server_test.go  ->  src/go/adbc/driver/flightsql/flightsql_adbc_server_test.go
- ../../go/adbc/driver/flightsql/flightsql_adbc_test.go  ->  src/go/adbc/driver/flightsql/flightsql_adbc_test.go
- ../../go/adbc/driver/flightsql/flightsql_connection.go  ->  src/go/adbc/driver/flightsql/flightsql_connection.go
- ../../go/adbc/driver/flightsql/flightsql_database.go  ->  src/go/adbc/driver/flightsql/flightsql_database.go
- ../../go/adbc/driver/flightsql/flightsql_driver.go  ->  src/go/adbc/driver/flightsql/flightsql_driver.go
- ../../go/adbc/driver/flightsql/flightsql_statement.go  ->  src/go/adbc/driver/flightsql/flightsql_statement.go
- ../../go/adbc/driver/flightsql/logging.go  ->  src/go/adbc/driver/flightsql/logging.go
- ../../go/adbc/driver/flightsql/record_reader.go  ->  src/go/adbc/driver/flightsql/record_reader.go
- ../../go/adbc/driver/flightsql/record_reader_test.go  ->  src/go/adbc/driver/flightsql/record_reader_test.go
- ../../go/adbc/driver/flightsql/timeouts.go  ->  src/go/adbc/driver/flightsql/timeouts.go
- ../../go/adbc/driver/flightsql/utils.go  ->  src/go/adbc/driver/flightsql/utils.go
- ../../go/adbc/driver/internal/driverbase/connection.go  ->  src/go/adbc/driver/internal/driverbase/connection.go
- ../../go/adbc/driver/internal/driverbase/database.go  ->  src/go/adbc/driver/internal/driverbase/database.go
- ../../go/adbc/driver/internal/driverbase/driver.go  ->  src/go/adbc/driver/internal/driverbase/driver.go
- ../../go/adbc/driver/internal/driverbase/driver_info.go  ->  src/go/adbc/driver/internal/driverbase/driver_info.go
- ../../go/adbc/driver/internal/driverbase/driver_info_test.go  ->  src/go/adbc/driver/internal/driverbase/driver_info_test.go
- ../../go/adbc/driver/internal/driverbase/driver_test.go  ->  src/go/adbc/driver/internal/driverbase/driver_test.go
- ../../go/adbc/driver/internal/driverbase/error.go  ->  src/go/adbc/driver/internal/driverbase/error.go
- ../../go/adbc/driver/internal/driverbase/logging.go  ->  src/go/adbc/driver/internal/driverbase/logging.go
- ../../go/adbc/driver/internal/shared_utils.go  ->  src/go/adbc/driver/internal/shared_utils.go
- ../../go/adbc/driver/panicdummy/panicdummy_adbc.go  ->  src/go/adbc/driver/panicdummy/panicdummy_adbc.go
- ../../go/adbc/driver/snowflake/bulk_ingestion.go  ->  src/go/adbc/driver/snowflake/bulk_ingestion.go
- ../../go/adbc/driver/snowflake/connection.go  ->  src/go/adbc/driver/snowflake/connection.go
- ../../go/adbc/driver/snowflake/connection_test.go  ->  src/go/adbc/driver/snowflake/connection_test.go
- ../../go/adbc/driver/snowflake/driver.go  ->  src/go/adbc/driver/snowflake/driver.go
- ../../go/adbc/driver/snowflake/driver_test.go  ->  src/go/adbc/driver/snowflake/driver_test.go
- ../../go/adbc/driver/snowflake/record_reader.go  ->  src/go/adbc/driver/snowflake/record_reader.go
- ../../go/adbc/driver/snowflake/snowflake_database.go  ->  src/go/adbc/driver/snowflake/snowflake_database.go
- ../../go/adbc/driver/snowflake/statement.go  ->  src/go/adbc/driver/snowflake/statement.go
- ../../adbc.h  ->  src/go/adbc/drivermgr/adbc.h
- ../../go/adbc/drivermgr/adbc_driver_manager.h  ->  src/go/adbc/drivermgr/adbc_driver_manager.h
- ../../go/adbc/drivermgr/doc.go  ->  src/go/adbc/drivermgr/doc.go
- ../../go/adbc/drivermgr/wrapper.go  ->  src/go/adbc/drivermgr/wrapper.go
- ../../go/adbc/drivermgr/wrapper_sqlite_test.go  ->  src/go/adbc/drivermgr/wrapper_sqlite_test.go
- ../../go/adbc/ext.go  ->  src/go/adbc/ext.go
- ../../go/adbc/go.mod  ->  src/go/adbc/go.mod
- ../../go/adbc/go.sum  ->  src/go/adbc/go.sum
- ../../go/adbc/infocode_string.go  ->  src/go/adbc/infocode_string.go
- ../../go/adbc/pkg/doc.go  ->  src/go/adbc/pkg/doc.go
- ../../go/adbc/pkg/flightsql/driver.go  ->  src/go/adbc/pkg/flightsql/driver.go
- ../../go/adbc/pkg/flightsql/utils.c  ->  src/go/adbc/pkg/flightsql/utils.c
- ../../go/adbc/pkg/flightsql/utils.h  ->  src/go/adbc/pkg/flightsql/utils.h
- ../../go/adbc/pkg/gen/main.go  ->  src/go/adbc/pkg/gen/main.go
- ../../go/adbc/pkg/panicdummy/driver.go  ->  src/go/adbc/pkg/panicdummy/driver.go
- ../../go/adbc/pkg/panicdummy/utils.c  ->  src/go/adbc/pkg/panicdummy/utils.c
- ../../go/adbc/pkg/panicdummy/utils.h  ->  src/go/adbc/pkg/panicdummy/utils.h
- ../../go/adbc/pkg/snowflake/driver.go  ->  src/go/adbc/pkg/snowflake/driver.go
- ../../go/adbc/pkg/snowflake/utils.c  ->  src/go/adbc/pkg/snowflake/utils.c
- ../../go/adbc/pkg/snowflake/utils.h  ->  src/go/adbc/pkg/snowflake/utils.h
- ../../go/adbc/sqldriver/doc.go  ->  src/go/adbc/sqldriver/doc.go
- ../../go/adbc/sqldriver/driver.go  ->  src/go/adbc/sqldriver/driver.go
- ../../go/adbc/sqldriver/driver_internals_test.go  ->  src/go/adbc/sqldriver/driver_internals_test.go
- ../../go/adbc/sqldriver/driver_test.go  ->  src/go/adbc/sqldriver/driver_test.go
- ../../go/adbc/sqldriver/flightsql/flightsql.go  ->  src/go/adbc/sqldriver/flightsql/flightsql.go
- ../../go/adbc/sqldriver/flightsql/flightsql_test.go  ->  src/go/adbc/sqldriver/flightsql/flightsql_test.go
- ../../go/adbc/standard_schemas.go  ->  src/go/adbc/standard_schemas.go
- ../../go/adbc/status_string.go  ->  src/go/adbc/status_string.go
- ../../go/adbc/tools.go  ->  src/go/adbc/tools.go
- ../../go/adbc/utils/utils.go  ->  src/go/adbc/utils/utils.go
- ../../go/adbc/validation/validation.go  ->  src/go/adbc/validation/validation.go
All files successfully copied to src/go/adbc
Trying 'go version' with GO_BIN at '/opt/homebrew/bin/go'
go version go1.22.0 darwin/arm64
Found vendored ADBC FlightSQL driver
** libs
using C compiler: ‘Apple clang version 15.0.0 (clang-1500.3.9.4)’
using SDK: ‘MacOSX14.4.sdk’
ccache clang -arch arm64 -I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/include" -DNDEBUG -I/Users/kirill/git/arrow-adbc/r/adbcflightsql/src/src -DADBC_EXPORT=""  -I/opt/homebrew/include    -fPIC  -O3 -c init.c -o init.o
(cd "/Users/kirill/git/arrow-adbc/r/adbcflightsql/src/go/adbc"; GOMAXPROCS= GOPATH="/Users/kirill/git/arrow-adbc/r/adbcflightsql/src/.go-path" GOCACHE="/Users/kirill/git/arrow-adbc/r/adbcflightsql/src/.go-cache" GOFLAGS=-modcacherw CC="ccache clang -arch arm64" CXX="ccache clang++ -arch arm64 -std=gnu++17" CGO_CFLAGS="-I"/Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/include" -DNDEBUG -I/Users/kirill/git/arrow-adbc/r/adbcflightsql/src/src -DADBC_EXPORT=""  -I/opt/homebrew/include  " CGO_LDFLAGS="-L/Users/kirill/git/arrow-adbc/r/adbcflightsql/src/go -ladbc_driver_flightsql -lresolv -framework Security  " "/opt/homebrew/bin/go" build -v -tags driverlib -o /Users/kirill/git/arrow-adbc/r/adbcflightsql/src/go/libadbc_driver_flightsql.a -buildmode=c-archive "./pkg/flightsql")
go: downloading github.com/apache/arrow/go/v16 v16.0.0-20240401151559-71321841eb6d
go: downloading google.golang.org/protobuf v1.33.0
go: downloading github.com/bluele/gcache v0.0.2
go: downloading golang.org/x/exp v0.0.0-20240318143956-a85f2c67cd81
go: downloading golang.org/x/sync v0.6.0
go: downloading google.golang.org/grpc v1.62.1
go: downloading golang.org/x/sys v0.18.0
go: downloading golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028
go: downloading github.com/google/flatbuffers v24.3.7+incompatible
go: downloading github.com/klauspost/compress v1.17.7
go: downloading github.com/pierrec/lz4/v4 v4.1.21
go: downloading golang.org/x/net v0.22.0
go: downloading github.com/golang/protobuf v1.5.4
go: downloading github.com/zeebo/xxh3 v1.0.2
go: downloading github.com/goccy/go-json v0.10.2
go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20240123012728-ef4313101c80
go: downloading golang.org/x/text v0.14.0
internal/coverage/rtcov
internal/race
internal/unsafeheader
internal/goarch
internal/godebugs
internal/goos
internal/goexperiment
runtime/internal/math
runtime/internal/sys
internal/chacha8rand
internal/cpu
internal/abi
internal/itoa
math/bits
unicode/utf8
sync/atomic
unicode
cmp
slices
github.com/apache/arrow/go/v16/arrow/internal/debug
math
encoding
unicode/utf16
internal/bytealg
golang.org/x/exp/constraints
golang.org/x/xerrors/internal
runtime/internal/atomic
hash/maphash
google.golang.org/protobuf/internal/flags
google.golang.org/protobuf/internal/set
log/internal
log/slog/internal
container/list
crypto/internal/alias
crypto/subtle
crypto/internal/boring/sig
vendor/golang.org/x/crypto/cryptobyte/asn1
internal/nettrace
vendor/golang.org/x/crypto/internal/alias
google.golang.org/grpc/serviceconfig
github.com/pierrec/lz4/v4/internal/lz4errors
golang.org/x/exp/maps
golang.org/x/exp/slices
runtime
github.com/klauspost/compress
internal/reflectlite
sync
internal/bisect
google.golang.org/protobuf/internal/pragma
log/slog/internal/buffer
internal/singleflight
internal/testlog
errors
internal/godebug
sort
runtime/cgo
internal/oserror
internal/safefilepath
path
io
strconv
math/rand
syscall
crypto/internal/nistec/fiat
vendor/golang.org/x/net/dns/dnsmessage
bytes
strings
github.com/google/flatbuffers/go
hash
bufio
crypto/internal/randutil
reflect
time
github.com/apache/arrow/go/v16/arrow/internal/flatbuf
internal/syscall/unix
internal/syscall/execenv
hash/fnv
hash/crc32
context
crypto/rc4
crypto
io/fs
crypto/x509/internal/macos
internal/poll
internal/intern
embed
html
regexp/syntax
internal/fmtsort
google.golang.org/protobuf/internal/editiondefaults
github.com/goccy/go-json/internal/runtime
encoding/binary
net/netip
os
vendor/golang.org/x/text/transform
net/http/internal/ascii
text/tabwriter
regexp
google.golang.org/grpc/backoff
google.golang.org/grpc/internal/grpcrand
google.golang.org/grpc/internal/buffer
golang.org/x/text/transform
github.com/apache/arrow/go/v16/arrow/endian
github.com/apache/arrow/go/v16/arrow/float16
encoding/base64
github.com/zeebo/xxh3
crypto/cipher
crypto/internal/edwards25519/field
crypto/internal/nistec
crypto/internal/boring
google.golang.org/protobuf/internal/detrand
crypto/des
fmt
golang.org/x/sys/cpu
io/ioutil
crypto/aes
crypto/sha512
crypto/internal/edwards25519
crypto/hmac
crypto/md5
crypto/sha1
crypto/sha256
encoding/pem
vendor/golang.org/x/net/route
vendor/golang.org/x/crypto/chacha20
vendor/golang.org/x/crypto/internal/poly1305
vendor/golang.org/x/crypto/hkdf
crypto/ecdh
path/filepath
google.golang.org/grpc/internal/envconfig
google.golang.org/grpc/internal/backoff
google.golang.org/grpc/internal/grpcsync
google.golang.org/grpc/keepalive
vendor/golang.org/x/crypto/chacha20poly1305
github.com/klauspost/compress/internal/snapref
github.com/klauspost/compress/zstd/internal/xxhash
github.com/pierrec/lz4/v4/internal/lz4block
github.com/pierrec/lz4/v4/internal/xxh32
os/signal
github.com/goccy/go-json/internal/errors
google.golang.org/protobuf/internal/errors
go/token
golang.org/x/xerrors
encoding/json
github.com/apache/arrow/go/v16/arrow/memory
math/big
google.golang.org/protobuf/encoding/protowire
compress/flate
google.golang.org/protobuf/internal/version
log
encoding/hex
google.golang.org/protobuf/reflect/protoreflect
github.com/apache/arrow/go/v16/arrow/bitutil
net/url
database/sql/driver
runtime/debug
golang.org/x/net/internal/timeseries
database/sql
vendor/golang.org/x/text/unicode/bidi
text/template/parse
compress/gzip
google.golang.org/protobuf/internal/encoding/messageset
google.golang.org/protobuf/internal/genid
google.golang.org/protobuf/internal/order
google.golang.org/protobuf/internal/strs
google.golang.org/protobuf/reflect/protoregistry
google.golang.org/protobuf/runtime/protoiface
google.golang.org/protobuf/internal/descfmt
google.golang.org/protobuf/internal/descopts
google.golang.org/protobuf/internal/encoding/text
google.golang.org/protobuf/proto
vendor/golang.org/x/text/secure/bidirule
vendor/golang.org/x/text/unicode/norm
vendor/golang.org/x/net/http2/hpack
mime
text/template
google.golang.org/protobuf/internal/encoding/defval
github.com/goccy/go-json/internal/decoder
google.golang.org/protobuf/internal/filedesc
github.com/goccy/go-json/internal/encoder
google.golang.org/protobuf/encoding/prototext
log/slog
github.com/apache/arrow/go/v16/arrow/decimal128
github.com/apache/arrow/go/v16/arrow/decimal256
crypto/elliptic
crypto/internal/boring/bbig
crypto/internal/bigmod
encoding/asn1
crypto/rand
crypto/dsa
crypto/ed25519
html/template
github.com/goccy/go-json/internal/encoder/vm_color_indent
crypto/rsa
vendor/golang.org/x/net/idna
google.golang.org/protobuf/internal/encoding/tag
vendor/golang.org/x/crypto/cryptobyte
google.golang.org/protobuf/internal/impl
crypto/x509/pkix
mime/quotedprintable
crypto/ecdsa
net/http/internal
google.golang.org/grpc/internal/grpclog
google.golang.org/grpc/grpclog
google.golang.org/grpc/attributes
google.golang.org/grpc/connectivity
golang.org/x/sys/unix
google.golang.org/grpc/internal/idle
google.golang.org/protobuf/internal/encoding/json
golang.org/x/text/unicode/bidi
golang.org/x/text/unicode/norm
golang.org/x/text/secure/bidirule
google.golang.org/protobuf/encoding/protojson
golang.org/x/net/http2/hpack
github.com/klauspost/compress/fse
github.com/pierrec/lz4/v4/internal/lz4stream
github.com/bluele/gcache
github.com/pierrec/lz4/v4
github.com/klauspost/compress/huff0
golang.org/x/sync/errgroup
github.com/apache/arrow/go/v16/arrow/memory/mallocator
github.com/klauspost/compress/zstd
net
golang.org/x/net/idna
google.golang.org/protobuf/internal/filetype
google.golang.org/protobuf/runtime/protoimpl
google.golang.org/protobuf/protoadapt
google.golang.org/protobuf/types/known/anypb
google.golang.org/protobuf/types/known/timestamppb
google.golang.org/protobuf/types/known/durationpb
google.golang.org/protobuf/types/descriptorpb
google.golang.org/genproto/googleapis/rpc/status
google.golang.org/grpc/binarylog/grpc_binarylog_v1
google.golang.org/protobuf/types/gofeaturespb
google.golang.org/protobuf/reflect/protodesc
github.com/golang/protobuf/proto
google.golang.org/grpc/internal/pretty
github.com/goccy/go-json/internal/encoder/vm_color
google.golang.org/grpc/internal/resolver/dns/internal
google.golang.org/grpc/internal/syscall
vendor/golang.org/x/net/http/httpproxy
net/textproto
crypto/x509
vendor/golang.org/x/net/http/httpguts
golang.org/x/net/http/httpguts
mime/multipart
crypto/tls
net/http/httptrace
google.golang.org/grpc/internal/credentials
google.golang.org/grpc/credentials
net/http
google.golang.org/grpc/resolver
google.golang.org/grpc/credentials/insecure
google.golang.org/grpc/peer
google.golang.org/grpc/balancer/grpclb/state
google.golang.org/grpc/internal/transport/networktype
google.golang.org/grpc/internal/resolver/passthrough
google.golang.org/grpc/internal
google.golang.org/grpc/internal/resolver/dns
google.golang.org/grpc/internal/resolver/unix
google.golang.org/grpc/metadata
google.golang.org/grpc/codes
google.golang.org/grpc/internal/channelz
google.golang.org/grpc/resolver/dns
google.golang.org/grpc/internal/status
google.golang.org/grpc/internal/balancerload
google.golang.org/grpc/internal/metadata
google.golang.org/grpc/internal/grpcutil
google.golang.org/grpc/stats
google.golang.org/grpc/tap
google.golang.org/grpc/status
google.golang.org/grpc/encoding
google.golang.org/grpc/encoding/proto
google.golang.org/grpc/internal/binarylog
google.golang.org/grpc/channelz
google.golang.org/grpc/balancer
google.golang.org/grpc/internal/serviceconfig
google.golang.org/grpc/balancer/base
google.golang.org/grpc/internal/resolver
google.golang.org/grpc/balancer/roundrobin
google.golang.org/grpc/internal/balancer/gracefulswitch
github.com/goccy/go-json/internal/encoder/vm_indent
golang.org/x/net/trace
net/http/httputil
golang.org/x/net/http2
google.golang.org/grpc/internal/transport
google.golang.org/grpc
github.com/apache/arrow/go/v16/arrow/flight/gen/flight
github.com/goccy/go-json/internal/encoder/vm
github.com/goccy/go-json
github.com/apache/arrow/go/v16/internal/json
github.com/apache/arrow/go/v16/arrow
github.com/apache/arrow/go/v16/arrow/arrio
github.com/apache/arrow-adbc/go/adbc/utils
github.com/apache/arrow/go/v16/arrow/encoded
github.com/apache/arrow/go/v16/arrow/internal
github.com/apache/arrow/go/v16/arrow/flight/flightsql/schema_ref
github.com/apache/arrow/go/v16/internal/utils
github.com/apache/arrow/go/v16/internal/bitutils
github.com/apache/arrow/go/v16/internal/hashing
github.com/apache/arrow/go/v16/arrow/array
github.com/apache/arrow/go/v16/arrow/internal/dictutils
github.com/apache/arrow-adbc/go/adbc
github.com/apache/arrow/go/v16/arrow/ipc
github.com/apache/arrow-adbc/go/adbc/driver/internal
github.com/apache/arrow-adbc/go/adbc/driver/internal/driverbase
github.com/apache/arrow/go/v16/arrow/flight
github.com/apache/arrow/go/v16/arrow/cdata
github.com/apache/arrow/go/v16/arrow/flight/flightsql
github.com/apache/arrow-adbc/go/adbc/driver/flightsql
# github.com/apache/arrow/go/v16/arrow/cdata
In file included from ../../.go-path/pkg/mod/github.com/apache/arrow/go/v16@v16.0.0-20240401151559-71321841eb6d/arrow/cdata/cdata.go:25:
/opt/homebrew/include/arrow/c/helpers.h:53:3: error: call to undeclared function 'assert'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  assert(dest != src);
  ^
/opt/homebrew/include/arrow/c/helpers.h:81:3: error: call to undeclared function 'assert'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  assert(dest != src);
  ^
/opt/homebrew/include/arrow/c/helpers.h:112:3: error: call to undeclared function 'assert'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
  assert(dest != src);
  ^
3 errors generated.
make: *** [gostatic] Error 1
ERROR: compilation failed for package ‘adbcflightsql’
* removing ‘/Users/kirill/Library/R/arm64/4.3/library/adbcflightsql’

Environment/Setup

macOS M2, latest updates, R 4.3.2, go 1.22.0

zeroshade commented 6 months ago

@paleolimbot interesting, is there an include missing from helpers.h? It's weird that this is failing here with R but not in other circumstances

paleolimbot commented 6 months ago

I don't think it's a missing include on Go's side, I think it's that the Go install is picking up the include file from /opt/homebrew (which does appear to be missing #include <assert.h>): https://github.com/apache/arrow/blob/8163d026b3c56253d9e33c0129fac5d9ba573c53/cpp/src/arrow/c/helpers.h

I have also had problems with go vendor not copying over the those headers in the subdirectory, and I wonder if it would be safer to put those files in the cdata/ directory.

The workaround would probably be to brew uninstall apache-arrow (with apologies since that will also uninstall gdal).

krlmlr commented 5 months ago

Thanks! Confirming workaround.