search

bvinc / go-sqlite-lite

SQLite driver for the Go programming language
BSD 3-Clause "New" or "Revised" License
327 stars 34 forks source link

Cross platform builds? #10

Open sudo-suhas opened 6 years ago

sudo-suhas commented 6 years ago

When I try to cross compile my project which uses go-sqlite-lite, I get the following output:

➜ GOOS=linux GOARCH=amd64 go build -o bin/application ./cmd/rgserver
extern/sqlite/sqllite_reader.go:8:2: build constraints exclude all Go files in <GOPATH>/go/src/my-company-domain.in/modules/revgeo/vendor/github.com/bvinc/go-sqlite-lite/sqlite3

My understanding is that this problem originated from use of CGO. I solved it by using https://github.com/karalabe/xgo (https://github.com/mitchellh/gox did not work):

xgo -targets=linux/amd64 ./cmd/rgserver
Checking docker installation...
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:21:31 2018
 OS/Arch:           darwin/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:29:02 2018
  OS/Arch:          linux/amd64
  Experimental:     true

Checking for required docker image karalabe/xgo-latest... found.
Cross compiling my-company-domain/modules/revgeo/cmd/rgserver...
Building locallymy-company-domain/modules/revgeo/cmd/rgserver...
Compiling for linux/amd64...
Cleaning up build environment...

Do you think it would make sense to add a note to the readme regarding this?

bvinc commented 6 years ago

Thanks for reporting this. It might be worth adding something to the Readme.

But the error message you pasted me seems to simply because Go disables cgo when cross compiling. You can reenable it by setting the CGO_ENABLED environment variable. It seems like Go is the one with a pretty bad error message in this case.

sudo-suhas commented 6 years ago

This is what I get with CGO_ENABLED=1:

➜ CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build -o bin/application ./cmd/rgserver
# my-company-domain/modules/revgeo/vendor/github.com/bvinc/go-sqlite-lite/sqlite3
sqlite3.c:32519:42: error: use of undeclared identifier 'pread64'
sqlite3.c:32537:42: error: use of undeclared identifier 'pwrite64'
sqlite3.c:32669:22: error: invalid application of 'sizeof' to an incomplete type 'struct unix_syscall []'
sqlite3.c:32678:22: error: invalid application of 'sizeof' to an incomplete type 'struct unix_syscall []'
sqlite3.c:32705:20: error: invalid application of 'sizeof' to an incomplete type 'struct unix_syscall []'
sqlite3.c:32722:16: error: invalid application of 'sizeof' to an incomplete type 'struct unix_syscall []'
sqlite3.c:13941:38: note: expanded from macro 'ArraySize'
sqlite3.c:32726:14: error: invalid application of 'sizeof' to an incomplete type 'struct unix_syscall []'
sqlite3.c:13941:38: note: expanded from macro 'ArraySize'
sqlite3.c:35376:11: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
sqlite3.c:32523:49: note: expanded from macro 'osPread64'
sqlite3.c:35488:17: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
sqlite3.c:32541:57: note: expanded from macro 'osPwrite64'
sqlite3.c:182866:7: warning: array index 2 is past the end of the array (which contains 2 elements) [-Warray-bounds]
sqlite3.c:182442:3: note: array 'a' declared here
sqlite3.c:182867:7: warning: array index 3 is past the end of the array (which contains 2 elements) [-Warray-bounds]
sqlite3.c:182442:3: note: array 'a' declared here
sqlite3.c:182868:7: warning: array index 4 is past the end of the array (which contains 2 elements) [-Warray-bounds]
sqlite3.c:182442:3: note: array 'a' declared here
sqlite3.c:182869:7: warning: array index 5 is past the end of the array (which contains 2 elements) [-Warray-bounds]
sqlite3.c:182442:3: note: array 'a' declared here
sqlite3.c:182870:7: warning: array index 6 is past the end of the array (which contains 2 elements) [-Warray-bounds]
sqlite3.c:182442:3: note: array 'a' declared here
sqlite3.c:182871:7: warning: array index 7 is past the end of the array (which contains 2 elements) [-Warray-bounds]
sqlite3.c:182442:3: note: array 'a' declared here
bvinc commented 6 years ago

And now that you've enabled CGO, it seems that the issue is that you don't have the C environment and header files that it expects. I'm wondering if this is outside the bounds of what should be expected of this project.

At least the first 2 error messages (and maybe the rest) are a result of a decision I made. I configure the SQLite C code to inform it that pread64 and pwrite64 are available on Linux. But I guess they aren't available in your Linux environment.

#cgo linux,!android CFLAGS: -DHAVE_FDATASYNC=1
#cgo linux,!android CFLAGS: -DHAVE_PREAD64=1 -DHAVE_PWRITE64=1

I'm not sure if this is something that I need to change or not. pread64 and pwrite64 are pretty much always available on Linux (with the exception of android). Maybe the proper solution is just to use gox to make sure you have a proper Linux environment.

What OS and architecture are you using to do cross compilation?

sudo-suhas commented 6 years ago

I would suggest either linking to an article which has instructions for setting up C env or just suggesting to use xgo to keep it simple. As I mentioned earlier, I wasn't able to build with gox:

➜ gox -osarch="linux/amd64" ./cmd/rgserver
Number of parallel builds: 11

-->     linux/amd64: my-company-domain/modules/revgeo/cmd/rgserver

1 errors occurred:
--> linux/amd64 error: exit status 1
Stderr: extern/sqlite/sqllite_reader.go:8:2: build constraints exclude all Go files in <GOPATH>/src/my-company-domain/modules/revgeo/vendor/github.com/bvinc/go-sqlite-lite/sqlite3

What OS and architecture are you using to do cross compilation?

I am on MacBook Pro and here's the output from go env:

➜ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/myuser/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/myuser/dev/work/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.1/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/w_/2y9q2qdx3bnb521w5bb1kvj9j8bs6w/T/go-build072110941=/tmp/go-build -gno-record-gcc-switches -fno-common"
bvinc commented 6 years ago

I found the same issue here https://github.com/mattn/go-sqlite3/issues/532

I would prefer, if possible, to just make sure that SQLite is properly configured, instead of needing this additional documentation.

I did a little research, and I think the problem is that I was configuring SQLite to use pread64 and pwrite64, which are linux-only system calls. But if I just configure it to use pread and pwrite instead, it'll be just as good, but it'll be more standards compliant and work in your scenario.

Can you try out the latest master and see if you still have the problem with cross compiling?

sudo-suhas commented 6 years ago

Still facing issues:

➜ CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build -a -o bin/application ./cmd/rgserver
# my-company-domain/modules/revgeo/vendor/github.com/bvinc/go-sqlite-lite/sqlite3
sqlite3.c:182866:7: warning: array index 2 is past the end of the array (which contains 2 elements) [-Warray-bounds]
sqlite3.c:182442:3: note: array 'a' declared here
sqlite3.c:182867:7: warning: array index 3 is past the end of the array (which contains 2 elements) [-Warray-bounds]
sqlite3.c:182442:3: note: array 'a' declared here
sqlite3.c:182868:7: warning: array index 4 is past the end of the array (which contains 2 elements) [-Warray-bounds]
sqlite3.c:182442:3: note: array 'a' declared here
sqlite3.c:182869:7: warning: array index 5 is past the end of the array (which contains 2 elements) [-Warray-bounds]
sqlite3.c:182442:3: note: array 'a' declared here
sqlite3.c:182870:7: warning: array index 6 is past the end of the array (which contains 2 elements) [-Warray-bounds]
sqlite3.c:182442:3: note: array 'a' declared here
sqlite3.c:182871:7: warning: array index 7 is past the end of the array (which contains 2 elements) [-Warray-bounds]
sqlite3.c:182442:3: note: array 'a' declared here
# my-company-domain/modules/revgeo/cmd/rgserver
/usr/local/Cellar/go/1.11.1/libexec/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
ld: warning: ignoring file /var/folders/w_/2y9q2qdx3bnb521w5bb1kvj9j8bs6w/T/go-link-156734203/go.o, file was built for unsupported file format ( 0x7F 0x45 0x4C 0x46 0x02 0x01 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 ) which is not the architecture being linked (x86_64): /var/folders/w_/2y9q2qdx3bnb521w5bb1kvj9j8bs6w/T/go-link-156734203/go.o
Undefined symbols for architecture x86_64:
  "__cgo_topofstack", referenced from:
      __cgo_18049202ccd9_C2func_getnameinfo in 000002.o
      __cgo_18049202ccd9_Cfunc_getnameinfo in 000002.o
      __cgo_18049202ccd9_C2func_getaddrinfo in 000004.o
      __cgo_18049202ccd9_Cfunc_gai_strerror in 000004.o
      __cgo_18049202ccd9_Cfunc_getaddrinfo in 000004.o
      __cgo_592b550a3e6d_Cfunc_sqlite3_backup_finish in 000018.o
      __cgo_592b550a3e6d_Cfunc_sqlite3_backup_init in 000018.o
      ...
  "__cgoexp_592b550a3e6d_go_busy_handler", referenced from:
      _go_busy_handler in 000017.o
  "__cgoexp_592b550a3e6d_go_commit_hook", referenced from:
      _go_commit_hook in 000017.o
  "__cgoexp_592b550a3e6d_go_rollback_hook", referenced from:
      _go_rollback_hook in 000017.o
  "__cgoexp_592b550a3e6d_go_update_hook", referenced from:
      _go_update_hook in 000017.o
  "__cgoexp_592b550a3e6d_strm_r_tramp", referenced from:
      _strm_r_tramp in 000017.o
  "__cgoexp_592b550a3e6d_strm_w_tramp", referenced from:
      _strm_w_tramp in 000017.o
  "__cgoexp_592b550a3e6d_xapply_conflict_tramp", referenced from:
      _xapply_conflict_tramp in 000017.o
  "__cgoexp_592b550a3e6d_xapply_filter_tramp", referenced from:
      _xapply_filter_tramp in 000017.o
  "_crosscall2", referenced from:
      _strm_w_tramp in 000017.o
      _strm_r_tramp in 000017.o
      _xapply_filter_tramp in 000017.o
      _xapply_conflict_tramp in 000017.o
      _go_busy_handler in 000017.o
      _go_commit_hook in 000017.o
      _go_rollback_hook in 000017.o
      ...
  "_main", referenced from:
     implicit entry/start for main executable
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
sudo-suhas commented 6 years ago

Should I try the instructions in https://github.com/mattn/go-sqlite3/issues/372#issuecomment-396863368?

lvvthinh commented 6 years ago

Same with me. Tried many solutions even this. Any help?

$ go env 
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/thinhle/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/thinhle/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11.2/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
sudo-suhas commented 6 years ago

@lvvthinh did you try xgo (see https://github.com/bvinc/go-sqlite-lite/issues/10#issue-374246653)

pedromorgan commented 5 years ago

So this is a show stopper for me. Cant get it cross compile (but a golang newbie ish also)

coolaj86 commented 5 years ago

Success

I had some success.

Pre-reqs:

mkdir -p github.com/bvinc
pushd github.com/bvinc
git clone https://github.com/bvinc/go-sqlite-lite.git
pushd go-sqlite-lite
echo "module github.com/bvinc/go-sqlite-lite" > go.mod
pushd examples/basic

Native compile:

env GOOS=darwin GOARCH=amd64 CGO_ENABLED=1 go build .

MacOS Host Compiling For Windows

brew install mingw-w64
env GOOS=windows GOARCH=386 CGO_ENABLED=1 CC=/usr/local/bin/i686-w64-mingw32-gcc CXX=/usr/local/bin/i686-w64-mingw32-g++ go build .
env GOOS=windows GOARCH=amd64 CGO_ENABLED=1 CC=/usr/local/bin/x86_64-w64-mingw32-gcc CXX=/usr/local/bin/x86_64-w64-mingw32-g++ go build .

MacOS Host Compiling For Linux

brew install FiloSottile/musl-cross/musl-cross
env GOOS=linux GOARCH=amd64 CGO_ENABLED=1 CC=x86_64-linux-musl-gcc CXX=x86_64-linux-musl-g++ go build .

Others

If it's that easy to get this to cross compile for Windows and Linux from MacOS, I have no worries about being able to do it from Linux - which is where it will all end up happening in "production". You can't automate Mac or Windows anyway, so no need to fret over trying to get it to go every which way, right?