Golang 1.16.4 image Undefined symbols

[leonjza]

I use this repositories docker images to crossbuild gowitness with cgo enabled for Linux, macOS & Windows. When building for macOS using the 1.15.10-darwin image, the build completes successfully as expected. However, using the 1.16.4-darwin image, the build eventually fails with:

problem

$ docker run --rm -v $(pwd):/go/src/github.com/sensepost/gowitness -w /go/src/github.com/sensepost/gowitness -e CGO_ENABLED=1 docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-darwin --build-cmd "make darwin" -p "darwin/amd64"

[ ... snip ... ]

go: downloading github.com/chromedp/sysutil v1.0.0
go: downloading github.com/josharian/intern v1.0.0
go: downloading github.com/gobwas/pool v0.2.1
go: downloading github.com/gobwas/httphead v0.1.0
# github.com/sensepost/gowitness
/usr/local/go/pkg/tool/linux_amd64/link: running o64-clang failed: exit status 1
Undefined symbols for architecture x86_64:
  "_clock_gettime", referenced from:
      _runtime.walltime_trampoline in go.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Makefile:30: recipe for target 'darwin' failed
make: *** [darwin] Error 2
Error: failed building for darwin/amd64: exit status 2
failed building for darwin/amd64: exit status 2
make: *** [darwin-release] Error 1

replication steps

• clone gowitness with https://github.com/sensepost/gowitness.git
• (optional when on macOS) build the project with make or go build -o gowitness main.go and see it compile fine
• build the project with using a golang-crossbuild image and observe the error with: docker run --rm -v $(pwd):/go/src/github.com/sensepost/gowitness -w /go/src/github.com/sensepost/gowitness -e CGO_ENABLED=1 docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-darwin --build-cmd "make darwin" -p "darwin/amd64"

[kuisathaverat]

I have made a pretty basic test and it compiles a basic C source code, I will check gowitness particulars to see if I see something.

#include <stdio.h>
int main() {
   printf("Hello, World!");
   return 0;
}

docker run --rm -v $(pwd):/app -w /app -e CGO_ENABLED=1 docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-darwin --build-cmd "o64-clang helloWorld.c -o helloWorld" -p "darwin/amd64"

> file helloWorld
a.out: Mach-O 64-bit executable x86_64

[kuisathaverat]

After, changing the optimization to 0 it works CGO_ENABLED=0, I am not sure it is related to the cross-compiler itself

> docker run --rm -v $(pwd):/go/src/github.com/sensepost/gowitness -w /go/src/github.com/sensepost/gowitness -e CGO_ENABLED=0 docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-darwin --build-cmd "go build -o gowitness main.go" -p "darwin/amd64"
>> Building using: cmd='go build -o gowitness main.go', env=[CC=o64-clang, CXX=o64-clang++, GOARCH=amd64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-amd64]
go: downloading github.com/olekukonko/tablewriter v0.0.5
go: downloading github.com/h2non/filetype v1.1.1
go: downloading github.com/remeh/sizedwaitgroup v1.0.0
go: downloading github.com/rs/zerolog v1.20.0
go: downloading github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749
go: downloading github.com/spf13/cobra v1.1.3
go: downloading github.com/tomsteele/go-nmap v0.0.0-20191202052157-3507e0b03523
go: downloading gorm.io/gorm v1.20.12
go: downloading github.com/mattn/go-runewidth v0.0.10
go: downloading github.com/chromedp/cdproto v0.0.0-20210508221054-d7cfa85db7d1
go: downloading github.com/chromedp/chromedp v0.7.2
go: downloading golang.org/x/net v0.0.0-20210224082022-3d97a244fca7
go: downloading gorm.io/driver/sqlite v1.1.4
go: downloading github.com/corona10/goimagehash v1.0.3
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/rivo/uniseg v0.2.0
go: downloading github.com/gobwas/ws v1.1.0-rc.5
go: downloading github.com/mailru/easyjson v0.7.7
go: downloading github.com/mattn/go-sqlite3 v1.14.6
go: downloading github.com/jinzhu/inflection v1.0.0
go: downloading github.com/jinzhu/now v1.1.1
go: downloading github.com/chromedp/sysutil v1.0.0
go: downloading github.com/gobwas/httphead v0.1.0
go: downloading github.com/gobwas/pool v0.2.1
go: downloading github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646
go: downloading github.com/josharian/intern v1.0.0

[leonjza]

Yeah, CGO_ENABLED=1 is needed for sqlite (src here).

[kuisathaverat]

For the make darwin command I have to edit the Makefile to change the line export CGO_ENABLED=0, after that it compiles

 docker run --rm -v $(pwd):/go/src/github.com/sensepost/gowitness -w /go/src/github.com/sensepost/gowitness -e CGO_ENABLED=0 docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-darwin --build-cmd "make darwin" -p "darwin/amd64"
>> Building using: cmd='make darwin', env=[CC=o64-clang, CXX=o64-clang++, GOARCH=amd64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-amd64]
GOOS=darwin GOARCH=amd64 go build -ldflags="-s -w -X=github.com/sensepost/gowitness/cmd.gitHash=0685be6 -X=github.com/sensepost/gowitness/cmd.goVer=go1.16.4_linux/amd64" -o 'build/gowitness-2.3.6-darwin-amd64'
go: downloading github.com/olekukonko/tablewriter v0.0.5
go: downloading github.com/remeh/sizedwaitgroup v1.0.0
go: downloading github.com/h2non/filetype v1.1.1
go: downloading github.com/rs/zerolog v1.20.0
go: downloading github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749
go: downloading github.com/spf13/cobra v1.1.3
go: downloading github.com/tomsteele/go-nmap v0.0.0-20191202052157-3507e0b03523
go: downloading gorm.io/gorm v1.20.12
go: downloading github.com/mattn/go-runewidth v0.0.10
go: downloading github.com/chromedp/cdproto v0.0.0-20210508221054-d7cfa85db7d1
go: downloading github.com/chromedp/chromedp v0.7.2
go: downloading golang.org/x/net v0.0.0-20210224082022-3d97a244fca7
go: downloading gorm.io/driver/sqlite v1.1.4
go: downloading github.com/corona10/goimagehash v1.0.3
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/rivo/uniseg v0.2.0
go: downloading github.com/gobwas/ws v1.1.0-rc.5
go: downloading github.com/mailru/easyjson v0.7.7
go: downloading github.com/mattn/go-sqlite3 v1.14.6
go: downloading github.com/jinzhu/inflection v1.0.0
go: downloading github.com/jinzhu/now v1.1.1
go: downloading github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646
go: downloading github.com/chromedp/sysutil v1.0.0
go: downloading github.com/gobwas/httphead v0.1.0
go: downloading github.com/gobwas/pool v0.2.1
go: downloading github.com/josharian/intern v1.0.0

[leonjza]

CGO needs to be enabled (which is why using this docker image is useful). It works fine (with CGO enabled) in the 1.15.x image but not the 1.16.x image. It will compile fine with CGO disabled, but sqlite will not function correctly. Without CGO enabled (either via the Makefile or -e), go-sqlite3 will not build the necessary native components, and instead stub methods. The successful compile (in both cases mentioned above with CGO disabled) will result in a runtime error stating that sqlite3 was not built correctly when you try and use it. In the case of gowitness, that looks as follows:

[leonjza]

I have reduced the example to replicate the error. Let's ignore gowitness from here.

Source (main.go):

package main

import (
    "database/sql"
    "log"

    _ "github.com/mattn/go-sqlite3"
)

type Table struct {
    Id   int
    Name string
}

func main() {
    db, err := sql.Open("sqlite3", "./sqlite.sqlite")
    if err != nil {
        log.Fatalf("Error: %v", err)
        return
    }
    defer db.Close()

    _, err = db.Exec("CREATE TABLE IF NOT EXISTS `c` (`id` INTEGER PRIMARY KEY AUTOINCREMENT, `name` VARCHAR(255) NOT NULL)")
    if err != nil {
        log.Fatalf("Error: %v", err)
        return
    }
    _, err = db.Exec("INSERT INTO c (name) VALUES ('test')")
    if err != nil {
        log.Fatalf("Error: %v", err)
        return
    }

    var qu Table
    q, err := db.Query("SELECT * FROM c")
    if err != nil {
        log.Fatalf("Error: %v", err)
        return
    }
    q.Next()
    if err = q.Scan(&qu.Id, &qu.Name); err != nil {
        log.Fatalf("Error: %v", err)
        return
    }

    log.Printf("Name is: %s\n", qu.Name)
}

Compile command:

docker run --rm -it -v $(pwd):/work -w /work -e CGO_ENABLED=1 docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-darwin --build-cmd "go build -o test main.go" -p "darwin/amd64"

Error:

❯ docker run --rm -it -v $(pwd):/work -w /work -e CGO_ENABLED=1 docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-darwin --build-cmd "go build -o test main.go" -p "darwin/amd64"
>> Building using: cmd='go build -o test main.go', env=[CC=o64-clang, CXX=o64-clang++, GOARCH=amd64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-amd64]
go: downloading github.com/mattn/go-sqlite3 v1.14.7
# command-line-arguments
/usr/local/go/pkg/tool/linux_amd64/link: running o64-clang failed: exit status 1
Undefined symbols for architecture x86_64:
  "_clock_gettime", referenced from:
      _runtime.walltime_trampoline in go.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Error: failed building for darwin/amd64: exit status 2
failed building for darwin/amd64: exit status 2

With CGO_ENABLED=0 it complies, but that defeats the purpose a bit ;)

❯ docker run --rm -it -v $(pwd):/work -w /work -e CGO_ENABLED=0 docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-darwin --build-cmd "go build -o test main.go" -p "darwin/amd64"
>> Building using: cmd='go build -o test main.go', env=[CC=o64-clang, CXX=o64-clang++, GOARCH=amd64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-amd64]
go: downloading github.com/mattn/go-sqlite3 v1.14.7
❯ ./test
2021/05/25 17:57:34 Error: Binary was compiled with 'CGO_ENABLED=0', go-sqlite3 requires cgo to work. This is a stub

[kuisathaverat]

I just found https://github.com/pocoproject/poco/issues/1453#issuecomment-254663587 and ring the bell, so I tried to compile the project with the latest MacOS SDK we have 10.14, it works. The debian9 image is the default and it uses MacOS SDK 10.11 the function that is not found looks like is on MacOS SDK 10.12 so it will not found it. We build several docker images based on Debian, the Debian 10 version is a little different has modern packages and in the Darwin case a newer MacOS SDK 10.14, the image is docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-darwin-debian10

>docker run --rm -v $(pwd):/go/src/github.com/sensepost/gowitness -w /go/src/github.com/sensepost/gowitness docker.elastic.co/beats-dev/golang-crossbuild:1.16.4-darwin-debian10 --build-cmd "make darwin" -p "darwin/amd64"

>> Building using: cmd='make darwin', env=[CC=o64-clang, CXX=o64-clang++, GOARCH=amd64, GOARM=, GOOS=darwin, PLATFORM_ID=darwin-amd64]
GOOS=darwin GOARCH=amd64 go build -ldflags="-s -w -X=github.com/sensepost/gowitness/cmd.gitHash=0685be6 -X=github.com/sensepost/gowitness/cmd.goVer=go1.16.4_linux/amd64" -o 'build/gowitness-2.3.6-darwin-amd64'
go: downloading github.com/h2non/filetype v1.1.1
go: downloading github.com/olekukonko/tablewriter v0.0.5
go: downloading github.com/remeh/sizedwaitgroup v1.0.0
go: downloading github.com/rs/zerolog v1.20.0
go: downloading github.com/shurcooL/httpfs v0.0.0-20190707220628-8d4bc4ba7749
go: downloading github.com/spf13/cobra v1.1.3
go: downloading github.com/tomsteele/go-nmap v0.0.0-20191202052157-3507e0b03523
go: downloading gorm.io/gorm v1.20.12
go: downloading github.com/mattn/go-runewidth v0.0.10
go: downloading github.com/chromedp/cdproto v0.0.0-20210508221054-d7cfa85db7d1
go: downloading github.com/chromedp/chromedp v0.7.2
go: downloading golang.org/x/net v0.0.0-20210224082022-3d97a244fca7
go: downloading gorm.io/driver/sqlite v1.1.4
go: downloading github.com/corona10/goimagehash v1.0.3
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/rivo/uniseg v0.2.0
go: downloading github.com/gobwas/ws v1.1.0-rc.5
go: downloading github.com/mailru/easyjson v0.7.7
go: downloading github.com/mattn/go-sqlite3 v1.14.6
go: downloading github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646
go: downloading github.com/jinzhu/inflection v1.0.0
go: downloading github.com/jinzhu/now v1.1.1
go: downloading github.com/chromedp/sysutil v1.0.0
go: downloading github.com/gobwas/pool v0.2.1
go: downloading github.com/gobwas/httphead v0.1.0
go: downloading github.com/josharian/intern v1.0.0

[leonjza]

Excellent thanks @kuisathaverat, the darwin-debian10 image for go 1.16.4 works perfectly! Sorry I missed that tag in the README.