Does not work with all Go binaries: consider supporting various file types and clarify the build warnings: refactor runtime detection logic

After waste my two days to figuring out how to deploy simple Go function to Netlify, I finally ended up this repository from Netlify support: https://answers.netlify.com/t/getting-no-functions-were-found-in-functions-directory-error-with-go-build/53107

I actually love the idea by using goblin library to detect which runtime we are currently using for the specified binary.

Description

I have a simple Go function that I want to deploy on Netlify. But netlify-cli always says: No Functions were found in functions directory. But wait. I 100% sure that function is there and something is wrong with build command. So I just dig dive from the netlify-cli to check what's going on under the hood. And finally ended up here.

What I have done

I cloned the repository, bumped goblin from 0.2 to 0.5.1 (latest), replaced ./tests/data/hello-world-go with mine. Then ran cargo test. And tadaaa. It failed.

Binary Details

$ file api

api: Mach-O 64-bit executable x86_64

$ exiftool api

ExifTool Version Number         : 12.27
File Name                       : api
Directory                       : .
File Size                       : 64 MiB
File Modification Date/Time     : 2022:03:10 21:47:19+03:00
File Access Date/Time           : 2022:03:10 23:00:52+03:00
File Inode Change Date/Time     : 2022:03:10 21:47:19+03:00
File Permissions                : -rwxr-xr-x
File Type                       : Mach-O executable
File Type Extension             :
MIME Type                       : application/octet-stream
CPU Architecture                : 64 bit
CPU Byte Order                  : Little endian
CPU Type                        : x86 64-bit
CPU Subtype                     : i386 (all) 64-bit
Object File Type                : Demand paged executable
Object Flags                    : (none)

$ uname -a

Darwin 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64 x86_64

$ go env

GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/furkan.turkal/Library/Caches/go-build"
GOENV="/Users/furkan.turkal/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/furkan.turkal/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/furkan.turkal/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/local/Cellar/go/1.17.6/libexec"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.17.6/libexec/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17.6"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/nq/vxjjn3311fg4q263qsxrghpcpzgp66/T/go-build3722978256=/tmp/go-build -gno-record-gcc-switches -fno-common"

P.S: I did not override any Go flags either in my build script or netlify.toml file.

Problem

This is why $ netlify build command does not see my function:

$ readelf -a api

readelf: Error: Not an ELF file - it has the wrong magic bytes at the start

$ readelf -x .note.go.buildid api

readelf: Error: Not an ELF file - it has the wrong magic bytes at the start

Because we enabled following features in Cargo.toml: features = ["alloc", "elf32", "elf64", "endian_fd"]

MY GO BINARY IS NOT AN ELF FILE??? Wow. My file type is: Mach-O executable

I enabled the following features, but couldn't make tests pass: "elf32", "elf64", "mach32", "mach64", "pe32", "pe64", "archive", "endian_fd". Probably I'm close but here's what I have done:

    let mach = match Mach::parse(data) {
        Ok(mach) => {
            match mach {
                Mach::Fat(f) => {
                }
                Mach::Binary(b) => {
                    let sym = match b.symbols {
                        None => {}
                        Some(s) => {
                            // TODO
                        }
                    };

                }
            }
        },
        _ => return Ok(None),
    };

CONCERNS

I am able to see my buildid my issuing following command:

$ go tool buildid api

Gn1rf6fgQXQfCuwWMc95/7fGmW1EHdtMgmbrEjMYu/Gc_UQp7DnnVjdzcbHCNj/rIEVguKbmGsWY9t4fadj

!!! Passing -trimpath -ldflags=-buildid= flag to compiler will break runtime detection. Some developer might pass this flag to get better reproducible builds.

CONSIDER PROVIDING BETTER UX FOR DEVELOPERS

Everything began with the following issue:

No Functions were found in functions directory

Please clarify this message if we don't support given binary' file type:

We found <foo> binary in your functions directory. But file type is <bar>. We currently support following types: <baz, qux, etc>

Consider rebuild your binary in different format.

Bla bla bla...

Ref: https://github.com/netlify/go-functions-example/issues/2

Additional Notes

Please clarify what kind of file types Netlify support, what is the right way to build Go binaries on the documentation: https://docs.netlify.com/functions/build-with-go/

I filed this issue because I'm someone who enjoys investigating strange things and who likes to give feedback. I'm not sure who to ping here, sorry in advance if I'm pinging wrong maintainers:

elf-cam: @calavera

zip-it-and-ship-it:detect_runtime @eduardoboucas

build:core_steps @ehmicky

Feel free to create cross issues to link this one to provide better UX for developers and customers.

Thanks for reading. I'm really exhausted, and I think I deserve a good sleep now.

netlify / binary-info

Does not work with all Go binaries: consider supporting various file types and clarify the build warnings: refactor runtime detection logic #5