search

prometheus / node_exporter

Exporter for machine metrics
https://prometheus.io/
Apache License 2.0
11.3k stars 2.38k forks source link

Cross compiled netbsd binaries segfault in netbsd 7.0 VM #402

Open discordianfish opened 7 years ago

discordianfish commented 7 years ago

Looking in ways to test the node-exporter across platforms and started with netbsd on virtualbox/vagrant, using this Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "netbsd/NetBSD-7.0"
  config.vm.synced_folder ".", "/usr/src/",
    :nfs => true
end

Ran promu crossbuild -p netbsd/amd64 on the host to cross compile the binary. In the VM though, the binary segfaults:

$ ./.build/netbsd-amd64/node_exporter    
[1]   Segmentation fault (core dumped) ./.build/netbsd-amd64/node_exporter
SuperQ commented 6 years ago

Is this still an issue?

discordianfish commented 6 years ago

@SuperQ Yes it is, just tried it again following the same process. This time though I got this warning when building/linking:

# github.com/prometheus/node_exporter
/tmp/go-link-362028283/000007.o: In function `_cgo_52b6daafb6d7_C2func_getaddrinfo':
/tmp/go-build/cgo-gcc-prolog:46: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

Now that I actually tried, the official releases also crash:

$ pwd
/usr/src/node_exporter-0.17.0-rc.0.netbsd-amd64
$ ./node_exporter
[1]   Segmentation fault (core dumped) ./node_exporter

Do we know anyone actually using these? Or have they been broke for everyone since forever? :)

als-git commented 2 years ago

And this is still true with the latest binary release for NetBSD, both i386 and amd64. Tested on NetBSD 9.3 amd64:

node_exporter-1.3.1.netbsd-386/node_exporter immediately appears to exit when invoked, and ktrace doesn't tell a happy story:

13280 1 ktrace EMUL "netbsd" 13280 1 ktrace CALL execve(0x7f7fff542a97,0x7f7fff5424f0,0x7f7fff542500) 13280 1 ktrace NAMI "/tmp/build/./node_exporter-1.3.1.netbsd-386/node_exporter" 13280 1 node_exporter NAMI "/emul/linux32" 13280 13280 node_exporter EMUL "linux32" 13280 13280 node_exporter RET execve -1 errno -2 No such file or directory 13280 13280 node_exporter CALL brk(0) 13280 13280 node_exporter RET brk 145002496/0x8a49000 13280 13280 node_exporter CALL brk(0x8a49800) 13280 13280 node_exporter RET brk 145006592/0x8a4a000 13280 13280 node_exporter CALL set_thread_area(0xbffffc10) 13280 13280 node_exporter RET set_thread_area 0 13280 13280 node_exporter CALL uname(0xbffffa9a) 13280 13280 node_exporter RET uname 0 13280 13280 node_exporter CALL set_tid_address(0x8a49368) 13280 13280 node_exporter RET set_tid_address 13280/0x33e0 13280 13280 node_exporter CALL set_robust_list(0x8a49370,0xc) 13280 13280 node_exporter RET set_robust_list 0 13280 13280 node_exporter CALL rt_sigaction(SIGRT0,0xbffff824,0,8) 13280 13280 node_exporter RET rt_sigaction 0 13280 13280 node_exporter CALL rt_sigaction(SIGRT1,0xbffff824,0,8) 13280 13280 node_exporter RET rt_sigaction 0 13280 13280 node_exporter CALL rt_sigprocmask(1,0xbffff994,0,8) 13280 13280 node_exporter RET rt_sigprocmask 0 13280 13280 node_exporter CALL ugetrlimit(3,0xbffff988) 13280 13280 node_exporter RET ugetrlimit 0 13280 13280 node_exporter CALL uname(0xbffffa1c) 13280 13280 node_exporter RET uname 0 13280 13280 node_exporter CALL netbsd32_readlink(0x87071c3,0xbfffeb90,0x1000) 13280 13280 node_exporter NAMI "/proc/self/exe" 13280 13280 node_exporter RET netbsd32_readlink 57/0x39 13280 13280 node_exporter CALL brk(0x8a6b000) 13280 13280 node_exporter RET brk 145141760/0x8a6b000 13280 13280 node_exporter CALL brk(0x8a6c000) 13280 13280 node_exporter RET brk 145145856/0x8a6c000 13280 13280 node_exporter CALL netbsd32_access(0x8706f9b,0) 13280 13280 node_exporter NAMI "/etc/ld.so.nohwcap" 13280 13280 node_exporter RET netbsd32_access -1 errno -2 No such file or directory 13280 13280 node_exporter CALL mprotect(0x89d5000,0x2000,PROT_READ) 13280 13280 node_exporter RET mprotect 0 13280 13280 node_exporter CALL getegid 13280 13280 node_exporter RET getegid 0 13280 13280 node_exporter CALL fstat64(0x8a3a12c,0xfffffffc) 13280 13280 node_exporter RET fstat64 -1 errno -9 Bad file descriptor 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL [430] 13280 13280 node_exporter RET [430] -1 errno -38 Too many processes 13280 13280 node_exporter CALL [430] 13280 13280 node_exporter RET [430] -1 errno -38 Too many processes 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address 13280 13280 node_exporter CALL netbsd32_write(0,2,1) 13280 13280 node_exporter RET netbsd32_write -1 errno -14 Bad address

ldd doesn't like it much: ldd: ./node_exporter-1.3.1.netbsd-386/node_exporter: not dynamically linked

and file thinks its for the wrong OS: ./node_exporter-1.3.1.netbsd-386/node_exporter: ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), statically linked, BuildID[sha1]=b35220fa1b5fc444d563c79685b1f6647d0a595e, for GNU/Linux 3.2.0, not stripped

The amd64 binary immediately segfaults: ./node_exporter-1.3.1.netbsd-amd64/node_exporter Segmentation fault (core dumped)

complete ktrace from that: 22821 1 ktrace EMUL "netbsd" 22821 1 ktrace CALL execve(0x7f7fffd9dc37,0x7f7fffd9d690,0x7f7fffd9d6a0) 22821 1 ktrace NAMI "/tmp/build/node_exporter-1.3.1.netbsd-amd64/node_exporter" 22821 1 node_exporter NAMI "/emul/linux" 22821 22821 node_exporter EMUL "linux" 22821 22821 node_exporter RET execve -1 errno -2 No such file or directory 22821 22821 node_exporter CALL brk(0) 22821 22821 node_exporter RET brk 16773120/0xfff000 22821 22821 node_exporter CALL brk(0xfffc80) 22821 22821 node_exporter RET brk 16777216/0x1000000 22821 22821 node_exporter CALL arch_prctl(0x1002,0xfff340) 22821 22821 node_exporter RET arch_prctl 0 22821 22821 node_exporter CALL uname(0x7f7fffffe930) 22821 22821 node_exporter RET uname 0 22821 22821 node_exporter CALL set_tid_address(0xfff610) 22821 22821 node_exporter RET set_tid_address 22821/0x5925 22821 22821 node_exporter CALL set_robust_list(0xfff620,0x18) 22821 22821 node_exporter RET set_robust_list 0 22821 22821 node_exporter CALL rt_sigaction(SIGRT0,0x7f7fffffe8b0,0,8) 22821 22821 node_exporter RET rt_sigaction 0 22821 22821 node_exporter CALL rt_sigaction(SIGRT1,0x7f7fffffe8b0,0,8) 22821 22821 node_exporter RET rt_sigaction 0 22821 22821 node_exporter CALL rt_sigprocmask(1,0x7f7fffffea28,0,8) 22821 22821 node_exporter RET rt_sigprocmask 0 22821 22821 node_exporter CALL #302 (unimplemented prlimit64) 22821 22821 node_exporter RET #302 (unimplemented prlimit64) -1 errno -38 Too many processes 22821 22821 node_exporter CALL readlink(0xc1baa3,0x7f7fffffda10,0x1000) 22821 22821 node_exporter NAMI "/proc/self/exe" 22821 22821 node_exporter RET readlink 57/0x39 22821 22821 node_exporter CALL brk(0x1021000) 22821 22821 node_exporter RET brk 16912384/0x1021000 22821 22821 node_exporter CALL mprotect(0xf45000,0x4000,PROT_READ) 22821 22821 node_exporter RET mprotect 0 22821 22821 node_exporter CALL futex(0x7f7fffffea80,2,0x7f7fffffea44,0x7f7fffffea48,0,0) 22821 22821 node_exporter RET futex -1 errno -38 Too many processes 22821 22821 node_exporter CALL removexattr(0,0x40000) 22821 22821 node_exporter RET removexattr -1 errno -95 Operation not supported 22821 22821 node_exporter PSIG SIGSEGV SIG_DFL: code=SEGV_MAPERR, addr=0xffffffffffffffa1, trap=14) 22821 22821 node_exporter NAMI "node_exporter.core"

ldd thinks it's the wrong kind of ELF binary: ldd: ./node_exporter-1.3.1.netbsd-amd64/node_exporter: invalid ELF class 2; expected 1

and file agrees: ./node_exporter-1.3.1.netbsd-amd64/node_exporter: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=430e02e0f1d04d30ed7944d4c9ff9cefc3893f74, for GNU/Linux 3.2.0, not stripped

Meanwhile, a natively built (from pkgsrc, on the same machine) node_exporter (1.0.1 so yes, old) binary gives these reports: ldd: /usr/pkg/bin/node_exporter: -lpthread.1 => /usr/lib/libpthread.so.1 -lc.12 => /usr/lib/libc.so.12

file: /usr/pkg/bin/node_exporter: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /usr/libexec/ld.elf_so, for NetBSD 9.3, Go BuildID=MLh9kWX33iIg4OYebIo4/WDlI__VFC1hvJzW6yzhR/8AT4Mvlcz89FR6w8bQVK/gBPRJdq27xIz7v9Av-h5, stripped

So in short, it looks whatever is being used for crossbuilding for the NetBSD target on (presumably) Linux host is horribly broken and does not produce working executables.

Would it be possible to actually native build them, given that crossbuilding clearly doesn't work?

discordianfish commented 2 years ago

Crossbuilding is implemented in https://github.com/prometheus/promu I'd be also up for native building but we need to figure out how to automate this etc etc. Just none of the maintainers is using freebsd. But if you want to look into either fixing promu or look into how to build natively on freebsd in circle ci, I'm happy to assist.