Many zlib tests are failing on armhf with system zlib

mbakke commented 4 years ago

Version: 10.19.0
Platform: armhf-linux
Subsystem: tests

What steps will reproduce the bug?

Build Node on ARMv7 with --shared-zlib, then run make test-ci-js.

How often does it reproduce? Is there a required condition?

Every time.

What is the expected behavior?

Tests succeed.

What do you see instead?

ot ok 2163 parallel/test-zlib
  ---
  duration_ms: 1.641
  severity: crashed
  exitcode: -7
  stack: |-
  ...
not ok 2164 parallel/test-zlib-brotli
  ---
  duration_ms: 1.441
  severity: crashed
  exitcode: -7
  stack: |-
  ...
not ok 2165 parallel/test-zlib-brotli-flush
  ---
  duration_ms: 1.636
  severity: crashed
  exitcode: -7
  stack: |-
  ...
not ok 2166 parallel/test-zlib-brotli-from-brotli
  ---
  duration_ms: 1.317
  severity: crashed
  exitcode: -7
  stack: |-
  ...
not ok 2167 parallel/test-zlib-brotli-from-string
  ---
  duration_ms: 1.418
  severity: crashed
  exitcode: -7
  stack: |-
  ...
not ok 2174 parallel/test-zlib-convenience-methods
  ---
  duration_ms: 1.524
  severity: crashed
  exitcode: -7
  stack: |-
  ...
not ok 2199 parallel/test-zlib-random-byte-pipes
  ---
  duration_ms: 1.445
  severity: crashed
  exitcode: -7
  stack: |-
  ...
not ok 2205 parallel/test-zlib-write-after-flush
  ---
  duration_ms: 1.315
  severity: crashed
  exitcode: -7
  stack: |-
  ...

Additional information

mbakke commented 4 years ago

Actually, these tests fail with the bundled zlib too. Any idea what I'm doing wrong?

bnoordhuis commented 4 years ago

Exit code -7 means the processes were terminated by a SIGBUS signal. Can you try this?

$ gdb --args out/Release/node test/parallel/test-zlib.js
> run
# wait for the crash, then post the output of:
> info registers
> thread apply all backtrace full

If you can test with the debug build, that'd be even better: make -j4 -C out BUILDTYPE=Debug and change out/Release/node to out/Debug/node.

mbakke commented 4 years ago

I don't have ARM hardware at hand, but did a debug build using Qemus user-mode emulation and binfmt_misc. Lo and behold, out/Debug/node test/parallel/test-zlib.js succeeds!

I'm fairly certain these tests failed in Qemu with the Release build type before, as some of our CI servers are using user-mode emulation. So I wonder if the failure could be related to compiler optimizations? This is using GCC 7.4.0 and glibc 2.29 FWIW.

Will conduct some more tests on the CI with different optimizations and compilers, and try to get access to proper hardware for debugging. Any other tips in the mean time welcome.

bnoordhuis commented 4 years ago

So I wonder if the failure could be related to compiler optimizations?

It's possible but of course it could also be a bug in qemu.

FWIW, our CI tests on real ARM hardware and doesn't have the issues you're seeing. I'm not sure if we build with the exact same version of gcc though.

nodejs / node