voltrondata-labs / arrow-benchmarks-ci

Benchmarks CI for Apache Arrow project
MIT License
0 stars 5 forks source link

benchmark pipelines: `cpp-micro` started failing with SIGABRT #160

Closed jgehrcke closed 11 months ago

jgehrcke commented 11 months ago

apache-arrow/arrow-bci-benchmark-on-arm64-t4g-linux-compute and apache-arrow/arrow-bci-benchmark-on-test-mac-arm started failing recently:

Screenshot from 2023-08-10 15-06-35

FAILED C++ cpp-micro 1
...
subprocess.CalledProcessError: Command '['archery', 'benchmark', 'run', '--output', PosixPath('/tmp/tmpfwvrnp6a'), '--repetitions', '1']' returned non-zero exit status 1.
...
  File "/var/lib/buildkite-agent/builds/aws-arm64-t4g-linux-compute-i-066058a7b23bc0795-1/apache-arrow/arrow-bci-benchmark-on-arm64-t4g-linux-compute/buildkite/benchmark/run.py", line 695, in run_all_benchmark_groups
    raise Exception("Build has failed benchmarks.")
Exception: Build has failed benchmarks.
...

a bit more raw error from one of the various failed builds:

subprocess.CalledProcessError: Command '['/var/folders/d9/n0lbzbf16nvgt7b83xhcssm80000gn/T/arrow-archery-n7qhlgby/WORKSPACE/build/release/arrow-compute-scalar-round-benchmark', '--benchmark_repetitions=1', '--benchmark_out=/var/folders/d9/n0lbzbf16nvgt7b83xhcssm80000gn/T/tmpjz6lomkv', '--benchmark_out_format=json']' 

died with <Signals.SIGABRT: 6>.

Interesting:

abort() is usually called by library functions which detect an internal error or some seriously broken constraint. For example malloc() will call abort() if its internal structures are damaged by a heap overflow.

process stderr:

Running /var/folders/d9/n0lbzbf16nvgt7b83xhcssm80000gn/T/arrow-archery-n7qhlgby/WORKSPACE/build/release/arrow-compute-scalar-round-benchmark

Run on (8 X 23.9998 MHz CPU s)

CPU Caches:

  L1 Data 64 KiB

  L1 Instruction 128 KiB

  L2 Unified 4096 KiB (x8)

Load Average: 1.35, 1.42, 2.45

-- Arrow Fatal Error --

Invalid: Rounding to -5 digits is out of range for type int16
austin3dickey commented 11 months ago

This has not happened for days.

image