[Release][Integration] Integration job fails on macOS conda AMD64 with SIGSEGV

raulcd commented 1 week ago

Describe the bug, including details regarding any error messages, version, and platform.

The verify-rc-source-integration-macos-conda-amd64 job has been failing since the 9th of October with a SIGSEGV:

2024-10-11T10:26:10.4002600Z Testing file flight_sql:extension
2024-10-11T10:26:10.4003710Z ======================================================================
2024-10-11T10:26:10.6458290Z WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
2024-10-11T10:26:10.8922820Z #
2024-10-11T10:26:10.8924670Z # A fatal error has been detected by the Java Runtime Environment:
2024-10-11T10:26:10.8925720Z #
2024-10-11T10:26:10.8926820Z #  SIGSEGV (0xb) at pc=0x000000010b043c3e, pid=27510, tid=259
2024-10-11T10:26:10.8928120Z #
2024-10-11T10:26:10.8929920Z # JRE version: OpenJDK Runtime Environment Zulu22.30+13-CA (22.0.1+8) (build 22.0.1+8)
2024-10-11T10:26:10.8933740Z # Java VM: OpenJDK 64-Bit Server VM Zulu22.30+13-CA (22.0.1+8, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-amd64)
2024-10-11T10:26:10.8935420Z # Problematic frame:
2024-10-11T10:26:10.8936190Z # C  [python3.13+0x2afc3e]  _PyObject_GC_New+0x13e
2024-10-11T10:26:10.8937530Z #
2024-10-11T10:26:10.8940090Z # No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
2024-10-11T10:26:10.8942890Z #
2024-10-11T10:26:10.8944600Z # An error report file with more information is saved as:
2024-10-11T10:26:10.8946530Z # /Users/runner/work/crossbow/crossbow/arrow/hs_err_pid27510.log
2024-10-11T10:26:10.9330250Z #
2024-10-11T10:26:10.9333700Z # If you would like to submit a bug report, please visit:
2024-10-11T10:26:10.9334690Z #   http://www.azul.com/support/
2024-10-11T10:26:10.9335900Z # The crash happened outside the Java Virtual Machine in native code.
2024-10-11T10:26:10.9337540Z # See problematic frame for where to report the bug.
2024-10-11T10:26:10.9339040Z #
2024-10-11T10:26:10.9373180Z arrow/dev/release/verify-release-candidate.sh: line 899: 27510 Abort trap: 6           LD_LIBRARY_PATH=$ARROW_CPP_EXE_PATH:$LD_LIBRARY_PATH archery integration --run-ipc --run-flight --run-c-data --with-cpp=${TEST_INTEGRATION_CPP} --with-java=${TEST_INTEGRATION_JAVA} --with-js=${TEST_INTEGRATION_JS} $INTEGRATION_TEST_ARGS
2024-10-11T10:26:10.9381060Z Failed to verify release candidate. See /var/folders/4b/7k50gk0j4f5bjk3799wdt8nw0000gn/T/arrow-HEAD.XXXXX.B1wU5Qbh for details.
2024-10-11T10:26:10.9625520Z ##[error]Process completed with exit code 134.

This is also failing on the maintenance branch.

The commits introduced are not related https://github.com/apache/arrow/compare/64891d1d176dd45f3fae574e1bcfac6fee197e5f...1c601a573e1e34b87a8e2e64db3c9786f3ca2bcd It seems to have to do with Python 3.13. The last successful job used Python 3.12.

Component(s)

Integration, Release

raulcd commented 1 week ago

@pitrou @lidavidm @kou any idea here? I am pinning Python 3.12 temporarily for conda on macOS, maybe that's good enough to release but we should investigate this one.

kou commented 1 week ago

I don't know why the Java integration test runner uses Python. @lidavidm Do you know why?

lidavidm commented 1 week ago

Hmm, that's odd. The Python bit should only be to spawn processes, so why is Java seemingly calling into Python?

And this is macOS amd64, so it'll be hard to reproduce...

pitrou commented 1 week ago

Yeah, that's odd. It's almost as if the fork+exec in subprocess hadn't fully unloaded Python (which sounds outlandish).

amoeba commented 1 week ago

I was able to reproduce this on my own crossbow repo after splicing in a tmate step and running the verification manually. Patch below. I'm poking around now.

github.macos.yml.patch

```patch diff --git a/dev/tasks/verify-rc/github.macos.yml b/dev/tasks/verify-rc/github.macos.yml index 4bc3fff71..82af4a02a 100644 --- a/dev/tasks/verify-rc/github.macos.yml +++ b/dev/tasks/verify-rc/github.macos.yml @@ -59,6 +59,9 @@ jobs: with: node-version: '20' + - name: Setup tmate session + uses: mxschmitt/action-tmate@v3 + - name: Run verification shell: bash env: ```

amoeba commented 1 week ago

Here's the output of hs_err_pid49681.log:

hs_err_pid49681.log

The above makes it look like JPype is involved after all?

pitrou commented 1 week ago

Ok, so it's the archery integration process that crashed, unlike what we've been assuming above. Flight SQL is probably not involved at all. The crash occurs after "flight_sql:extension" has finished, when initializing the C Data Interface testing bridge with JPype.

~~I'm curious: did your crash occur deterministically in the same place as in https://github.com/ursacomputing/crossbow/actions/runs/11289830425/job/31400455755 ?~~

amoeba commented 1 week ago

I'll start a new run with PYTHONFAULTHANDLER=1 on.

The failure I reproduced crashed in the C Data Interface group whereas the original report crashed in Flight.

amoeba commented 1 week ago

Here's what I get with PYTHONFAULTHANDLER=1:

##########################################################
C Data Interface: C++ exporting, Java importing
##########################################################
WARNING: Unknown module: org.apache.arrow.memory.core specified to --add-opens
Fatal Python error: Segmentation fault

Current thread 0x000000010a578600 (most recent call first):
  File "/private/var/folders/t5/14gs56bj7djcylkj26c908n40000gn/T/arrow-HEAD.XXXXX.FsfNaVDF/mambaforge/envs/conda-source/lib/python3.13/site-packages/jpype/_jclass.py", line 147 in _jclassPost
  File "/private/var/folders/t5/14gs56bj7djcylkj26c908n40000gn/T/arrow-HEAD.XXXXX.FsfNaVDF/mambaforge/envs/conda-source/lib/python3.13/site-packages/jpype/_jclass.py", line 99 in __new__
  File "/private/var/folders/t5/14gs56bj7djcylkj26c908n40000gn/T/arrow-HEAD.XXXXX.FsfNaVDF/mambaforge/envs/conda-source/lib/python3.13/site-packages/jpype/_core.py", line 275 in initializeResources
  File "/private/var/folders/t5/14gs56bj7djcylkj26c908n40000gn/T/arrow-HEAD.XXXXX.FsfNaVDF/mambaforge/envs/conda-source/lib/python3.13/site-packages/jpype/_core.py", line 250 in startJVM
  File "/Users/runner/work/crossbow/crossbow/arrow/dev/archery/archery/integration/tester_java.py", line 92 in setup_jpype
  File "/Users/runner/work/crossbow/crossbow/arrow/dev/archery/archery/integration/tester_java.py", line 108 in __init__
  File "/Users/runner/work/crossbow/crossbow/arrow/dev/archery/archery/integration/tester_java.py", line 356 in make_c_data_importer
  File "/Users/runner/work/crossbow/crossbow/arrow/dev/archery/archery/integration/runner.py", line 427 in _compare_c_data_implementations
  File "/Users/runner/work/crossbow/crossbow/arrow/dev/archery/archery/integration/runner.py", line 130 in run_c_data
  File "/Users/runner/work/crossbow/crossbow/arrow/dev/archery/archery/integration/runner.py", line 658 in run_all_tests
  File "/Users/runner/work/crossbow/crossbow/arrow/dev/archery/archery/cli.py", line 810 in integration
  File "/private/var/folders/t5/14gs56bj7djcylkj26c908n40000gn/T/arrow-HEAD.XXXXX.FsfNaVDF/mambaforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 783 in invoke
  File "/private/var/folders/t5/14gs56bj7djcylkj26c908n40000gn/T/arrow-HEAD.XXXXX.FsfNaVDF/mambaforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 1434 in invoke
  File "/private/var/folders/t5/14gs56bj7djcylkj26c908n40000gn/T/arrow-HEAD.XXXXX.FsfNaVDF/mambaforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 1688 in invoke
  File "/private/var/folders/t5/14gs56bj7djcylkj26c908n40000gn/T/arrow-HEAD.XXXXX.FsfNaVDF/mambaforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 1078 in main
  File "/private/var/folders/t5/14gs56bj7djcylkj26c908n40000gn/T/arrow-HEAD.XXXXX.FsfNaVDF/mambaforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 1157 in __call__
  File "/private/var/folders/t5/14gs56bj7djcylkj26c908n40000gn/T/arrow-HEAD.XXXXX.FsfNaVDF/mambaforge/envs/conda-source/bin/archery", line 8 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, _cffi_backend, _jpype (total: 13)
arrow/dev/release/verify-release-candidate.sh: line 985: 34642 Segmentation fault: 11  LD_LIBRARY_PATH=$ARROW_CPP_EXE_PATH:$LD_LIBRARY_PATH archery integration --run-ipc --run-flight --run-c-data --with-cpp=${TEST_INTEGRATION_CPP} --with-java=${TEST_INTEGRATION_JAVA} --with-js=${TEST_INTEGRATION_JS} --with-go=${TEST_INTEGRATION_GO} $INTEGRATION_TEST_ARGS
Failed to verify release candidate. See /var/folders/t5/14gs56bj7djcylkj26c908n40000gn/T/arrow-HEAD.XXXXX.FsfNaVDF for details.

Edit: PS I'll be offline until later my time.

kou commented 1 week ago

Wow. So this is the same problem as #44389. Let's merge #44388. We can use Python 3.13 or later again after JPype supports Python 3.13 or later.

kou commented 1 week ago

Issue resolved by pull request 44388 https://github.com/apache/arrow/pull/44388

apache / arrow

[Release][Integration] Integration job fails on macOS conda AMD64 with SIGSEGV #44386

Describe the bug, including details regarding any error messages, version, and platform.

Component(s)