[Java][Integration][Release] verify-rc-source-integration-linux-conda-latest-amd64 fails with MemoryError

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

The verify-rc-source-integration-linux-conda-latest-amd64 is failing consistently with:

##########################################################
  C Data Interface: C++ exporting, Java importing
  ##########################################################
Traceback (most recent call last):
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/bin/archery", line 8, in <module>
    sys.exit(archery())
             ~~~~~~~^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/arrow/dev/archery/archery/cli.py", line 872, in integration
    run_all_tests(**args)
    ~~~~~~~~~~~~~^^^^^^^^
  File "/arrow/dev/archery/archery/integration/runner.py", line 713, in run_all_tests
    runner.run_c_data()
    ~~~~~~~~~~~~~~~~~^^
  File "/arrow/dev/archery/archery/integration/runner.py", line 148, in run_c_data
    self._compare_c_data_implementations(producer, consumer)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/arrow/dev/archery/archery/integration/runner.py", line 461, in _compare_c_data_implementations
    with consumer.make_c_data_importer() as importer:
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/arrow/dev/archery/archery/integration/tester_java.py", line 356, in make_c_data_importer
    return JavaCDataImporter(self.debug, self.args)
  File "/arrow/dev/archery/archery/integration/tester_java.py", line 108, in __init__
    setup_jpype()
    ~~~~~~~~~~~^^
  File "/arrow/dev/archery/archery/integration/tester_java.py", line 92, in setup_jpype
    jpype.startJVM(jpype.getDefaultJVMPath(),
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
                   "-Djava.class.path=" + jar_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
                   "-Xrs",
                   ^^^^^^^
                   *_JAVA_OPTS)
                   ^^^^^^^^^^^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/jpype/_core.py", line 250, in startJVM
    initializeResources()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/jpype/_core.py", line 275, in initializeResources
    _jpype._java_lang_String = _jpype.JClass("java.lang.String")
                               ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/jpype/_jclass.py", line 99, in __new__
    return _jpype._getClass(jc)
           ~~~~~~~~~~~~~~~~^^^^
  File "/tmp/arrow-HEAD.LSZ9o/miniforge/envs/conda-source/lib/python3.13/site-packages/jpype/_jclass.py", line 147, in _jclassPost
    for cls in res.class_.getDeclaredClasses():
               ^^^^^^^^^^
MemoryError
Failed to verify release candidate. See /tmp/arrow-HEAD.LSZ9o for details.

This is also failing on the maintenance branch for 18.0.0

Component(s)

Integration, Java, Release

[lidavidm]

This may also be a Python 3.13-related issue. https://github.com/jpype-project/jpype/issues/1204

[lidavidm]

While that issue is closed, I don't see a corresponding commit. It's possible this will be tied to Python <= 3.12 for the foreseeable future. Unfortunately I'm also not sure if we can feasibly use something other than jpype here...