[Integration][CI] verify-rc-source-integration-linux-ubuntu-22.04-amd64 is failing around JS

[raulcd]

Describe the bug, including details regarding any error messages, version, and platform.

The verify-rc-source-integration-linux-ubuntu-22.04-amd64 is failing on several JS producing / consuming stages:

  ################# FAILURES #################
  FAILED TEST: primitive_no_batches C++ producing,  JS consuming
  <class 'RuntimeError'>: Command failed: /arrow/js/bin/integration.ts -a /tmp/tmptj7xy7d2/b41218cd_generated_primitive_no_batches.consumer_stream_as_file -j /tmp/arrow-integration-726dm6bp/generated_primitive_no_batches.json --mode VALIDATE
  With output:

another example (https://github.com/ursacomputing/crossbow/actions/runs/11147590724/job/31018012263):

  ################# FAILURES #################
  FAILED TEST: map_non_canonical C++ producing,  JS consuming
  <class 'subprocess.CalledProcessError'>: Command '/arrow/js/bin/stream-to-file.ts < /tmp/tmpvvkmxtx1/cfe4a30b_generated_map_non_canonical.producer_file_as_stream > /tmp/tmpvvkmxtx1/cfe4a30b_generated_map_non_canonical.consumer_stream_as_file' returned non-zero exit status 133.

  FAILED TEST: primitive JS producing,  C++ consuming
  <class 'subprocess.CalledProcessError'>: Command '/arrow/js/bin/file-to-stream.ts < /tmp/tmpvvkmxtx1/b4860393_generated_primitive.json_as_file > /tmp/tmpvvkmxtx1/b4860393_generated_primitive.producer_file_as_stream' returned non-zero exit status 133.

  FAILED TEST: custom_metadata JS producing,  Java consuming
  <class 'subprocess.CalledProcessError'>: Command '/arrow/js/bin/file-to-stream.ts < /tmp/tmpvvkmxtx1/219d461d_generated_custom_metadata.json_as_file > /tmp/tmpvvkmxtx1/219d461d_generated_custom_metadata.producer_file_as_stream' returned non-zero exit status 133.

  FAILED TEST: primitive_no_batches Go producing,  JS consuming
  <class 'RuntimeError'>: Command failed: /arrow/js/bin/integration.ts -a /tmp/tmpvvkmxtx1/3b3ee219_generated_primitive_no_batches.json_as_file -j /tmp/arrow-integration-67snajw_/generated_primitive_no_batches.json --mode VALIDATE

Those jobs have been triggered 4-5 times (as seen on the links) and are failing on different PRs. I am unsure on what the problem is.

Component(s)

Continuous Integration, Integration

[raulcd]

@kou @pitrou I have no idea what the problem is but seems to be failing always around JS. Any idea what could be? It also seems to only fail on the Ubuntu 22.04 which makes things weirder. Probably something we should investigate / fix for the release

[pitrou]

Well, is there anyone knowledgeable in JS that we can ask to take a look?

[raulcd]

@domoritz @trxcllnt maybe?

[trxcllnt]

Difficult to tell what's going wrong without seeing any output. Is there an easy way to replicate the failures locally?

[amoeba]

We should be able to reproduce this locally with archery. Let me try first and then point you in the right direction @trxcllnt.

[amoeba]

Hrm, this isn't failing for me locally. I set up archery and ran,

$ archery docker run -e TEST_DEFAULT=0 -e TEST_INTEGRATION=1 conda-verify-rc bash
$ /arrow/dev/release/setup-ubuntu.sh
$ export JAVA_HOME="/usr/lib/jvm/java-11-openjdk-arm64"
$ TEST_DEFAULT=0 TEST_INTEGRATION=1 /arrow/dev/release/verify-release-candidate.sh

@trxcllnt: Here's the complete output from the failed test on CI,

``` 2024-10-03T10:37:23.8786244Z ##[group]Integration: Test: Result 2024-10-03T10:37:23.8786581Z ################# FAILURES ################# 2024-10-03T10:37:23.8787029Z FAILED TEST: primitive_no_batches C++ producing, JS consuming 2024-10-03T10:37:23.8788480Z : Command failed: /arrow/js/bin/integration.ts -a /tmp/tmptj7xy7d2/b41218cd_generated_primitive_no_batches.consumer_stream_as_file -j /tmp/arrow-integration-726dm6bp/generated_primitive_no_batches.json --mode VALIDATE 2024-10-03T10:37:23.8789619Z With output: 2024-10-03T10:37:23.8789842Z -------------- 2024-10-03T10:37:23.8790091Z 2024-10-03T10:37:23.8790098Z 2024-10-03T10:37:23.8790174Z # 2024-10-03T10:37:23.8790376Z # Fatal error in , line 0 2024-10-03T10:37:23.8790946Z # Check failed: ReadSingleBytecodeData( source_.Get(), SlotAccessorForHandle(&ret, isolate())) == 1. 2024-10-03T10:37:23.8791523Z # 2024-10-03T10:37:23.8791695Z # 2024-10-03T10:37:23.8791863Z # 2024-10-03T10:37:23.8792066Z #FailureMessage Object: 0x7f59117f6db0 2024-10-03T10:37:23.8792418Z ----- Native stack trace ----- 2024-10-03T10:37:23.8792605Z 2024-10-03T10:37:23.8792698Z 1: 0xd3f611 [node] 2024-10-03T10:37:23.8792980Z 2: 0x217b8b1 V8_Fatal(char const*, ...) [node] 2024-10-03T10:37:23.8793550Z 3: 0x15707b1 [node] 2024-10-03T10:37:23.8794091Z 4: 0x156fad6 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8795546Z 5: 0x156fe89 int v8::internal::Deserializer::ReadSingleBytecodeData >(unsigned char, v8::internal::SlotAccessorForHandle) [node] 2024-10-03T10:37:23.8797101Z 6: 0x157078f v8::internal::Deserializer::ReadObject() [node] 2024-10-03T10:37:23.8797853Z 7: 0x156fad6 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8799067Z 8: 0x156e7ab int v8::internal::Deserializer::ReadSingleBytecodeData(unsigned char, v8::internal::SlotAccessorForHeapObject) [node] 2024-10-03T10:37:23.8800360Z 9: 0x156f8e1 v8::internal::Deserializer::ReadData(v8::internal::Handle, int, int) [node] 2024-10-03T10:37:23.8801296Z 10: 0x156fba8 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8802492Z 11: 0x156e7ab int v8::internal::Deserializer::ReadSingleBytecodeData(unsigned char, v8::internal::SlotAccessorForHeapObject) [node] 2024-10-03T10:37:23.8803777Z 12: 0x156f8e1 v8::internal::Deserializer::ReadData(v8::internal::Handle, int, int) [node] 2024-10-03T10:37:23.8804719Z 13: 0x156fba8 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8805976Z 14: 0x156e7ab int v8::internal::Deserializer::ReadSingleBytecodeData(unsigned char, v8::internal::SlotAccessorForHeapObject) [node] 2024-10-03T10:37:23.8807532Z 15: 0x156f8e1 v8::internal::Deserializer::ReadData(v8::internal::Handle, int, int) [node] 2024-10-03T10:37:23.8808480Z 16: 0x156fba8 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8809681Z 17: 0x156e7ab int v8::internal::Deserializer::ReadSingleBytecodeData(unsigned char, v8::internal::SlotAccessorForHeapObject) [node] 2024-10-03T10:37:23.8810965Z 18: 0x156f8e1 v8::internal::Deserializer::ReadData(v8::internal::Handle, int, int) [node] 2024-10-03T10:37:23.8811914Z 19: 0x156fba8 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8813112Z 20: 0x156e7ab int v8::internal::Deserializer::ReadSingleBytecodeData(unsigned char, v8::internal::SlotAccessorForHeapObject) [node] 2024-10-03T10:37:23.8814398Z 21: 0x156f8e1 v8::internal::Deserializer::ReadData(v8::internal::Handle, int, int) [node] 2024-10-03T10:37:23.8815333Z 22: 0x156fba8 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8816804Z 23: 0x156e7ab int v8::internal::Deserializer::ReadSingleBytecodeData(unsigned char, v8::internal::SlotAccessorForHeapObject) [node] 2024-10-03T10:37:23.8818108Z 24: 0x156f8e1 v8::internal::Deserializer::ReadData(v8::internal::Handle, int, int) [node] 2024-10-03T10:37:23.8819043Z 25: 0x156fba8 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8820403Z 26: 0x156e7ab int v8::internal::Deserializer::ReadSingleBytecodeData(unsigned char, v8::internal::SlotAccessorForHeapObject) [node] 2024-10-03T10:37:23.8822318Z 27: 0x156f8e1 v8::internal::Deserializer::ReadData(v8::internal::Handle, int, int) [node] 2024-10-03T10:37:23.8823643Z 28: 0x156fba8 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8825498Z 29: 0x156e7ab int v8::internal::Deserializer::ReadSingleBytecodeData(unsigned char, v8::internal::SlotAccessorForHeapObject) [node] 2024-10-03T10:37:23.8827178Z 30: 0x156f8e1 v8::internal::Deserializer::ReadData(v8::internal::Handle, int, int) [node] 2024-10-03T10:37:23.8828164Z 31: 0x156fba8 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8830246Z 32: 0x156e7ab int v8::internal::Deserializer::ReadSingleBytecodeData(unsigned char, v8::internal::SlotAccessorForHeapObject) [node] 2024-10-03T10:37:23.8832525Z 33: 0x156f8e1 v8::internal::Deserializer::ReadData(v8::internal::Handle, int, int) [node] 2024-10-03T10:37:23.8834076Z 34: 0x156fba8 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8836346Z 35: 0x156e7ab int v8::internal::Deserializer::ReadSingleBytecodeData(unsigned char, v8::internal::SlotAccessorForHeapObject) [node] 2024-10-03T10:37:23.8838388Z 36: 0x156f8e1 v8::internal::Deserializer::ReadData(v8::internal::Handle, int, int) [node] 2024-10-03T10:37:23.8840024Z 37: 0x156fba8 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8841966Z 38: 0x156e7ab int v8::internal::Deserializer::ReadSingleBytecodeData(unsigned char, v8::internal::SlotAccessorForHeapObject) [node] 2024-10-03T10:37:23.8843996Z 39: 0x156f8e1 v8::internal::Deserializer::ReadData(v8::internal::Handle, int, int) [node] 2024-10-03T10:37:23.8845512Z 40: 0x156fba8 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8847596Z 41: 0x156e7ab int v8::internal::Deserializer::ReadSingleBytecodeData(unsigned char, v8::internal::SlotAccessorForHeapObject) [node] 2024-10-03T10:37:23.8849567Z 42: 0x156f8e1 v8::internal::Deserializer::ReadData(v8::internal::Handle, int, int) [node] 2024-10-03T10:37:23.8851033Z 43: 0x156fba8 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8852862Z 44: 0x156e7ab int v8::internal::Deserializer::ReadSingleBytecodeData(unsigned char, v8::internal::SlotAccessorForHeapObject) [node] 2024-10-03T10:37:23.8854983Z 45: 0x156f8e1 v8::internal::Deserializer::ReadData(v8::internal::Handle, int, int) [node] 2024-10-03T10:37:23.8856884Z 46: 0x156fba8 v8::internal::Deserializer::ReadObject(v8::internal::SnapshotSpace) [node] 2024-10-03T10:37:23.8859027Z 47: 0x156fe89 int v8::internal::Deserializer::ReadSingleBytecodeData >(unsigned char, v8::internal::SlotAccessorForHandle) [node] 2024-10-03T10:37:23.8861047Z 48: 0x157078f v8::internal::Deserializer::ReadObject() [node] 2024-10-03T10:37:23.8862004Z 49: 0x1575322 v8::internal::ObjectDeserializer::Deserialize() [node] 2024-10-03T10:37:23.8863641Z 50: 0x1575534 v8::internal::ObjectDeserializer::DeserializeSharedFunctionInfo(v8::internal::Isolate*, v8::internal::SerializedCodeData const*, v8::internal::Handle) [node] 2024-10-03T10:37:23.8866450Z 51: 0x1568272 v8::internal::CodeSerializer::Deserialize(v8::internal::Isolate*, v8::internal::AlignedCachedData*, v8::internal::Handle, v8::ScriptOriginOptions, v8::internal::MaybeHandle) [node] 2024-10-03T10:37:23.8868235Z 52: 0xfc171b [node] 2024-10-03T10:37:23.8870508Z 53: 0xfc19fd v8::internal::Compiler::GetSharedFunctionInfoForScriptWithCachedData(v8::internal::Isolate*, v8::internal::Handle, v8::internal::ScriptDetails const&, v8::internal::AlignedCachedData*, v8::ScriptCompiler::CompileOptions, v8::ScriptCompiler::NoCacheReason, v8::internal::NativesFlag) [node] 2024-10-03T10:37:23.8872785Z 54: 0xf2180c [node] 2024-10-03T10:37:23.8874017Z 55: 0xf21918 v8::ScriptCompiler::CompileUnboundScript(v8::Isolate*, v8::ScriptCompiler::Source*, v8::ScriptCompiler::CompileOptions, v8::ScriptCompiler::NoCacheReason) [node] 2024-10-03T10:37:23.8875800Z 56: 0xcae901 node::contextify::ContextifyScript::New(v8::FunctionCallbackInfo const&) [node] 2024-10-03T10:37:23.8877202Z 57: 0xf5f8ff v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [node] 2024-10-03T10:37:23.8878059Z 58: 0xf5feb5 [node] 2024-10-03T10:37:23.8878817Z 59: 0xf605d3 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node] 2024-10-03T10:37:23.8879682Z 60: 0x196adf6 [node] ```

I see a few pointers when googling bits of the above stack trace but need more time to figure out what might be going on.

[trxcllnt]

From the v8::ScriptCompiler::CompileUnboundScript reference in the stack trace, it seems like the problem may be in ts-node or swc, since we use those to transpile the source typescript file. It's possible an update to one of those dependencies broke things, and/or an update to those dependencies would fix it.

[amoeba]

Thanks @trxcllnt. Re: dependencies, I see a number of dependabot PRs but none look like likely culprits. If we can reproduce the CI failure, looking at ts-node and/or swc looks like a reasonable next step.