apache / arrow-rs

Official Rust implementation of Apache Arrow
https://arrow.apache.org/
Apache License 2.0
2.55k stars 764 forks source link

Archery Integration Test with c# failing on main #6577

Open alamb opened 4 days ago

alamb commented 4 days ago

Describe the bug archery integration test is failing on main : https://github.com/apache/arrow-rs/actions/runs/11363774836/job/31608436255

 ################# FAILURES #################
  FAILED TEST: primitive Rust producing,  C# consuming
  <class 'RuntimeError'>: Command failed: /__w/arrow-rs/arrow-rs/csharp/artifacts/Apache.Arrow.Flight.IntegrationTest/Debug/net8.0/Apache.Arrow.Flight.IntegrationTest client --port 44909 --path /tmp/arrow-integration-hk4dbqi4/generated_primitive.json
  With output:
  --------------
  Unhandled exception: Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error connecting to subchannel.", DebugException="System.ArgumentException: IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')")
   ---> System.ArgumentException: IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')
     at System.Net.Dns.GetHostEntryOrAddressesCoreAsync(String hostName, Boolean justReturnParsedIp, Boolean throwOnIIPAny, Boolean justAddresses, AddressFamily family, CancellationToken cancellationToken)
     at System.Net.Dns.GetHostAddressesAsync(String hostNameOrAddress, AddressFamily family, CancellationToken cancellationToken)
     at System.Net.Sockets.SocketAsyncEventArgs.DnsConnectAsync(DnsEndPoint endPoint, SocketType socketType, ProtocolType protocolType)
     at System.Net.Sockets.Socket.ConnectAsync(SocketAsyncEventArgs e, Boolean userSocket, Boolean saeaCancelable)
     at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
     at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
     at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.OnConnect(Socket socket, DnsEndPoint endpoint, CancellationToken cancellationToken)
     at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.TryConnectAsync(ConnectContext context, Int32 attempt)
     --- End of inner exception stack trace ---
     at Grpc.Net.Client.Internal.HttpContentClientStreamReader`2.MoveNextCore(CancellationToken cancellationToken)
     at Apache.Arrow.Flight.Internal.RecordBatchReaderImplementation.ReadSchemaAsync(CancellationToken cancellationToken) in /__w/arrow-rs/arrow-rs/csharp/src/Apache.Arrow.Flight/Internal/RecordBatchReaderImplementation.cs:line 74
     at Apache.Arrow.Flight.Internal.RecordBatchReaderImplementation.ReadNextRecordBatchAsync(CancellationToken cancellationToken) in /__w/arrow-rs/arrow-rs/csharp/src/Apache.Arrow.Flight/Internal/RecordBatchReaderImplementation.cs:line 118
     at Apache.Arrow.Flight.FlightRecordBatchStreamReader.MoveNext(CancellationToken cancellationToken) in /__w/arrow-rs/arrow-rs/csharp/src/Apache.Arrow.Flight/FlightRecordBatchStreamReader.cs:line 64
     at Apache.Arrow.Flight.IntegrationTest.Scenarios.JsonTestScenario.ConsumeFlightLocation(FlightClient client, FlightTicket ticket, RecordBatch[] batches) in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/Scenarios/JsonTestScenario.cs:line 160
     at Apache.Arrow.Flight.IntegrationTest.Scenarios.JsonTestScenario.RunClient(Int32 serverPort) in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/Scenarios/JsonTestScenario.cs:line 109
     at Apache.Arrow.Flight.IntegrationTest.FlightClientCommand.Execute() in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightClientCommand.cs:line 45
     at Apache.Arrow.Flight.IntegrationTest.Program.<>c.<<Main>b__0_0>d.MoveNext() in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/Program.cs:line 50

To Reproduce

Expected behavior Should pass

Additional context Not sure if this is related to https://github.com/apache/arrow-rs/issues/6568 or not

itsjunetime commented 2 days ago

I think #6585 might fix this as well

alamb commented 19 hours ago

I think #6585 might fix this as well

Thanks @itsjunetime -- unfortunately it seems like the tests are still failing even with #6585 (recent example) . I'll see what I can do

alamb commented 19 hours ago

I poked around in the arrow repo. There are two PRs that went in around the same time this test started failing that could be related

https://github.com/apache/arrow/pull/44377 specifically looks related as this test appears to be new. I will ask some questions there and make a PR to disable this test in arrow-rs while we investigate more

alamb commented 19 hours ago

Here is another more specific message from the logs https://github.com/apache/arrow-rs/actions/runs/11409436304/job/31749661931

 ======================================================================
  Testing file null
  Traceback (most recent call last):
    File "/__w/arrow-rs/arrow-rs/dev/archery/archery/integration/util.py", line 136, in run_cmd
      output = subprocess.check_output(cmd, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/envs/arrow/lib/python3.11/subprocess.py", line 466, in check_output
      return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/conda/envs/arrow/lib/python3.11/subprocess.py", line 571, in run
      raise CalledProcessError(retcode, process.args,
  subprocess.CalledProcessError: Command '['/__w/arrow-rs/arrow-rs/csharp/artifacts/Apache.Arrow.Flight.IntegrationTest/Debug/net8.0/Apache.Arrow.Flight.IntegrationTest', 'client', '--port', '46599', '--path', '/tmp/arrow-integration-2gthhuy4/generated_null.json']' returned non-zero exit status 1.

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/__w/arrow-rs/arrow-rs/dev/archery/archery/integration/runner.py", line 435, in _run_flight_test_case
      consumer.flight_request(port, **client_args)
    File "/__w/arrow-rs/arrow-rs/dev/archery/archery/integration/tester_csharp.py", line 218, in flight_request
      run_cmd(cmd)
    File "/__w/arrow-rs/arrow-rs/dev/archery/archery/integration/util.py", line 145, in run_cmd
      raise RuntimeError(sio.getvalue())
  RuntimeError: Command failed: /__w/arrow-rs/arrow-rs/csharp/artifacts/Apache.Arrow.Flight.IntegrationTest/Debug/net8.0/Apache.Arrow.Flight.IntegrationTest client --port 46599 --path /tmp/arrow-integration-2gthhuy4/generated_null.json
  With output:
  --------------
  Unhandled exception: Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error connecting to subchannel.", DebugException="System.ArgumentException: IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')")
   ---> System.ArgumentException: IPv4 address 0.0.0.0 and IPv6 address ::0 are unspecified addresses that cannot be used as a target address. (Parameter 'hostName')
     at System.Net.Dns.GetHostEntryOrAddressesCoreAsync(String hostName, Boolean justReturnParsedIp, Boolean throwOnIIPAny, Boolean justAddresses, AddressFamily family, CancellationToken cancellationToken)
     at System.Net.Dns.GetHostAddressesAsync(String hostNameOrAddress, AddressFamily family, CancellationToken cancellationToken)
     at System.Net.Sockets.SocketAsyncEventArgs.DnsConnectAsync(DnsEndPoint endPoint, SocketType socketType, ProtocolType protocolType)
     at System.Net.Sockets.Socket.ConnectAsync(SocketAsyncEventArgs e, Boolean userSocket, Boolean saeaCancelable)
     at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
     at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
     at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.OnConnect(Socket socket, DnsEndPoint endpoint, CancellationToken cancellationToken)
     at Grpc.Net.Client.Balancer.Internal.SocketConnectivitySubchannelTransport.TryConnectAsync(ConnectContext context, Int32 attempt)
     --- End of inner exception stack trace ---
     at Apache.Arrow.Flight.Internal.RecordBatchReaderImplementation.ReadSchemaAsync(CancellationToken cancellationToken) in /__w/arrow-rs/arrow-rs/csharp/src/Apache.Arrow.Flight/Internal/RecordBatchReaderImplementation.cs:line 74
     at Apache.Arrow.Flight.Internal.RecordBatchReaderImplementation.ReadNextRecordBatchAsync(CancellationToken cancellationToken) in /__w/arrow-rs/arrow-rs/csharp/src/Apache.Arrow.Flight/Internal/RecordBatchReaderImplementation.cs:line 118
     at Apache.Arrow.Flight.FlightRecordBatchStreamReader.MoveNext(CancellationToken cancellationToken) in /__w/arrow-rs/arrow-rs/csharp/src/Apache.Arrow.Flight/FlightRecordBatchStreamReader.cs:line 64
     at Apache.Arrow.Flight.IntegrationTest.Scenarios.JsonTestScenario.ConsumeFlightLocation(FlightClient client, FlightTicket ticket, RecordBatch[] batches) in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/Scenarios/JsonTestScenario.cs:line 160
     at Apache.Arrow.Flight.IntegrationTest.Scenarios.JsonTestScenario.RunClient(Int32 serverPort) in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/Scenarios/JsonTestScenario.cs:line 109
     at Apache.Arrow.Flight.IntegrationTest.FlightClientCommand.Execute() in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightClientCommand.cs:line 45
     at Apache.Arrow.Flight.IntegrationTest.Program.<>c.<<Main>b__0_0>d.MoveNext() in /__w/arrow-rs/arrow-rs/csharp/test/Apache.Arrow.Flight.IntegrationTest/Program.cs:line 50
  --- End of stack trace from previous location ---
     at System.CommandLine.Invocation.AnonymousCommandHandler.InvokeAsync(InvocationContext context)
     at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseParseDirective>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()

  --------------
alamb commented 19 hours ago

Here is the job definition: https://github.com/apache/arrow-rs/blob/1666a4d7b237370c167e2b90b45d1354e28f500a/.github/workflows/integration.yml#L50-L114

alamb commented 19 hours ago

Posted upstream on https://github.com/apache/arrow/pull/44377#issuecomment-2424869128 to see if we can get any hints on how to fix this