search

apache / arrow

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing
https://arrow.apache.org/
Apache License 2.0
14.28k stars 3.47k forks source link

[C++][FlightRPC] Sending 0-length buffers to gRPC can result in indefinite hangs on MacOS/iOS platforms #43447

Closed ziglerari closed 1 month ago

ziglerari commented 1 month ago

Describe the bug, including details regarding any error messages, version, and platform.

A gRPC issue has been identified where transmitting buffers of zero length leads to a persistent hang on MacOS/iOS platforms.

Such zero-length buffers may arise, for example, in the context of using the Array structure alongside a validity bitmap, as outlined in the Arrow Spec.

Essentially, when every element within an Array is valid (i.e., not null), it's possible to represent this state with a null validity bitmap, indicating that all elements are valid. This scenario is realized through the use of a null buffer, as demonstrated here: https://github.com/apache/arrow/blob/187197c369058f7d1377c1b161c469a9e4542caf/cpp/src/arrow/ipc/writer.cc#L165-L179

The relevant sections of code from both transport mechanisms are provided below for reference: https://github.com/apache/arrow/blob/187197c369058f7d1377c1b161c469a9e4542caf/cpp/src/arrow/flight/transport/grpc/serialization_internal.cc#L283-L287

https://github.com/apache/arrow/blob/187197c369058f7d1377c1b161c469a9e4542caf/cpp/src/arrow/flight/transport/ucx/ucx_internal.cc#L590-L591

Upon examining the UCX transport's approach, it's evident that a precaution is taken to avoid sending zero-length buffers. This strategy appears prudent, as it eliminates the need to forward non-transmittable buffers to the transport layer, potentially offering a solution to the issue observed with gRPC.

Component(s)

C++, FlightRPC

lidavidm commented 1 month ago

Issue resolved by pull request 43448 https://github.com/apache/arrow/pull/43448