search

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
34.05k stars 5.78k forks source link

Accelerated DAG: Ensure chunks are received in order on the receiver node #46603

Open jackhumphries opened 4 months ago

jackhumphries commented 4 months ago

What happened + What you expected to happen

Ensure that mutable object chunks (sent via PushMutableObject RPC) will always be received in order on the receiver node. See #46498 for context.

Versions / Dependencies

3.0

Reproduction script

test_channel.py::test_payload_large

Issue Severity

Medium: It is a significant difficulty but I can work around it.

rkooo567 commented 3 months ago

@kevin85421 does it affect the correctness? as long as we know the seqno, it should be fine right?