Open Catch-Bull opened 1 month ago
@jjyao I think it is necessary to discussion on whether this is a bug.
If this is a bug, we need to modify the function InlineDependencies
: to put arguments into plasma when the inlined arguments exceed task_rpc_inlined_bytes_limit
, but it seems not easy.
@Catch-Bull I think it's a known limitation: #7269. Our workaround was to increase grpc request size limit so that in most cases, the sum of inlined arguments won't exceed the limit.
Agree that the fix you suggested is not easy.
What happened + What you expected to happen
Ray supports controlling the total size of inlined task arguments within a specified range through config
task_rpc_inlined_bytes_limit
, but when the arguments is inlined returns value, configtask_rpc_inlined_bytes_limit
not work.Versions / Dependencies
ray: 2.23.0 python: 3.10.14
Reproduction script
test.py:
run command:
this script will hang forever, and found this error on
python-core-driver-01000000ffffffffffffffffffffffffffffffffffffffffffffffff_xxxx.log
:Issue Severity
Medium: It is a significant difficulty but I can work around it.