ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
34.26k stars 5.81k forks source link

Unclear error when using generator tasks #25836

Open frank-lsf opened 2 years ago

frank-lsf commented 2 years ago

What happened + What you expected to happen

When I run the attached code, Ray crashes with the following Python exception:

2022-06-16 04:19:13,182 INFO services.py:1477 -- View the Ray dashboard at http://127.0.0.1:8265
Traceback (most recent call last):
  File "/home/ubuntu/raysort/yield.py", line 13, in <module>
    print(ray.get(f.options(num_returns=n).remote(n)))
  File "/home/ubuntu/miniconda3/envs/raysort/lib/python3.9/site-packages/ray/_private/client_mode_hook.py", line 105, in wrapper
    return func(*args, **kwargs)
  File "/home/ubuntu/miniconda3/envs/raysort/lib/python3.9/site-packages/ray/worker.py", line 2183, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(UnboundLocalError): ray::f() (pid=371241, ip=172.31.27.175)
UnboundLocalError: local variable 'i' referenced before assignment

Which left me quite clueless. My guess is that Ray decides to use the new generator task once it sees the generator flag of function f and the num_returns annotation, but in this case the function did not yield anything (since the use_yield flag is off), and this was a surprise to Ray.

My suggestion is either (1) allow an explicit override in ray.remote to force a task to be a generator task or a regular task, or (2) at least make the error message more meaningful (expected generator, but task did not yield anything).

cc @stephanie-wang

Versions / Dependencies

nightly

Reproduction script

import ray

@ray.remote
def f(n, use_yield=False):
    if use_yield:
        for i in range(n):
            yield i
    else:
        return [i for i in range(n)]

n = 10
print(ray.get(f.options(num_returns=n).remote(n)))

Issue Severity

Low: It annoys or frustrates me.

zhe-thoughts commented 2 years ago

Marking as P2 based on Issue Severity. @scv119 @stephanie-wang feel free to correct me