Runtime error in 'ray' : RuntimeError: Failed to unpickle serialized exception

[immiscible-blade]

I am getting this runtime error at random instances while running the code to reproduce the results on GAIA. Can anyone help me with this?

The entire error is the following:

Traceback (most recent call last):
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/exceptions.py", line 49, in from_ray_exception
    return pickle.loads(ray_exception.serialized_exception)
TypeError: APIStatusError.__init__() missing 2 required keyword-only arguments: 'response' and 'body'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/_private/serialization.py", line 412, in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/_private/serialization.py", line 294, in _deserialize_object
    return RayError.from_bytes(obj)
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/exceptions.py", line 43, in from_bytes
    return RayError.from_ray_exception(ray_exception)
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/exceptions.py", line 52, in from_ray_exception
    raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception
Traceback (most recent call last):
  File "/home/tkarthikeyan/Agents/sibyl/Sibyl-System/main.py", line 406, in <module>
    answers = list(agent_pool.map(lambda agent, row: try_except_ask(agent,row), ds))
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/util/actor_pool.py", line 113, in get_generator
    yield self.get_next()
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/util/actor_pool.py", line 309, in get_next
    return ray.get(future)
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/_private/worker.py", line 2623, in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/_private/worker.py", line 863, in get_objects
    raise value
ray.exceptions.RaySystemError: System error: Failed to unpickle serialized exception
traceback: Traceback (most recent call last):
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/exceptions.py", line 49, in from_ray_exception
    return pickle.loads(ray_exception.serialized_exception)
TypeError: APIStatusError.__init__() missing 2 required keyword-only arguments: 'response' and 'body'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/_private/serialization.py", line 412, in deserialize_objects
    obj = self._deserialize_object(data, metadata, object_ref)
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/_private/serialization.py", line 294, in _deserialize_object
    return RayError.from_bytes(obj)
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/exceptions.py", line 43, in from_bytes
    return RayError.from_ray_exception(ray_exception)
  File "/home/tkarthikeyan/miniconda3/envs/sibyl/lib/python3.10/site-packages/ray/exceptions.py", line 52, in from_ray_exception
    raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception

[Ag2S1]

Have you tested your OpenAI API (GPT-4o's ChatCompletion API) and Bing API separately?

[immiscible-blade]

Hi, the model seems to work fine now (with GPT-4o). The error seems to come if I try some other model eg, gpt-4-turbo or gpt-3.5-turbo. However, there was another content filtering error at line 342 for one of the questions, for which I added a try-except block to bypass it for now.

Anyways, is there any way I can use other models apart from GPT-4o with your code for experimenting?

[Ag2S1]

I think we can only add more fault tolerance and retry mechanisms, as other models still have a gap in capabilities