xorbitsai / xorbits

Scalable Python DS & ML, in an API compatible & lightning fast way.
https://xorbits.readthedocs.io
Apache License 2.0
1.1k stars 67 forks source link

BUG: asyncio.exceptions.InvalidStateError: invalid state #659

Open codingl2k1 opened 1 year ago

codingl2k1 commented 1 year ago

Describe the bug

A clear and concise description of what the bug is.

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/xorbits/_mars/services/scheduling/worker/execution.py", line 402, in internal_run_subtask
    subtask_info.result = await self._retry_run_subtask(
  File "/usr/local/lib/python3.10/dist-packages/xorbits/_mars/services/scheduling/worker/execution.py", line 513, in _retry_run_subtask
    return await _retry_run(subtask, subtask_info, _run_subtask_once)
  File "/usr/local/lib/python3.10/dist-packages/xorbits/_mars/services/scheduling/worker/execution.py", line 111, in _retry_run
    raise ex
  File "/usr/local/lib/python3.10/dist-packages/xorbits/_mars/services/scheduling/worker/execution.py", line 73, in _retry_run
    return await target_async_func(*args)
  File "/usr/local/lib/python3.10/dist-packages/xorbits/_mars/services/scheduling/worker/execution.py", line 480, in _run_subtask_once
    await mo.wait_actor_pool_recovered(sub_pool_address, self.address)
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/core.py", line 138, in call_with_client
    r = await wait_response
  File "/usr/local/lib/python3.10/dist-packages/xoscar/backends/core.py", line 88, in _listen
    future.set_result(message)
asyncio.exceptions.InvalidStateError: invalid state

To Reproduce

To help us to reproduce this bug, please provide information below:

  1. Your Python version
  2. The version of Xorbits you use
  3. Versions of crucial packages, such as numpy, scipy and pandas
  4. Full stack of the error.
  5. Minimized code to reproduce the error.

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

Add any other context about the problem here.