Error when using Lock from thread in asyncio

davidbrochart commented 2 weeks ago

Things to check first

[X] I have searched the existing issues and didn't find my bug already reported there
[X] I have checked that my bug is still present in the latest release

AnyIO version

4.4.0

Python version

3.12.5

What happened?

With the asyncio backend, this code fails with TypeError: cannot create weak reference to 'NoneType' object. It works with the trio backend.

Traceback

Traceback (most recent call last):
  File "/home/david/git/pycrdt/foo.py", line 11, in 
    run(main)
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_core/_eventloop.py", line 74, in run
    return async_backend.run(func, args, {}, backend_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2034, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2022, in wrapper
    return await func(*args)
           ^^^^^^^^^^^^^^^^^
  File "/home/david/git/pycrdt/foo.py", line 9, in main
    await to_thread.run_sync(worker, lock)
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/pycrdt/foo.py", line 5, in worker
    from_thread.run_sync(lock.release)
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/from_thread.py", line 80, in run_sync
    return async_backend.run_sync_from_thread(func, args, token=token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2239, in run_sync_from_thread
    return f.result()
           ^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2230, in wrapper
    f.set_result(func(*args))
                 ^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_core/_synchronization.py", line 202, in release
    if self._owner_task != get_current_task():
                           ^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_core/_testing.py", line 63, in get_current_task
    return get_async_backend().get_current_task()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2484, in get_current_task
    return AsyncIOTaskInfo(current_task())  # type: ignore[arg-type]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1839, in __init__
    task_state = _task_states.get(task)
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/pycrdt/lib/python3.12/weakref.py", line 452, in get
    return self.data.get(ref(key),default)
                         ^^^^^^^^
TypeError: cannot create weak reference to 'NoneType' object

How can we reproduce the bug?

from anyio import Lock, from_thread, to_thread, run

def worker(lock):
    from_thread.run(lock.acquire)
    from_thread.run_sync(lock.release)

async def main():
    lock = Lock()
    await to_thread.run_sync(worker, lock)

run(main)

agronholm commented 2 weeks ago

The immediate cause is that it's trying to create a weak reference to None. It's trying to do this because current_task() returns None, but I'm still trying to figure out why it would do that.

davidbrochart commented 2 weeks ago

BTW with #761 the error is different: RuntimeError: The current task is not holding this lock.

Traceback

Traceback (most recent call last):
  File "/home/david/git/anyio/foo.py", line 11, in 
    run(main)
  File "/home/david/git/anyio/src/anyio/_core/_eventloop.py", line 74, in run
    return async_backend.run(func, args, {}, backend_options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2173, in run
    return runner.run(wrapper())
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/anyio/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/anyio/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2161, in wrapper
    return await func(*args)
           ^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/foo.py", line 9, in main
    await to_thread.run_sync(worker, lock)
  File "/home/david/git/anyio/src/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2326, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 872, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/foo.py", line 5, in worker
    from_thread.run_sync(lock.release)
  File "/home/david/git/anyio/src/anyio/from_thread.py", line 80, in run_sync
    return async_backend.run_sync_from_thread(func, args, token=token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2388, in run_sync_from_thread
    return f.result()
           ^^^^^^^^^^
  File "/home/david/micromamba/envs/anyio/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/david/micromamba/envs/anyio/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 2379, in wrapper
    f.set_result(func(*args))
                 ^^^^^^^^^^^
  File "/home/david/git/anyio/src/anyio/_backends/_asyncio.py", line 1728, in release
    raise RuntimeError("The current task is not holding this lock")
RuntimeError: The current task is not holding this lock

smurfix commented 2 weeks ago

RuntimeError: The current task is not holding this lock.

That's to be expected.

The workaround is to use a nice 'async with lock:` block, and call back to the thread from within that.

davidbrochart commented 2 weeks ago

Oh I see, thanks. I guess I should close this issue then, and not give Alex more work than needed.

agronholm commented 2 weeks ago

I agree with @smurfix 's suggestion, but I would still like to understand why there was an error, and why it only happens with asyncio, as the effects of the problem may not be limited to locking.

agronholm commented 2 weeks ago

For posterity, the reason for this was that I had opted to use call_soon_threadsafe() as the backing implementation for run_sync_from_thread(). As such, the code runs in an environment where there is no backing task, hence returning None from current_task(). I'm reopening this as this may cause trouble further down the line, but it's not a priority.

agronholm / anyio