Getting "await wasn't used with future" with --reactor asyncio

[radifalco]

I feel like this is pilot error, but just in case. I've been using twisted and pytest-twisted for a while and I was finally happy to move to the asyncio reactor to see if I could mix and match twisted and asyncio. My plan was to replace txmongo with motor. So I wrote my first seemingly simple test:

async def list_dirs():
    return await asyncio.to_thread(os.listdir, "/")

@inlineCallbacks
def test_asyncio_from_inlineCallbacks():
    r = yield defer.ensureDeferred(list_dirs())
    assert r == []

And this fails with:

Traceback (most recent call last):
  File "/testing.py", line 27, in test_asyncio_from_inlineCallbacks
    r = yield defer.ensureDeferred(list_dirs())
  File "/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py", line 1697, in _inlineCallbacks
    result = context.run(gen.send, result)
  File "/test/testing.py", line 22, in list_dirs
    return await asyncio.to_thread(os.listdir, "/")
  File "/usr/local/lib/python3.10/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
RuntimeError: await wasn't used with future

I've tried several variations and I just can't get it to work. Switching between pytest @inlineCallback to twisted, switched from pytest ensureDeferred to twisted. Using functions vs decorators to convert. No good. Here's some stats:

Twisted: 22.10.0 Python: 3.10.11 pytest: 7.3.1 pytest-twisted: 1.14.0

Start up arguments --reactor asyncio.

I do install the async reactor myself in my __init__.py. But I don't imagine that's the issue. It's just this at that top of my init file:

from twisted.internet import asyncioreactor

asyncioreactor.install()

Sorry if this isn't an Issue, I wasn't sure where to post about it.

[altendky]

async def list_dirs():
    return await asyncio.to_thread(os.listdir, "/")

@inlineCallbacks
def test_asyncio_from_inlineCallbacks():
    r = yield defer.ensureDeferred(list_dirs())
    assert r == []

As a side note, if you are moving to the future then don't be using inlineCallbacks. Just stick to async/await.

But more to an issue here, don't you need to actually cross over from twisted to asyncio explicitly? https://meejah.ca/blog/python3-twisted-and-asyncio references .fromFuture() for example.

[radifalco]

@altendky of course, but we still have a lot of inlineCallbacks and I want to make sure I can await asyncio code fro that code. Do you know why the snippet above isn't working? Or even this one?

async def list_dirs():
    return await asyncio.to_thread(os.listdir, "/")

@ensureDeferred
async def test_basic():
    r = await list_dirs()
    assert r == []

Also, I appreciate the link but I've seen it many times and it's what I'm trying to do here. Use the async reactor in order to mix and match twisted and coroutines. But I can't even get this simple test to work so I assume I must have something setup incorrectly. Or is this not the right way to do it? I guess I just need to ensure that I can call asyncio stuff like Motor from inside of some of my existing inlineCallbacks code.

[radifalco]

Well, this works. Does that mean that for every asyncio coroutine I want to use in existing twisted code needs to be turned into an asyncio future and then turned into a deferred?

async def list_dirs():
    return await asyncio.to_thread(os.listdir, "/")

@inlineCallbacks
def test_basic():
    r = yield Deferred.fromFuture(asyncio.ensure_future(list_dirs()))
    assert r == []

[altendky]

That's the gist, yes. The event loop integrations generally provide for the event loops coexisting in the same thread, not for completely seamless crossing of the boundaries.