async/await for twisted ApplicationSession

[slav0nic]

From the version 16.4 twisted support async/await (py3.5+) https://twistedmatrix.com/documents/current/core/howto/defer-intro.html#coroutines-with-async-await

but it doesn't work in crossbar (crossbar-0.15, autobahn-0.16.0, py3.5, twisted-16.4.1, txaio-2.5.1):

from autobahn.twisted.wamp import ApplicationSession
from autobahn.wamp.exception import ApplicationError

class AuthenticatorSession(ApplicationSession):
    async def onJoin(self, details):
        async def authenticate(realm, authid, details):
            ticket = details['ticket']
            ....
            return data['role']

        try:
            await self.register(authenticate, 'com.app.authenticate')
            self.log.info("Dynamic authenticator registered")
        except Exception as e:
            raise e

in logs i have warning: ...python3.5/site-packages/twisted/internet/defer.py:587: builtins.RuntimeWarning: coroutine 'onJoin' was never awaited

so, will be fine support this or if this is my fail - add example (re #653)

[meejah]

I looked into this a little; it revealed a couple bugs:

• Twisted had a bug in Deferred support for this (https://twistedmatrix.com/trac/ticket/8861)
• txaio will needs a couple minor changes

[meejah]

...but the good news is that with those changes, it does in fact work. The first step will be to wait for the Twisted 16.5.0 release, though (which does contain the .send fixes).

[hawkowl]

16.6 is out which has all the fixes. As far as I know, it should be solid now.

[ksamuel]

Hello,

Does it work now ?

And more importantly, can I use it for the websocket part with twisted ?

I'm currently working on some writing material for Tobias, and he adviced:

"Could you post your findings to above issue, so we can take @meejah in the loop, because he might know better" :)

I tried something like this:

class EchoClientProtocol(WebSocketClientProtocol):
    ...
    async def onOpen(self):
       await do_async_stuff()
    ...

It works with asyncio of course, but not with twisted, where I get the "coroutine 'onOpen' was never awaited" mentioned previously.

Did I get something wrong or the support is not in yet ? I don't need the feature, I just need to know what's the best code style I can use for the writing.

---

>>> import twisted
>>> twisted.__version__
'17.5.0'
>>> import autobahn
>>> autobahn.__version__
'17.8.1'

[meejah]

I believe we'd need to enhance either Autobahn (or txaio) so that async def methods for the Autobahn/WebSocket callbacks work properly -- basically, you need to wrap them in an ensureDeferred call for Twisted to turn the async-def method into a Deferred-returning method...

[meejah]

That said, pure "user" code should be able to take advantage of Python3 features right now like so (untested):

class Component(ApplicationSession):
    def onJoin(self, details):
        return ensureDeferred(self._on_join(details))

    async def _on_join(self, details):
        print("joined {}".format(details))
        await self._other_async()

[oberstet]

@meejah this sounds like we could get away with adding overrides for onJoin etc to autobahn.twisted.wamp.ApplicationSession, the bodies of the overrides simply returning the base result wrapped in ensureDeferred ?

[meejah]

Hmm, maybe? I've not played with ensureDeferred enough to know if that'll play nicely with Py2 code using "not async-def".

I think we might need to check if calling self.onJoin (et al) returned a co-routine or not -- which might make it a good thing to just put in txaio.ensure_future instead (i.e. that can then "just work" no matter if the function returns a Deferred directly, uses @inlineCallbacks or async def)

[oberstet]

@ksamuel rgd the AB chapter, I guess even if we get it working easily and quickly (as above seems to indicate), essentially all code out there uses AB/Twisted in an inlineCallback/Deferred kind of style, and hence one could argue the text should use that style as well. On the other hand, the new style is just .. nicer=) a little syntax sugar goes a long way. Opinions?

@meejah yeah, sounds good! txaio is a better place for this mangling of return values into one type. I am wondering: do you mean a new function (txaio.ensure_future) or the existing txaio.as_future:

(cpy362_1) oberstet@thinkpad-t430s:~/scm/crossbario/crossbar-fabric-public$ find /home/oberstet/scm/crossbario/autobahn-python/autobahn -name "*.py" -exec grep -Hi "as_future" {} \;
/home/oberstet/scm/crossbario/autobahn-python/autobahn/util.py:            future = txaio.as_future(handler, *args, **kwargs)
/home/oberstet/scm/crossbario/autobahn-python/autobahn/wamp/component.py:                    d = txaio.as_future(self._entry, reactor, session)
/home/oberstet/scm/crossbario/autobahn-python/autobahn/wamp/component.py:        d = txaio.as_future(comp.start, reactor)
/home/oberstet/scm/crossbario/autobahn-python/autobahn/wamp/protocol.py:            lambda _: txaio.as_future(self.onConnect),
/home/oberstet/scm/crossbario/autobahn-python/autobahn/wamp/protocol.py:                    lambda _: txaio.as_future(self.onJoin, details),
/home/oberstet/scm/crossbario/autobahn-python/autobahn/wamp/protocol.py:                d = txaio.as_future(self.onLeave, details)
/home/oberstet/scm/crossbario/autobahn-python/autobahn/wamp/protocol.py:                d = txaio.as_future(self.onChallenge, challenge)
/home/oberstet/scm/crossbario/autobahn-python/autobahn/wamp/protocol.py:                    d = txaio.as_future(self.onLeave, details)
/home/oberstet/scm/crossbario/autobahn-python/autobahn/wamp/protocol.py:                d = txaio.as_future(self.onLeave, details)
/home/oberstet/scm/crossbario/autobahn-python/autobahn/wamp/protocol.py:                        future = txaio.as_future(handler.fn, *invoke_args, **invoke_kwargs)
/home/oberstet/scm/crossbario/autobahn-python/autobahn/wamp/protocol.py:                            on_reply = txaio.as_future(endpoint.fn, *invoke_args, **invoke_kwargs)
/home/oberstet/scm/crossbario/autobahn-python/autobahn/wamp/protocol.py:            d = txaio.as_future(self.onLeave, details)
/home/oberstet/scm/crossbario/autobahn-python/autobahn/wamp/protocol.py:        d = txaio.as_future(self.onDisconnect)
/home/oberstet/scm/crossbario/autobahn-python/autobahn/websocket/protocol.py:                f = txaio.as_future(self.onConnect, request)

[meejah]

Oh, sorry yes I meant existing as_future

[oberstet]

Ah, right;) Ok, then as far as I see, there is actually 2 issues here:

1. txaio: add the ensureDeferred support to txaio.as_future
2. AB: actually use it (also in WebSocket code) ;) as above grep shows, it is only used in onConnect right now, but it should be used in the other hooks as well ..

Both can be done without breaking interfaces.

Does that make sense?

[ksamuel]

Opinions?

@oberstet: I will start writing all code using the inlineCallback/Deferred syntax. If we get a release allowing async / await before I submit my final draft, I'll add a note with an example using the new syntax and a mention AB supports it. But most of the code will use the inlineCallback/Deferred syntax in order to be congruent with the rest of the book.

[meejah]

@ksamuel I think that makes the most sense; most Twisted users will likely "expect" to see "straight Deferreds" or "@inlineCallbacks" style code.

[oberstet]

async/await should be fully usable on Py3/Twisted now with Autobahn