Twisted 24.10.0 incompatibility

[karolyi]

Hello,

Twisted 24.10.0 is officially out: https://github.com/twisted/twisted/releases/tag/twisted-24.10.0

Daphne seems to be incompatible with it. I'm running 4.1.2. The traceback is:

2024-10-28 17:44:47,491 ERROR    Traceback (most recent call last):
  File "/home/project/venv/lib/python3.10/site-packages/daphne/http_protocol.py", line 150, in process
    self.channel._networkProducer.resumeProducing()
AttributeError: 'HTTPChannel' object has no attribute '_networkProducer'

[carltongibson]

@karolyi Thanks for the report!

Will need to dig in. If you're able to investigate, super.

[carltongibson]

Looks like tests pass with 24.3 but fail with 24.7.

https://github.com/twisted/twisted/releases/tag/twisted-24.7.0

[carltongibson]

Hmmm. But my test project is working with both 24.7 and 24.10, so it's not immediately obvious where the problem comes up. Will have to dig.

Are you able to provide a demo that triggers the error @karolyi ?

[karolyi]

It's a pretty huge project and I'm spent right now (been through a huge dev process, taking a lot outta me), but from what I can see, it happens on websocket connections.

That's all I can tell now, not sure it helps. I'm pretty busy in the upcoming days (reached a huge milestone and have to get the whole thing into production), so for now I went back to 24.7.0 which works.

I might put something together later on to reproduce the bug, if you don't manage to do it until then.

[carltongibson]

@karolyi OK, no stress. Thanks for the heads up. I will try and pin it down.

If any more info comes your way, please do add it (but not an issue if you're busy).

Take care.

[karolyi]

🤝

[carltongibson]

OK, failures I was seeing were an artefact. (I had a checkout on a branch prior to #525 🤦) Without that I haven't got much to go on currently... 🤔

[carltongibson]

There used to be a conditional check at the error location

https://github.com/django/daphne/commit/0b37e80614fabb8ade419972d93c1115d13a365d#diff-1ed03cbfb8907b3705de148e4a604b28290b730db6df250bac1081215be50301R114

... which explains the comment.

Need to see when that disappeared. (Bets now that it was me that removed it 😅)

[carltongibson]

From the comments on b72349d:

AFAICS: The _networkProducer should be set unconditionally by twisted in connectionMade — so there's something else going on.

[hagen00]

I have this error on production. Are there any workarounds?

[karolyi]

I'm onto something here, please hold on.

[karolyi]

So the issue clearly can be linked to running daphne (and not django's runserver) with a socket passed, e.g.:

env DJANGO_SETTINGS_MODULE=project.settings daphne --websocket_timeout=43200 --access-log=logs/daphne.log -u=/tmp/asgi-daphne-project.sock --proxy-headers --proxy-headers-host X-Real-IP project.asgi:application

This way I can reproduce the bug locally too. But in order to test this, you'll need a minimally configured nginx (which I luckily have) that will listen on the actual http port and forward the messages to the unix socket.

@hagen00 your best course of action is the same as mine, going back to 24.7.0 until this gets fixed.

[karolyi]

Full traceback:

2024-10-29 10:52:35,759 ERROR    Exception inside application: 'HTTPChannel' object has no attribute '_networkProducer'
Traceback (most recent call last):
  File "/home/karolyi/project/venv/lib/python3.10/site-packages/channels/routing.py", line 62, in __call__
    return await application(scope, receive, send)
  File "/home/karolyi/project/venv/lib/python3.10/site-packages/django/core/handlers/asgi.py", line 160, in __call__
    await self.handle(scope, receive, send)
  File "/home/karolyi/project/venv/lib/python3.10/site-packages/django/core/handlers/asgi.py", line 190, in handle
    await self.send_response(response, send)
  File "/home/karolyi/project/venv/lib/python3.10/site-packages/django/core/handlers/asgi.py", line 289, in send_response
    await send(
  File "/home/karolyi/project/venv/lib/python3.10/site-packages/daphne/server.py", line 241, in handle_reply
    protocol.handle_reply(message)
  File "/home/karolyi/project/venv/lib/python3.10/site-packages/daphne/http_protocol.py", line 274, in handle_reply
    self.finish()
  File "/home/karolyi/project/venv/lib/python3.10/site-packages/daphne/http_protocol.py", line 230, in finish
    http.Request.finish(self)
  File "/home/karolyi/project/venv/lib/python3.10/site-packages/twisted/web/http.py", line 1240, in finish
    self._cleanup()
  File "/home/karolyi/project/venv/lib/python3.10/site-packages/twisted/web/http.py", line 966, in _cleanup
    self.channel.requestDone(self)
  File "/home/karolyi/project/venv/lib/python3.10/site-packages/twisted/web/http.py", line 2643, in requestDone
    self._networkProducer.resumeProducing()
AttributeError: 'HTTPChannel' object has no attribute '_networkProducer'

Fun fact, it does happen with normal HTTP requests too, but at least those get served.

On a second look (and looking at the full traceback), this seems more like a twisted bug, not a daphne one.

[karolyi]

I'm opening a bugreport in twisted for this.

[hagen00]

Yes, thanks a lot! Pinning it like this in requirements.txt fixed it for me

Twisted==24.7.0

[carltongibson]

Good work @karolyi! 🎁

👀

[karolyi]

Thanks, good things happen when you get a good night's sleep and are through the morning coffee ritual, sacrificing a little time for this before the actual paid work begins. :)

What's interesting to me is that the exception line coming from production (see the first comment) differs from that of my dev env that has the bug reproduced. Nonetheless, it seems this is a bug in twisted.

Let's give some time to them to get this figured out.

[adiroiban]

Hi. It looks like daphne uses Twisted private API

https://github.com/django/daphne/blob/06afd9b94aa82ebdfea945a1810ce25d32d3593b/daphne/http_protocol.py#L149-L150

---

At Twisted we put a lot of effort to keep backward compatibilty, but this is only for the public API.

---

If there is any need to use a private API, please create a GitHub issue to twisted/twisted and we can see how the public API can be updated to support the missing functionality.

---

If you need to use private API, we highly recommend to setup automated tests using the twisted/twisted trunk branch. With pip it should be easy to install Twisted via git.

---

There is an ongoing effort to improve the performance of Twisted and this will involve doing some internal private changes.

[adiroiban]

Now, there is also the other error reported by László https://github.com/django/daphne/issues/535#issuecomment-2443752769 and this looks like a Twisted bug

  File "/home/karolyi/project/ticketshop/venv/lib/python3.10/site-packages/daphne/http_protocol.py", line 230, in finish
    http.Request.finish(self)
  File "/home/karolyi/project/ticketshop/venv/lib/python3.10/site-packages/twisted/web/http.py", line 1240, in finish
    self._cleanup()
  File "/home/karolyi/project/ticketshop/venv/lib/python3.10/site-packages/twisted/web/http.py", line 966, in _cleanup
    self.channel.requestDone(self)
  File "/home/karolyi/project/ticketshop/venv/lib/python3.10/site-packages/twisted/web/http.py", line 2643, in requestDone
    self._networkProducer.resumeProducing()
AttributeError: 'HTTPChannel' object has no attribute '_networkProducer'

[carltongibson]

This way I can reproduce the bug locally too.

@karolyi I don't suppose you can bisect the issue to a particular commit in twisted can you?

[karolyi]

Sorry but no. I'm not in the position right now where I can put in the effort for it.

Hope I could help with attributing the issue.

[carltongibson]

OK, no problem. I still need to get a reproduce working locally, then I can run git bisect on twisted.

[aDramaQueen]

Sure thing, here you go: daphne_twisted_error

As it turnes out, this is a UNIX socket problem. If you run the example over unix sockets, the error is thrown. If you run it over HTTP everything works?!?

[carltongibson]

Thanks for that @aDramaQueen 👍

[smithdc1]

I tried to bisect this issue. I got it down to the below.

Although looking at this patch I don't quite understand how this relates to the issue here so I could be wrong.

a7bf9331f78b216e1f00322fabb34de64a91ea85 is the first bad commit
commit a7bf9331f78b216e1f00322fabb34de64a91ea85
Author: [REDACTED]
Date:   Tue Sep 10 13:37:40 2024 -0400

    Set TCP_NODELAY on HTTP client and server protocols.

 src/twisted/web/_http2.py     |  3 +++
 src/twisted/web/_newclient.py |  6 +++++-
 src/twisted/web/http.py       | 10 +++++++++-
 3 files changed, 17 insertions(+), 2 deletions(-)

[p12tic]

The way it breaks is that HTTP channel setup breaks when trying to set unsupported TCP_NODELAY on unix socket. The internal state becomes wrong and then code fails again with:

AttributeError: 'HTTPChannel' object has no attribute '_networkProducer'

You can see both errors in this bug report: https://github.com/buildbot/buildbot/issues/8207

[carltongibson]

There's a twisted release candidate that may address this:

https://mail.python.org/archives/list/twisted@python.org/message/QBUMLE32SMC7GX4K4PTN2HNVILH2Q4FK/

If folks could test quickly and feedback that would be helpful.

[hagen00]

Running Twisted==24.11.0rc1 works for me. No errors. I will post on https://github.com/twisted/twisted/issues/12347 as well.