Service discovery not working on ARM Arch Linux

[drunk3nm0nk3y]

Distro: ARM Arch Linux - linux 3.13.2-1 Hardware: ODROID-XU Board Python version: 2.7.6 Twisted version: twisted 13.2.0 Coherence version: 0.6.7 (git/devel)

The installation runs without problems and I can start coherence, but there seems to be a problem with the service discovery. No UPnP-Device can find my mediaserver/backend, but I can acces it through the Web-Browser by using the URL: http://HOST_IP:PORT/BACKEND_UID . Here is the debugging output:

$ coherence -c coherence.config 
No handlers could be found for logger "coherence"
2014-05-27 13:31:48,405 WARNING coherence: Coherence UPnP framework version 0.6.7 starting... (base.py:293)
2014-05-27 13:31:48,409 INFO coherence: running on host: 192.168.2.52 (base.py:317)
2014-05-27 13:31:48,411 WARNING ssdp: There seems to be already a SSDP server running on this host, no need starting a second one. (ssdp.py:58)
2014-05-27 13:31:48,417 WARNING webserver: WebServer on port 30020 ready (base.py:131)
2014-05-27 13:31:48,417 INFO coherence: adding plugin 'DvbStore' (base.py:433)
2014-05-27 13:31:48,440 INFO coherence: Activating DvbStore plugin as MediaServer... (base.py:447)
2014-05-27 13:31:48,716 WARNING mediaserver: MediaServer DvbStore (<coherence.backends.dvb_storage.DvbStore object at 0x9cc570>) activated with id anyUID-will-do (media_server.py:616)
2014-05-27 13:31:48,719 INFO coherence: creating  upnp:rootdevice uuid:anyUID-will-do::upnp:rootdevice (base.py:644)
2014-05-27 13:31:48,720 INFO coherence: creating upnp:rootdevice  uuid:anyUID-will-do::upnp:rootdevice (base.py:646)
2014-05-27 13:31:48,733 DEBUG coherence: SimpleRoot getChild anyUID-will-do, <Request at 0x9b2440 method=GET uri=/anyUID-will-do/description-2.xml clientproto=HTTP/1.0> (base.py:56)
2014-05-27 13:31:48,751 DEBUG coherence: SimpleRoot getChild anyUID-will-do, <Request at 0xb5c0e468 method=GET uri=/anyUID-will-do/ConnectionManager/scpd.xml clientproto=HTTP/1.0> (base.py:56)
2014-05-27 13:31:48,763 DEBUG coherence: SimpleRoot getChild anyUID-will-do, <Request at 0xb5c148f0 method=GET uri=/anyUID-will-do/ContentDirectory/scpd.xml clientproto=HTTP/1.0> (base.py:56)
2014-05-27 13:31:48,801 INFO coherence: adding device uuid:anyUID-will-do uuid:anyUID-will-do::upnp:rootdevice MediaServer (base.py:659)
2014-05-27 13:31:48,806 DEBUG coherence: SimpleRoot getChild anyUID-will-do, <Request at 0xb5c111c0 method=SUBSCRIBE uri=/anyUID-will-do/ContentDirectory/subscribe clientproto=HTTP/1.1> (base.py:56)
2014-05-27 13:31:48,842 DEBUG coherence: SimpleRoot getChild anyUID-will-do, <Request at 0xb5c115a8 method=SUBSCRIBE uri=/anyUID-will-do/ConnectionManager/subscribe clientproto=HTTP/1.1> (base.py:56)
2014-05-27 13:31:48,852 DEBUG coherence: SimpleRoot getChild events, <Request at 0xb5c14e90 method=NOTIFY uri=/events clientproto=HTTP/1.1> (base.py:56)
2014-05-27 13:31:48,861 DEBUG coherence: SimpleRoot getChild events, <Request at 0xb5c14e68 method=NOTIFY uri=/events clientproto=HTTP/1.1> (base.py:56)
2014-05-27 13:31:49,202 DEBUG coherence: SimpleRoot getChild events, <Request at 0xb5c14e68 method=NOTIFY uri=/events clientproto=HTTP/1.1> (base.py:56)
2014-05-27 13:32:39,498 INFO coherence: listchilds / (base.py:73)
2014-05-27 13:32:39,574 DEBUG coherence: SimpleRoot getChild favicon.ico, <Request at 0xb5c144b8 method=GET uri=/favicon.ico clientproto=HTTP/1.1> (base.py:56)
2014-05-27 13:32:39,576 WARNING coherence: Cannot find device for requested name: favicon.ico (base.py:68)
2014-05-27 13:32:45,989 DEBUG coherence: SimpleRoot getChild anyUID-will-do, <Request at 0xb5c1cb20 method=GET uri=/anyUID-will-do clientproto=HTTP/1.1> (base.py:56)
2014-05-27 13:32:58,766 DEBUG coherence: SimpleRoot getChild anyUID-will-do, <Request at 0x9b28c8 method=GET uri=/anyUID-will-do/ContentDirectory clientproto=HTTP/1.1> (base.py:56)
2014-05-27 13:33:02,575 DEBUG coherence: SimpleRoot getChild anyUID-will-do, <Request at 0xb5c1cb20 method=GET uri=/anyUID-will-do/ContentDirectory/control clientproto=HTTP/1.1> (base.py:56)
Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/protocols/basic.py", line 571, in dataReceived
    why = self.lineReceived(line)
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/web/http.py", line 1655, in lineReceived
    self.allContentReceived()
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/web/http.py", line 1730, in allContentReceived
    req.requestReceived(command, path, version)
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/web/http.py", line 826, in requestReceived
    self.process()
--- <exception caught here> ---
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/upnp/core/utils.py", line 254, in process
    self.render(resrc)
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/web/server.py", line 238, in render
    body = resrc.render(self)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/upnp/core/soap_service.py", line 107, in render
    tree = parse_xml(data)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/upnp/core/utils.py", line 56, in parse_xml
    return et_parse_xml(data, encoding)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/extern/et.py", line 138, in parse_xml
    return ET.ElementTree(parser.close())
cElementTree.ParseError: no element found: line 1, column 0

I also tried the newest version of twisted (v14.0), but the result was the same. Any ideas?

[micxer]

This line is the interesting one: 2014-05-27 13:31:48,411 WARNING ssdp: There seems to be already a SSDP server running on this host, no need starting a second one. (ssdp.py:58) I had that before and it's not an issue related to your architecture. Some distributions ship with an already active daemon doing SSDP (like minissdp on Debian). Find that one, kill it and Coherence will run fine.

[drunk3nm0nk3y]

@micxer Thanks for the reply. Well, I have a fresh installation of Arch Linux on the system, which comes without any unnecessary software. Anyway, I checked for any running SSDP Services and there are non. It seems to me that this message

2014-05-27 13:31:48,411 WARNING ssdp: There seems to be already a SSDP server running on this host, no need starting a second one. (ssdp.py:58)

is missleading. I think that the real problem is here:

Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/protocols/basic.py", line 571, in dataReceived
    why = self.lineReceived(line)
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/web/http.py", line 1655, in lineReceived
    self.allContentReceived()
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/web/http.py", line 1730, in allContentReceived
    req.requestReceived(command, path, version)
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/web/http.py", line 826, in requestReceived
    self.process()
--- <exception caught here> ---
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/upnp/core/utils.py", line 254, in process
    self.render(resrc)
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/web/server.py", line 238, in render
    body = resrc.render(self)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/upnp/core/soap_service.py", line 107, in render
    tree = parse_xml(data)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/upnp/core/utils.py", line 56, in parse_xml
    return et_parse_xml(data, encoding)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/extern/et.py", line 138, in parse_xml
    return ET.ElementTree(parser.close())
cElementTree.ParseError: no element found: line 1, column 0

Today I noticed that there is another error message, which occurs when interrupting coherence (Ctr+C):

^CUnhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/internet/defer.py", line 577, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/internet/defer.py", line 839, in _cbDeferred
    self.callback(self.resultList)
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/internet/defer.py", line 382, in callback
    self._startRunCallbacks(result)
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/internet/defer.py", line 490, in _startRunCallbacks
    self._runCallbacks()
--- <exception caught here> ---
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/internet/defer.py", line 577, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/base.py", line 571, in homecleanup
    self.ssdp_server.shutdown()
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/upnp/core/ssdp.py", line 67, in shutdown
    if self.resend_notify_loop.running:
exceptions.AttributeError: 'SSDPServer' object has no attribute 'resend_notify_loop'

[htgoebel]

@drunk3nm0nk3y: Did you try the latest development-version? Can you please put a print repr(data) just before "coherence/upnp/core/soap_service.py", line 107 ( tree = parse_xml(data)) If this gives to much output, frame that line by a try-except cElementTree.ParseError claause and print data in the except-part (and reraise) Then post the printed output. Thanks.

[drunk3nm0nk3y]

1) @htgoebel : Yes it's the latest development-version. I added the print command 'print repr(data)' like you told me, but it did not add any additional output. Seems like the line is not being called.

2) I found out that this error:

Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/protocols/basic.py", line 571, in dataReceived
    why = self.lineReceived(line)
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/web/http.py", line 1655, in lineReceived
    self.allContentReceived()
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/web/http.py", line 1730, in allContentReceived
    req.requestReceived(command, path, version)
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/web/http.py", line 826, in requestReceived
    self.process()
--- <exception caught here> ---
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/upnp/core/utils.py", line 254, in process
    self.render(resrc)
  File "/usr/lib/python2.7/site-packages/Twisted-13.2.0-py2.7-linux-armv7l.egg/twisted/web/server.py", line 238, in render
    body = resrc.render(self)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/upnp/core/soap_service.py", line 108, in render
    tree = parse_xml(data)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/upnp/core/utils.py", line 56, in parse_xml
    return et_parse_xml(data, encoding)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/extern/et.py", line 138, in parse_xml
    return ET.ElementTree(parser.close())
cElementTree.ParseError: no element found: line 1, column 0

is only triggered by opening http://HOST_IP:30020/BACKEND_UID/ContentDirectory/control I think that it has nothing to do with the actual SSDP problem. Maybe we can ignore this for now.

3) I wanted to see whats going wrong in ssdp.py, so I added a self.warning(err) in line 58 of /upnp/core/ssdp.py. This is the error message:

WARNING ssdp: Couldn't listen on any:1900: [Errno 92] Protocol not available. (ssdp.py:58)
WARNING ssdp: There seems to be already a SSDP server running on this host, no need starting a second one. (ssdp.py:59)

So it seems like SSDP can't listen on the port 1900. However port 1900 is free (confirmed this with 'netstat')

[htgoebel]

Hmm, Errno 92 is not very expressive :-(

Please check if the kernel headers on your system are matching the actually running kernel (according to http://bugs.python.org/issue19901#msg205343) and if you compiled Python yourself: if the headers used match the actual kernel.

[drunk3nm0nk3y]

Thanks for the information, I checked the kernel and header version and everything seems to be fine. The only thing I'm not sure about are the linux-api-headers, which don't match the kernel and kernel-headers version. However the linux-api-headers are installed as an dependency for glibc, so I think that they have nothing to do with python, coherence or twisted. Here is a listing of the installed packages

local/libutil-linux 2.24.1-6 (base base-devel)
    util-linux runtime libraries
local/linux-api-headers 3.13.2-1
local/linux-firmware 20140316.dec41bc-1
local/linux-odroid-xu 3.4.91-1
local/linux-odroid-xu-headers 3.4.91-1
    Header files and scripts for building modules for linux kernel - ODROID-XU
local/util-linux 2.24.1-6 (base base-devel)

Btw. opening sockets/ports and binding to them by using the socket-api works fine with other python applications.

I wanted to have more useful output on the terminal, so I uncommented the try/except in the previously mentioned lines of "ssdp.py". Here is the output:

$ coherence -c ~/coherence.config 
2014-06-05 12:25:28,202 WARNING coherence: Coherence UPnP framework version 0.6.7 starting... (base.py:293)
Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 416, in fireEvent
    DeferredList(beforeResults).addCallback(self._continueFiring)
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 306, in addCallback
    callbackKeywords=kw)
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 295, in addCallbacks
    self._runCallbacks()
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
--- <exception caught here> ---
  File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 429, in _continueFiring
    callable(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/EGG-INFO/scripts/coherence", line 70, in main
    c = Coherence(config)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/base.py", line 227, in __new__
    obj.setup(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/base.py", line 313, in setup
    self.setup_part2()
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/base.py", line 327, in setup_part2
    self.ssdp_server = SSDPServer(test=unittest, interface=self.hostname)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/upnp/core/ssdp.py", line 50, in __init__
    self._port = reactor.listenMulticast(SSDP_PORT, self, listenMultiple=True)
  File "/usr/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 387, in listenMulticast
    p.startListening()
  File "/usr/lib/python2.7/site-packages/twisted/internet/udp.py", line 175, in startListening
    self._bindSocket()
  File "/usr/lib/python2.7/site-packages/twisted/internet/udp.py", line 195, in _bindSocket
    raise error.CannotListenError(self.interface, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on any:1900: [Errno 92] Protocol not available.

What makes me curious is that it says "Couldn't listen on any:1900", does "any" stand for any network interface, is it okay that it says "any" and not localhost or anything similar?

[htgoebel]

Me fool. I did not seen the any:1900 earlier. Of course, any is not a valid interface name. But where does it come from? Can you please 1) add print interface to ssdp.py, line 47 (just before the joinGroup line) 2) search your config for any and/or whatever that print-statement tells you.

(If you like, you can try the most recent version, I just added logging the CannotListenError.)

[drunk3nm0nk3y]

Sry for answering so late. I had to reinstall the entire system. However that's a good thing, because the previous installation was made by an research assistant at the university and with this fresh installation I am absolutely sure that everything is configured properly. I also pulled the latest changes from the development branch. I was hoping that the fresh installation would change anything, but sadly the error is still the same.

1) Had to insert the print interface before self.port = reactor.listenMulti..., because this line causes the exception. Code snippet from ssdp.py:

            #try:
            print '------------'
            print interface
            print '------------'
            self.port = reactor.listenMulticast(SSDP_PORT, self, listenMultiple=True)
            #self.port.setLoopbackMode(1)
            self.port.joinGroup(SSDP_ADDR, interface=interface)

The output looks like this:

$ coherence -c ~/coherence.config 
2014-06-11 12:11:05,553 WARNING coherence: Coherence UPnP framework version 0.6.7 starting... (base.py:292)
------------
192.168.2.54
------------
Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 416, in fireEvent
    DeferredList(beforeResults).addCallback(self._continueFiring)
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 306, in addCallback
    callbackKeywords=kw)
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 295, in addCallbacks
    self._runCallbacks()
  File "/usr/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
--- <exception caught here> ---
  File "/usr/lib/python2.7/site-packages/twisted/internet/base.py", line 429, in _continueFiring
    callable(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/EGG-INFO/scripts/coherence", line 70, in main
    c = Coherence(config)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/base.py", line 226, in __new__
    obj.setup(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/base.py", line 312, in setup
    self.setup_part2()
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/base.py", line 326, in setup_part2
    self.ssdp_server = SSDPServer(test=unittest, interface=self.hostname)
  File "/usr/lib/python2.7/site-packages/Coherence-0.6.7-py2.7.egg/coherence/upnp/core/ssdp.py", line 48, in __init__
    self.port = reactor.listenMulticast(SSDP_PORT, self, listenMultiple=True)
  File "/usr/lib/python2.7/site-packages/twisted/internet/posixbase.py", line 387, in listenMulticast
    p.startListening()
  File "/usr/lib/python2.7/site-packages/twisted/internet/udp.py", line 175, in startListening
    self._bindSocket()
  File "/usr/lib/python2.7/site-packages/twisted/internet/udp.py", line 195, in _bindSocket
    raise error.CannotListenError(self.interface, self.port, le)
twisted.internet.error.CannotListenError: Couldn't listen on any:1900: [Errno 92] Protocol not available.

As you can see interface is actually my correct local IP address

2) I checked my config-file and it does not contain the word any. Here is the section about the networking stuff (wlan0 is the correct interface name!):

<controlpoint>yes</controlpoint>
<use_dbus>no</use_dbus>
<web-ui>no</web-ui>
<serverport>30020</serverport>
<interface active="yes">wlan0</interface>

Btw. setting interface active to no doesn't change anything.

[htgoebel]

This looks bad :-( Can you please check if your kernel support multicast? AFAIK you should get some output on this commands:

grep _MROUTE /boot/config
sysctl -a | grep mcast

[drunk3nm0nk3y]

I tried the first command you mentioned, but there is no /boot/config/ in Arch Linux. I suppose an equivalent thing to do would be checking the config of the kernel source:

$ grep CONFIG_IP_MULTICAST /usr/src/linux-3.4.91-2-ARCH/.config 
CONFIG_IP_MULTICAST=y

so multicast seems to be enabled.

Here are the results of the second command, looks good to me:

$ sudo sysctl -a | grep mcast
net.ipv4.neigh.default.mcast_solicit = 3
net.ipv4.neigh.eth0.mcast_solicit = 3
net.ipv4.neigh.lo.mcast_solicit = 3
net.ipv4.neigh.wlan0.mcast_solicit = 3
net.ipv6.neigh.default.mcast_solicit = 3
net.ipv6.neigh.eth0.mcast_solicit = 3
net.ipv6.neigh.lo.mcast_solicit = 3
net.ipv6.neigh.wlan0.mcast_solicit = 3

I have the same output on an other machine with a working installation of coherence.

I also did a netstat -g to check the multicast group memberships of the devices:

$ netstat -g
IPv6/IPv4 Group Memberships
Interface       RefCnt Group
--------------- ------ ---------------------
lo              1      all-systems.mcast.net
eth0            1      all-systems.mcast.net
wlan0           1      all-systems.mcast.net
lo              1      ff02::1
eth0            1      ff02::1
wlan0           1      ff02::1:ff17:99ee
wlan0           1      ff02::1

[drunk3nm0nk3y]

Since this bug is so mysterious, I started to mess around a little bit with ssdp.py and doing some try and error tests. As we know by now this is the line causing the error:

self.port = reactor.listenMulticast(SSDP_PORT, self, listenMultiple=True)

The twisted wiki describes the listenMultiple parameter like this: "If set to True, allows multiple sockets to bind to the same address and port number at the same time"

So I set listenMultiple to False just to see if the outcome differs and it actually did the trick. There is no more error. The important question is: Is it really necessary to set listenMultiple=True, do we really have multiple sockets binding to port 1900? Is this really a valid fix/workaround?

[htgoebel]

I'm afraid I can not answer your questions. The source history doe not give any hints on this. I would appreciate if you could recherché this.

[drunk3nm0nk3y]

Hi and sry for the late answer. I will definitely recherché this and also test if there are any drawbacks by turning 'listenMultiple' off. I will report back as soon as I know more. Again thanks for your support!

[unintended]

this is a bug of twisted. Fixed in 14.0 https://github.com/twisted/twisted/commit/a2e902684761d8a71d455bd43e74022a1f4d6996

[drunk3nm0nk3y]

Hey, it's been a long time. I will close this issue, since this seems to be a bug in twisted 14.0.

@htgoebel I just wanted to thank you for all the time you put into maintaining coherence and your grate support

[htgoebel]

@drunk3nm0nk3y Thanks :-)