search

michaelarnauts / home-assistant-comfoconnect

Home Assistant integration for Zehnder ComfoAir trough a Zehnder ComfoConnect LAN C.
Other
34 stars 13 forks source link

Integration doesn't always properly reconnect to the bridge #43

Open michaelarnauts opened 7 months ago

michaelarnauts commented 7 months ago

Since there are a lot of similar issues, I'll create a new issue and link them all to this one.

This integration sometimes doesn't reconnect properly after the connection was dropped to the bridge. The exact reason for these disconnects are unknown, but it might be related to one of those:

Since these disconnects happen, it is important that we have a proper reconnect logic in place. Unfortunately, when the bridge disconnects (maybe it just crashes and restarts?), it seems it doesn't properly close the TCP connection, so we don't know that we got disconnected until we make a request and don't get a reply in due time. That's why there is a keepalive that is send every 30 seconds.

More information about the new reconnect logic is written here: https://github.com/michaelarnauts/aiocomfoconnect/pull/23

Please try the latest code in Home Assistant with HACS. I've created a release for 0.1.9 for the release before these changes, in case you want to rollback. If you want to test the new reconnect logic, please select the "master" branch in HACS when (re)installing.

image

michaelarnauts commented 7 months ago

Related issues:

freemann commented 7 months ago

Awesome! Thanks for both of your time, very very appreciated!!

Did the update, 4 new sensors were added and I enabled all 38 sensors.

[edit] The new sensors are from the ComfoFond which we don't have :)

zollak commented 7 months ago

Thank you very much for your work! I updated it and now it connects properly. I will let you know if there are any issues, but it will probably be fine now! Again, nice job @litinoveweedle and @michaelarnauts !

joshuavandermeulen commented 7 months ago

Thank you. I will start testing the updated Hacs integration today.

freemann commented 7 months ago

For now it looks rock solid! Very nice job guys!!

[edit 03-04-2024 07:40] Still going strong!

joshuavandermeulen commented 7 months ago

Having some issues here. The fan is not updating it's state after +- one day. But when pushing the fan speed button it is strangely working but still gives an error message. Ps. DHCP lease time is set very high.

litinoveweedle commented 7 months ago

Hello all, first big thanks to @michaelarnauts! All the credits for the fix belongs to him. I did only learn a lot from him during last few days ;-)

As you are probably aware, the fix provided solves situation where your HA looses connection the the ComfoConnect gateway. In the past, the integration was not able to handle such state completely gracefully, without raising nuch of errors and warnings. As about the root cause, I was finally able to find it and fix it also based on information provided by @michaelarnauts.

Disclaimer: I did never used phone app with my gateway, as I didn't had a reason to do so. Also my gateway is not connected to the network with internet access - so no cloud connection. But due to this I was not aware, that there is new updated firmware - as you get firmware update prompt during registering of your gateway with mobile application.

As I discovered original shipped firmware version U1.1.10 was having following issue: My standard DHCP server has lease time 10 minutes. 5 minutes after getting IP (DHCP ACK) gateway always asked for DHCP renew (DHCP RENEW). At this moment, even if the gateway kept responding to the ping, it immediately ceases all data communication to ANY client (both Home Assistant as well as mobile app). After 5-20 sec Comfoconnect HA integration timeout, raised exception and restarted communication. At that time gateway is already able to reestablish communication - but only to repeat whole story 5 minutes later. I did validated this behavior with tcpdump and also by increasing DHCP lease time.

I was able to solve this DHCP renew related disconnect by upgrading my gateway to firmware U1.5.1. This can be done from mobile app. Please see firmware release notes:

So for anyone previously having disconnect/reconnect issue please check your firmware version and also network stability.

After firmware upgrade and update of the integration everything works perfect for last few days. I am also planning to simulate some network instability artificially to test improvements in the integration - as right now my connection is rock stable.

Again huge thanks to @michaelarnauts!

joshuavandermeulen commented 7 months ago

This is not stable at all for me. After one day it is not possible anymore to control the comfoAirQ. I'm returning to the default comfoconnect integration in home assistant. @michaelarnauts @litinoveweedle any chance that you want to offer remote support for some compensation of course...? This problem is driving me nuts for months now

litinoveweedle commented 7 months ago

Ok, after several more days of using integration I had almost no issues. That is probably mostly due to removing the root cause of the instability (by firmware upgrade). But I observe really crazy chain of exceptions to be triggered, which required HA restart. Something went really crazy and I did not tried to trigger this, it simple happened somehow....

I am attaching filtered HA log:

home-assistant.log

litinoveweedle commented 7 months ago

This is not stable at all for me. After one day it is not possible anymore to control the comfoAirQ. I'm returning to the default comfoconnect integration in home assistant. @michaelarnauts @litinoveweedle any chance that you want to offer remote support for some compensation of course...? This problem is driving me nuts for months now

Would it be please possible to describe your problem in more technical manners? Did you checked your FW version? Do you have errors in the home assistant log? Please try to collect more information first, than lets see what we can do to help.

roofburner commented 7 months ago

My integration is running fine. Fingers crossed.

Some "specs":

Home Assistant (docker) --> Core 2024.4.1 --> Frontend 20240307.0

Zehnder ComfoAirQ integration via HACS --> v0.1.9

ComfoAir Q350 B R ST ERV Prem --> Firmware: R1.11.0 ComfoConnect LAN C --> Firmware: R1.5.1

@litinoveweedle Hope this helps?

michaelarnauts commented 7 months ago

My integration is running fine. Fingers crossed.

Some "specs":

Home Assistant (docker) --> Core 2024.4.1 --> Frontend 20240307.0

Zehnder ComfoAirQ integration via HACS --> v0.1.9

ComfoAir Q350 B R ST ERV Prem --> Firmware: R1.11.0 ComfoConnect LAN C --> Firmware: R1.5.1

@litinoveweedle Hope this helps?

v0.1.9 is still the old version. You'll need to select master to have the latest changes.

litinoveweedle commented 7 months ago

v0.1.9 is still the old version. You'll need to select master to have the latest changes.

And that unfortunately goes as well for me, mea culpa. I just installed 'Master' branch. So please disregard my previous report.

freemann commented 7 months ago

Here it's working flawlessly.

Core > 2024.4.2
Supervisor > 2024.04.0
Operating System > 12.1
Frontend > 20240404.1
Zehnder ComfoAirQ integration > Master
From log "Attempting install of aiocomfoconnect==0.1.9"
ComfoAir Q350 > Firmware: R1.11.0 
ComfoConnect LAN C > Firmware: R1.5.1

I added a +/- 5h log from yesterday. home-assistant_comfoconnect_2024-04-10T12-21-05.800Z.log

roofburner commented 7 months ago

I just switched to the 'master' version. Thanks for the hint!

michaelarnauts commented 7 months ago

I'll try to create a 0.1.10 version or something to avoid more confusion.

joshuavandermeulen commented 7 months ago

This is not stable at all for me. After one day it is not possible anymore to control the comfoAirQ. I'm returning to the default comfoconnect integration in home assistant. @michaelarnauts @litinoveweedle any chance that you want to offer remote support for some compensation of course...? This problem is driving me nuts for months now

Would it be please possible to describe your problem in more technical manners? Did you checked your FW version? Do you have errors in the home assistant log? Please try to collect more information first, than lets see what we can do to help.

I will try... :) I was not on the Hacs integration for a long time since it was not stable for me. After changing DHCP lease to 10 days on my router I got it working quiet stable on the default confoconnect integration for sometimes more than a week (probably these 10 days) allthough at some moments it was losing connection earlier. After hearing about the new HACS version I switched back to it, also changed the DHCP lease time back to 1 day to test. Quiet fast (less then a day later) I noticed that the fan state was not updating anymore in Home Assistant GUI but it was still controllable manually (that is different from the default comfoconnect integration). After 1 day it was also not controllable anymore. This is why again I switched back to the default integration. Since my DHCP lease was still on 1 day, today it lost connection again, so that is definitely (one of) the issue(s).

What I see in the logs now (still on the default integration): image

I will switch back now to the HACS integration and try to collect some logs. Should I turn on 'enable defug logging' and collect these logs?

Here are some details about my configuration:

Home Assistant (Synology VM) --> Core 2024.4.1 Zehnder ComfoAirQ integration via HACS --> v0.1.9 ComfoAir Q350 --> Firmware: R1.11.0 (most recent firmware) ComfoConnect LAN C --> Firmware: R1.5.1 (most recent firmware)

joshuavandermeulen commented 7 months ago

I did not get the master version as well. So now starting to test the master version.

edit: I noticed that installing the master version from scratch it is not possible to connect to the ComfoAir system (using GUI). So I first downloaded 0.1.9 again, connected it and after that did a 'redownload' to the master version + restart

michaelarnauts commented 7 months ago

To avoid all the confusion, I've released 0.1.10. It's easy to go back to the previous release anyway.

joshuavandermeulen commented 7 months ago

The home assistant log is filled with errors...

Logger: homeassistant Source: /usr/src/homeassistant/homeassistant/runner.py:146 First occurred: 10:09:59 PM (412 occurrences) Last logged: 10:39:35 PM

Error doing job: Exception in callback ComfoConnect._unhold_sensors() Traceback (most recent call last): File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/comfoconnect.py", line 68, in _unhold_sensors if self._sensors_values[sensor_id] is not None:


KeyError: 65

Logger: homeassistant
Source: /usr/src/homeassistant/homeassistant/runner.py:146
First occurred: 10:21:03 PM (91 occurrences)
Last logged: 10:39:34 PM

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/bridge.py", line 108, in _read_messages
    await self._process_message()
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/bridge.py", line 239, in _process_message
    self._event_bus.emit(message.cmd.reference, message.msg)
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/bridge.py", line 56, in emit
    del self.listeners[event_name]
        ~~~~~~~~~~~~~~^^^^^^^^^^^^
KeyError: 1

Logger: homeassistant
Source: /usr/src/homeassistant/homeassistant/runner.py:146
First occurred: 10:07:37 PM (1556 occurrences)
Last logged: 10:39:34 PM

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/bridge.py", line 213, in _process_message
    message = await self._read()
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/bridge.py", line 176, in _read
    msg_len_buf = await self._reader.readexactly(4)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/streams.py", line 750, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 4 expected bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/bridge.py", line 108, in _read_messages
    await self._process_message()
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/bridge.py", line 247, in _process_message
    raise AioComfoConnectNotConnected
aiocomfoconnect.exceptions.AioComfoConnectNotConnected

Logger: aiocomfoconnect.bridge
Source: runner.py:189
First occurred: 10:06:47 PM (1724 occurrences)
Last logged: 10:39:29 PM

Timeout while connecting to bridge 192.168.0.35
Timeout while waiting for response from bridge
Failed to decode message: Error parsing message

Logger: homeassistant
Source: /usr/src/homeassistant/homeassistant/runner.py:146
First occurred: 10:39:22 PM (1 occurrences)
Last logged: 10:39:22 PM

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/comfoconnect.py", line 79, in _reconnect_loop
    read_task = await self._connect(uuid)
                ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/bridge.py", line 95, in _connect
    self._reader, self._writer = await asyncio.wait_for(asyncio.open_connection(self.host, self.PORT), TIMEOUT)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/streams.py", line 48, in open_connection
    transport, _ = await loop.create_connection(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1120, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1102, in create_connection
    sock = await self._connect_sock(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1005, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.12/asyncio/selector_events.py", line 641, in sock_connect
    return await fut
           ^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/selector_events.py", line 681, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('192.168.0.35', 56747)

Logger: homeassistant.helpers.entity
Source: helpers/entity.py:951
First occurred: 10:10:22 PM (36 occurrences)
Last logged: 10:38:52 PM

Update for select.comfoairq_balance_mode fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 951, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1268, in async_device_update
    await self.async_update()
  File "/config/custom_components/comfoconnect/select.py", line 199, in async_update
    self._attr_current_option = await self.entity_description.get_value_fn(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/comfoconnect.py", line 323, in get_balance_mode
    result_06 = await self.cmd_rmi_request(bytes([0x83, UNIT_SCHEDULE, SUBUNIT_06, 0x01]))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/bridge.py", line 137, in _send
    raise AioComfoConnectNotConnected
aiocomfoconnect.exceptions.AioComfoConnectNotConnected
roofburner commented 7 months ago

Can you please check/share/confirm the versions you use?

michaelarnauts commented 7 months ago

I have never seen this:

ConnectionRefusedError: [Errno 111] Connect call failed ('192.168.0.35', 56747)

I would guess your bridge is running an old Firmware, or you have some odd network issues going on?

joshuavandermeulen commented 7 months ago

I have never seen this:

ConnectionRefusedError: [Errno 111] Connect call failed ('192.168.0.35', 56747)

I would guess your bridge is running an old Firmware, or you have some odd network issues going on?

No it is not old firmware.... And no odd network issues that I know off. I'm using all ubiquiti unify network devices. Also the Comfo control app is working flawlessly.

Home Assistant (Synology VM) --> Core 2024.4.1 Zehnder ComfoAirQ integration via HACS --> v0.1.10 ComfoAir Q350 --> Firmware: R1.11.0 (most recent firmware) ComfoConnect LAN C --> Firmware: R1.5.1 (most recent firmware)

freemann commented 7 months ago

I have never seen this:

ConnectionRefusedError: [Errno 111] Connect call failed ('192.168.0.35', 56747)

https://github.com/michaelarnauts/home-assistant-comfoconnect/issues/9#issuecomment-1526189226 last logline

michaelarnauts commented 7 months ago

I have never seen this: ConnectionRefusedError: [Errno 111] Connect call failed ('192.168.0.35', 56747)

#9 (comment) last logline

Thanks! But anyway, I have no clue what could be going on. I understand this can be frustrating, but remember I'm just another user like all of you. I don't work for Zehnder and I need to carefully plan the free time I have.

Feel free to experiment with network settings, cable lengths or any other changes you can try to pinpoint the cause. It might even be an idea to contact Zehnder and ask them why the bridge is disconnecting?

joshuavandermeulen commented 7 months ago

What I tried today is replacing the interface cable to the lan c module. I noticed that Zehnder has a specific recommendation on what cable it needs to be. And the cable that I used was not the same. There might have been an instability because of that resulting in unusual errors... Fingers crossed.

litinoveweedle commented 7 months ago

@joshuavandermeulen would you be able to capture traffic to/from gateway on your HA interface using tcpdump? It would be great to see, what is going on.

joshuavandermeulen commented 7 months ago

No clue how to do this... First testing the default home assistant integration with the new cable. If that is stable (so far so good) I wil try the hacs integration again and check the logs

JohnyDeejay commented 7 months ago

My integration is running fine. Fingers crossed.

Some "specs":

Home Assistant (docker) --> Core 2024.4.1 --> Frontend 20240307.0

Zehnder ComfoAirQ integration via HACS --> v0.1.9

ComfoAir Q350 B R ST ERV Prem --> Firmware: R1.11.0 ComfoConnect LAN C --> Firmware: R1.5.1

@litinoveweedle Hope this helps?

How did you update the Q350 firmware? I'm using Q450 SI TR ERV but the latest firmware is R1.10.0

roofburner commented 7 months ago

I have a personal Comfoconnect Cloud account to use in the App

joshuavandermeulen commented 7 months ago

You can request an account with an e-mail to support@zehnder.be

litinoveweedle commented 7 months ago

I would not expect unit FW to have direct impact on the issue. Regarding how to use tcpdump, it depends on the type of the HA installation.

As I would expect you are using passion OS, then you can find the way how to install it here:

https://community.home-assistant.io/t/tcpdump-installation/250287/4

Tcpdump should help to observe and understand network traffic between HA and getaway. At least in my case it was an eye opener. If anyone would need help we can do it together remotely.

michaelarnauts commented 7 months ago

You can request an account with an e-mail to support@zehnder.be

Is this for an installer account so you can do firmware updates?

joshuavandermeulen commented 7 months ago

You can request an account with an e-mail to support@zehnder.be

Is this for an installer account so you can do firmware updates?

Yes

zollak commented 6 months ago

Here are some details about my configuration:

Home Assistant (HP ProDesk 400 G4, Proxmox, HA OS VM) --> OS Version: Home Assistant OS 12.2 --> Home Assistant Core: 2024.4.3 Zehnder ComfoAirQ integration via HACS --> "version": "0.1.10-dev" -> still the previous "master" version that was updated 2 weeks ago. ComfoAir Q350 SI TR ERV --> Firmware: R1.9.0 (I have started to get a registered installer account) ComfoConnect LAN C --> Firmware: R1.5.1 DHCP lease time for LAN C: --> 44 days

as I see, I have no issues in the last two weeks, when I updated to the "master" version... I hope it helps.

One more thing regarding the network: I use many VLANs. The Proxmox host has only one physical network card, so I use trunk port on my managed switch for it. As my network expanded, I added more and more VLANs, which appeared as a new virtual interface in the HA OS... The interesting thing was that every network card also had a default route, which resulted in more network "default" routes (because HA OS should have only 1 default route, i.e. gateway). This caused many network anomalies, because HA sometimes sent out a packet in one direction, but there, e.g. the firewall blocked it, while other times, when the package went in the right direction by chance, it managed to reach its destination. When restarting the HA, the default route of the HA sometimes changed arbitrarily... I had to manually delete the unnecessary default routes, so the route table finally works well based on the networks address for each interfaces. If someone is also using several VLANs in a similar way, I recommend that you look into it! ➜ ~ route -n

If you need more info about this issue and its solution, let me know, I'm happy to give more details.

joshuavandermeulen commented 6 months ago

All problems with my installation were resolved by replacing the cable between my comfoconnect lan c module and the Zehnder ventilation system with a single core cable (instead of a multi core). There must have been connection issues between the two devices... It is stable for 2 weeks now.

litinoveweedle commented 6 months ago

Hello guys, anyone faced this message in HA recently?

2024-04-20 16:02:02.369 WARNING (Recorder) [homeassistant.components.recorder.util] The system could not validate that the sqlite3 database at //config/home-assistant_v2.db was shutdown cleanly
2024-04-20 16:02:02.384 WARNING (Recorder) [homeassistant.components.recorder.util] Ended unfinished session (id=565 from 2024-04-19 23:51:45.488175)

https://github.com/home-assistant/core/issues/73679#issuecomment-2067702303

Its appear after HA restart, but only if HA was running at least for few hours. (need to test more to find more precise timeframe).

The working hypothesis is , that some integration is blocking main loop so recorder could not correctly close open SQLite database. As there was a lot of work around threading in comfoconnect recently I just want to check with you first. I am unfortunately not precisely aware when this appeared...

litinoveweedle commented 5 months ago

I am still troubleshooting my HA shutdown issues and I observed random exceptions raised by integration during HA shutdown.

On Shutdown

2024-06-11 11:20:34.185 INFO (MainThread) [aiocomfoconnect.bridge] The connection was closed.
2024-06-11 11:20:34.203 INFO (MainThread) [aiocomfoconnect.comfoconnect] We got disconnected. Reconnecting.
.....
2024-06-11 11:20:34.324 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved (None):   File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 223, in <module>
    sys.exit(main())
  File "/usr/src/homeassistant/homeassistant/__main__.py", line 209, in main
    exit_code = runner.run(runtime_conf)
  File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run
    return loop.run_until_complete(setup_and_run_hass(runtime_config))
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 672, in run_until_complete
    self.run_forever()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 639, in run_forever
    self._run_once()
  File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1980, in _run_once
    handle._run()
  File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
    self._context.run(self._callback, *self._args)
  File "/usr/src/homeassistant/homeassistant/helpers/event.py", line 1603, in _interval_listener
    self.hass.async_run_hass_job(self._run_job, dt_util.utcnow(), background=True)
  File "/usr/src/homeassistant/homeassistant/core.py", line 939, in async_run_hass_job
    return self._async_add_hass_job(hassjob, *args, background=background)
  File "/usr/src/homeassistant/homeassistant/core.py", line 748, in _async_add_hass_job
    task = create_eager_task(
  File "/usr/src/homeassistant/homeassistant/util/async_.py", line 37, in create_eager_task
    return Task(coro, loop=loop, name=name, eager_start=True)
  File "/config/custom_components/comfoconnect/__init__.py", line 159, in send_keepalive
    await bridge.connect(entry.data[CONF_LOCAL_UUID])
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/comfoconnect.py", line 116, in connect
    reconnect_task = self._loop.create_task(_reconnect_loop())
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/comfoconnect.py", line 101, in _reconnect_loop
    await read_task
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/bridge.py", line 113, in _read_messages
    await self._process_message()
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/bridge.py", line 218, in _process_message
    message = await self._read()
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/aiocomfoconnect/bridge.py", line 181, in _read
    msg_len_buf = await self._reader.readexactly(4)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/asyncio/streams.py", line 752, in readexactly
    await self._wait_for_data('readexactly')
  File "/usr/local/lib/python3.12/asyncio/streams.py", line 531, in _wait_for_data
    raise RuntimeError(
RuntimeError: readexactly() called while another coroutine is already waiting for incoming data