Feature request: Network connection timeout / ability to reconnect to meshtastic + matrix

RipperFox commented 7 months ago

Hi there, I had the gateway running for a while with a Lora32 module accessed via WiFi, but had two mayor problems:

The connection to the Meshtastic devices sometimes somehow takes too long and times out. After 3 times connection attempts are aborted and the rest of the code tries to use a non existent mesh connection without any status output e.g. to Matrix and no attempt to ever connect again
Also: If the connection at the start succeeds, but the TCP connection to the Meshtatsic device is lost later, there are no reconnect attempts made.

Keep up the good work!

Best regards,

Ripper

jeremiah-k commented 6 months ago

We appreciate the feedback. When you have time, would you mind pulling the latest changes in main and testing to see if it reconnects properly now? Or if you're using the Windows packages let me know and I'll create a pre-release.

Thanks.

RipperFox commented 6 months ago

Thanks for your effort. I pulled the update, updated meshtastic python module to 2.3.0 and ran it:

2024-03-12 09:30:41 +0000 INFO:Meshtastic:Connecting to host 192.168.1.225 ...
2024-03-12 09:31:11 +0000 WARNING:Meshtastic:Attempt #1 failed. Retrying in 2 secs... Timed out waiting for connection completion
2024-03-12 09:31:17 +0000 INFO:Meshtastic:Connected to 6820 / HELTEC_V3
2024-03-12 09:31:18 +0000 DEBUG:Plugin:health:Started with priority=10
2024-03-12 09:31:18 +0000 DEBUG:Plugin:map:Started with priority=10
2024-03-12 09:31:18 +0000 DEBUG:Plugin:help:Started with priority=10
2024-03-12 09:31:19 +0000 INFO:Matrix:Connecting ...
2024-03-12 09:31:20 +0000 INFO:Matrix:Joined room '#localmeshtastic:127.0.0.1' successfully
2024-03-12 09:31:20 +0000 INFO:Meshtastic:Listening for inbound radio messages ...
2024-03-12 09:31:20 +0000 INFO:Matrix:Listening for inbound matrix messages ...
2024-03-12 09:31:20 +0000 INFO:Matrix:Syncing with server...

Then manually restarted the Heltec Board - TCP connection got terminated & the reconnection occured:

ERROR:root:Unexpected OSError, terminating meshtastic reader... [Errno 104] Connection reset by peer
2024-03-12 10:38:53 +0000 ERROR:Meshtastic:Lost connection. Reconnecting...
2024-03-12 10:38:53 +0000 INFO:Meshtastic:Connecting to host 192.168.1.225 ...
2024-03-12 10:39:23 +0000 WARNING:Meshtastic:Attempt #1 failed. Retrying in 2 secs... Timed out waiting for connection completion
2024-03-12 10:39:28 +0000 INFO:Meshtastic:Connected to 6820 / HELTEC_V3
Exception in thread Thread-30:
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 1394, in run
    self.function(*self.args, **self.kwargs)
  File "/home/relay/meshtastic-matrix-relay/.venv/lib/python3.11/site-packages/meshtastic/mesh_interface.py", line 632, in callback
    self._sendToRadio(p)
  File "/home/relay/meshtastic-matrix-relay/.venv/lib/python3.11/site-packages/meshtastic/mesh_interface.py", line 690, in _sendToRadio
    self._sendToRadioImpl(toRadio)
  File "/home/relay/meshtastic-matrix-relay/.venv/lib/python3.11/site-packages/meshtastic/stream_interface.py", line 115, in _sendToRadioImpl
    self._writeBytes(header + b)
  File "/home/relay/meshtastic-matrix-relay/.venv/lib/python3.11/site-packages/meshtastic/tcp_interface.py", line 72, in _writeBytes
    self.socket.send(b)
BrokenPipeError: [Errno 32] Broken pipe

Last exception continually occurs.. Seems the library won't open the TCP connection again but still tries to send something to the radio.. Didn't test with real radio comms yet..

geoffwhittington commented 6 months ago

Thanks for testing @RipperFox

I developed using meshtastic==2.2.7 but based on your feedback I installed meshtastic==2.3.0. I reproduced the Broken pipe messages but in my testing I am able to reconnect. I am rebooting the device in order to mimic an outage. How are you testing?

A notable difference between my setup and yours is:

I am using a T-Beam (maybe there's a firmware difference)
I am using meshtastic.local for the device host name. Since you have your IP hard-coded there may be a chance when the device reconnects to the network that it gets a different IP.

jeremiah-k commented 2 months ago

@RipperFox

When you have some time, would try the recent changes in main? I think I finally have it reconnecting reliably with the network and serial interfaces.

jeremiah-k commented 2 months ago

Closing as completed for serial and network interfaces.

geoffwhittington / meshtastic-matrix-relay

Feature request: Network connection timeout / ability to reconnect to meshtastic + matrix #62