search

mautrix / telegram

A Matrix-Telegram hybrid puppeting/relaybot bridge
https://matrix.to/#/#telegram:maunium.net
GNU Affero General Public License v3.0
1.34k stars 175 forks source link

Exponential backoff for reconnection attempts #299

Closed WGH- closed 5 years ago

WGH- commented 5 years ago

Giving up reconnection attempts forever after less than 1 minute of failures doesn't seem like a sane default for a program that is expected to run unattended.

I think there should be exponential backoff capped at 1 hour (or maybe even less).

aiohttp==3.5.4
alembic==1.0.7
async-generator==1.10
async-timeout==3.0.1
attrs==18.2.0
chardet==3.0.4
commonmark==0.8.1
cryptg==0.1.0.7
decorator==4.3.2
future==0.17.1
future-fstrings==0.4.5
idna==2.8
idna-ssl==1.1.0
imageio==2.4.1
imageio-ffmpeg==0.2.0
lxml==4.3.1
Mako==1.0.7
MarkupSafe==1.1.0
mautrix-appservice==0.3.7
mautrix-telegram==0.4.0.post1
moviepy==0.2.3.5
multidict==4.5.2
numpy==1.16.1
Pillow==5.4.1
pyaes==1.6.1
pyasn1==0.4.5
python-dateutil==2.8.0
python-editor==1.0.4
python-magic==0.4.15
rsa==4.0
ruamel.yaml==0.15.87
six==1.12.0
SQLAlchemy==1.2.17
Telethon==1.2
telethon-session-sqlalchemy==0.2.6
tqdm==4.31.1
typing-extensions==3.7.2
yarl==1.3.0
[2019-03-15 10:29:33,068] [WARNING@telethon.network.mtprotosender] OSError while receiving [Errno 110] Connection timed out
[2019-03-15 10:29:33,099] [INFO@telethon.network.mtprotosender] Connecting to 149.154.167.51:443...
[2019-03-15 10:29:34,117] [WARNING@telethon.network.mtprotosender] Attempt 1 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:34,119] [WARNING@telethon.network.mtprotosender] Attempt 2 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:35,147] [WARNING@telethon.network.mtprotosender] Attempt 3 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:35,148] [WARNING@telethon.network.mtprotosender] Attempt 4 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:36,187] [WARNING@telethon.network.mtprotosender] Attempt 5 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:36,188] [INFO@telethon.network.mtprotosender] Failed reconnection retry 1/5
[2019-03-15 10:29:36,188] [INFO@telethon.network.mtprotosender] Connecting to 149.154.167.51:443...
[2019-03-15 10:29:36,190] [WARNING@telethon.network.mtprotosender] Attempt 1 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:37,227] [WARNING@telethon.network.mtprotosender] Attempt 2 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:37,228] [WARNING@telethon.network.mtprotosender] Attempt 3 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:38,267] [WARNING@telethon.network.mtprotosender] Attempt 4 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:38,269] [WARNING@telethon.network.mtprotosender] Attempt 5 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:38,270] [INFO@telethon.network.mtprotosender] Failed reconnection retry 2/5
[2019-03-15 10:29:38,270] [INFO@telethon.network.mtprotosender] Connecting to 149.154.167.51:443...
[2019-03-15 10:29:39,307] [WARNING@telethon.network.mtprotosender] Attempt 1 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:39,308] [WARNING@telethon.network.mtprotosender] Attempt 2 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:42,427] [WARNING@telethon.network.mtprotosender] Attempt 3 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:42,428] [WARNING@telethon.network.mtprotosender] Attempt 4 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:43,467] [WARNING@telethon.network.mtprotosender] Attempt 5 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:43,468] [INFO@telethon.network.mtprotosender] Failed reconnection retry 3/5
[2019-03-15 10:29:43,469] [INFO@telethon.network.mtprotosender] Connecting to 149.154.167.51:443...
[2019-03-15 10:29:43,470] [WARNING@telethon.network.mtprotosender] Attempt 1 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:46,587] [WARNING@telethon.network.mtprotosender] Attempt 2 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:46,588] [WARNING@telethon.network.mtprotosender] Attempt 3 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:47,627] [WARNING@telethon.network.mtprotosender] Attempt 4 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:47,628] [WARNING@telethon.network.mtprotosender] Attempt 5 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:47,629] [INFO@telethon.network.mtprotosender] Failed reconnection retry 4/5
[2019-03-15 10:29:47,630] [INFO@telethon.network.mtprotosender] Connecting to 149.154.167.51:443...
[2019-03-15 10:29:50,747] [WARNING@telethon.network.mtprotosender] Attempt 1 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:50,749] [WARNING@telethon.network.mtprotosender] Attempt 2 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:51,787] [WARNING@telethon.network.mtprotosender] Attempt 3 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:51,788] [WARNING@telethon.network.mtprotosender] Attempt 4 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:54,907] [WARNING@telethon.network.mtprotosender] Attempt 5 at connecting failed: ConnectionResetError:
[2019-03-15 10:29:54,908] [INFO@telethon.network.mtprotosender] Failed reconnection retry 5/5
[2019-03-15 10:29:54,909] [ERROR@telethon.network.mtprotosender] Failed to reconnect automatically.
[2019-03-15 10:29:54,909] [INFO@telethon.network.mtprotosender] Disconnecting from 149.154.167.51...
[2019-03-15 10:29:54,912] [INFO@telethon.network.mtprotosender] Disconnection from 149.154.167.51 complete!
tulir commented 5 years ago

This probably needs to be added to the library. As you can see in the logs, it's handled by telethon.

WGH- commented 5 years ago

https://telethon.readthedocs.io/en/latest/telethon.client.html#telethon.client.telegrambaseclient.TelegramBaseClient

Isn't connection_retries what we need? It doesn't give us exponential backoff, but at least the bot would keep trying instead of giving up forever.

If retry_delay is raised to, say, 1 minute, attempting to reconnect every minute doesn't seem too hard on network, isn't it?

tulir commented 5 years ago

Oh yeah, I guess I could make the existing options configurable