Handling of connection error exceptions

[b-ferd]

I'm running OrigamiBot as part of a multi thread server process. From within the telegram communication thread, I run bot.start() (which starts more Origamibot threads in the background), then I use that thread for forwarding messages from the system to the bot asynchronously (i.e.: from the main thread, not triggered by incoming bot messages; I don't use the asyncio concept). This all works quite well.

Things seem to become a bit ugly when for example the network is down, or the server is not reachable for any other reason.

This leads to a flurry of messages like these:

2024-03-11 19:23:03,398 | bot.py:1443 | PID 2282 | ERROR | Exception while polling
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 169, in _new_conn
    conn = connection.create_connection(
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 700, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1017, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 353, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 181, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7b90347264d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 756, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 574, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url: /BOTID:BOTTOKEN/getUpdates (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7b90347264d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/marc/.local/lib/python3.10/site-packages/origamibot/core/bot.py", line 1441, in _listen_loop
    updates = self.get_updates()
  File "/home/marc/.local/lib/python3.10/site-packages/origamibot/core/bot.py", line 314, in get_updates
    updates = get_updates(
  File "/home/marc/.local/lib/python3.10/site-packages/origamibot/core/api_request.py", line 115, in get_updates
    updates = request(
  File "/home/marc/.local/lib/python3.10/site-packages/origamibot/core/api_request.py", line 91, in request
    response = requests.post(
  File "/usr/lib/python3/dist-packages/requests/api.py", line 119, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 544, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 657, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.telegram.org', port=443): Max retries exceeded with url: /BOTID:BOTTOKEN/getUpdates (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7b90347264d0>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'))

(Bot data has been replaced with BOTID and BOTTOKEN in above log).

I would like to somewhat gracefully catch these, log them as a single concise message to syslog ("Telegram Server Connection lost"), and then try to recover (i.e. the system should continue to operate, and silently try to reconnect to the telegram server in the background).

How is this best done? Is it even possible with Origamibot? Is the above behavior a bug? Or am I doing something wrong?

My code looks about as this (edited for brevity):

bot = Bot(bot_token) 
bot.add_listener(MessageListener(bot))
bot.add_commands(BotsCommands(bot))
bot.start()
...
while(True):
    bot.send_message(chat_id, "testmessage") # chat_id stored globally, the chat we want to send info to
    # sleep for a while  

Funny: The bot seems to reconnect, and commands I type in the telegram channel do get through to the system. But messages my systems send to the telegram channel via the bot (see above example) are lost.

[t-n-u-z]

Maybe you have already found a solution yourself. If not, here is my guess.

First of all, you posted the log of the update polling running in the background. This is logged, but should run properly again as soon as the internet connection is re-established. (you wrote this as the funny postscript)

As far as I can see, this library has no integrated error handling for sending a message. However, you should be able to integrate this yourself quite easily.

bot = Bot(bot_token)
bot.add_listener(MessageListener(bot))
bot.add_commands(BotsCommands(bot))
bot.start()
...
while True:  # your default task loop
    while True:  # loop to retry after exception
        try:
            bot.send_message(chat_id, "testmessage")  # chat_id stored globally, the chat we want to send info to
        except IOError:  # may be more precise, e.g. requests.exceptions.ConnectionError
            logging.exception("Telegram Server Connection lost")
            time.sleep(5)  # wait 5 seconds before retry
            continue
        else:
            break  # break inner loop if send_message was successfully
    # sleep for a while  

But my example has one big problem. It keeps trying to send the message until it succeeds. This could end in an endless loop.

I hope this helps you.

[b-ferd]

I did not find a solution, I ran into more issues like this and decided to switch to a different bot framework that offers better error handling. Thanks for trying to help!