Connection reset by peer (again)

[beta48]

Hello, Telepot is great, it is working perfectly (over longtime) with version 7.1 and python 2.7.

After installing version 12.0 I get from time to time an error message when trying to send a textmessage. When catching this error (and logging it), a later attempt after some seconds to send another message works perfectly. It seems to me to be this problem with urllib again discussed for version 8 but obviously not fixed in version 12. The problem seems to appear after an idle time (no messages sent) of some hours.

What can I do?

Error log:

ERROR | Textnachricht konnte nicht gesendet werden. Traceback (most recent call last): File "/home/pi/Projekte/Mybot/telegram.py", line 483, in telegram_text bot.sendMessage(shared.telegram_users_authentified[i][1], message, reply_markup=show_individual_keyboard) File "/usr/local/lib/python2.7/dist-packages/telepot/__init__.py", line 445, in sendMessage return self._api_request('sendMessage', _rectify(p)) File "/usr/local/lib/python2.7/dist-packages/telepot/__init__.py", line 434, in _api_request return api.request((self._token, method, params, files), **kwargs) File "/usr/local/lib/python2.7/dist-packages/telepot/api.py", line 130, in request r = fn(*args, **kwargs) #fnmust be thread-safe File "/usr/local/lib/python2.7/dist-packages/urllib3/request.py", line 148, in request_encode_body return self.urlopen(method, url, **extra_kw) File "/usr/local/lib/python2.7/dist-packages/urllib3/poolmanager.py", line 321, in urlopen response = conn.urlopen(method, u.request_uri, **kw) File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 649, in urlopen _stacktrace=sys.exc_info()[2]) File "/usr/local/lib/python2.7/dist-packages/urllib3/util/retry.py", line 357, in increment raise six.reraise(type(error), error, _stacktrace) File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 600, in urlopen chunked=chunked) File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 379, in _make_request httplib_response = conn.getresponse(buffering=True) File "/usr/lib/python2.7/httplib.py", line 1073, in getresponse response.begin() File "/usr/lib/python2.7/httplib.py", line 415, in begin version, status, reason = self._read_status() File "/usr/lib/python2.7/httplib.py", line 371, in _read_status line = self.fp.readline(_MAXLINE + 1) File "/usr/lib/python2.7/socket.py", line 476, in readline data = self._sock.recv(self._rbufsize) File "/usr/lib/python2.7/ssl.py", line 714, in recv return self.read(buflen) File "/usr/lib/python2.7/ssl.py", line 608, in read v = self._sslobj.read(len or 1024) ProtocolError: ('Connection aborted.', error(104, 'Connection reset by peer'))

Simplified python function used to send the messages.

# Text per Telegram senden  
def telegram_text(message,anwen_ID):
    global bot

    # Custom Keyboard
    show_individual_keyboard = {'keyboard': [['/scharf', '/teilscharf'],['/unscharf', '/status']], 'resize_keyboard': True}

    try:
        bot.sendMessage(str(anwen_ID), message, reply_markup=show_individual_keyboard)
        log.info("Textnachricht an ID " + str(anwen_ID) + " geschickt")
    except:
        log.exception("Textnachricht an ID " + str(anwen_ID) + " konnte nicht gesendet werden.")

[nickoala]

That problem that you mentioned ( #87 ) was supposedly fixed by introducing a retry parameter in the connection pool (see my comment on Jul 26, 2016). Maybe your situation, due to whatever reason, demands a higher number of retries. Can you put this section at the beginning of your program and see if it makes any difference?

import telepot.api

telepot.api._pools = {
    'default': urllib3.PoolManager(num_pools=3, maxsize=10, retries=6, timeout=30),
}

[beta48]

Thanks for the quick answer!

What I don't understand: If you don't have reduced the number of retries between version 7.1 and the new version with the urllib: Increasing the number now seems to be a kind of a workaround repairing a hidden problem introduced after 7.1 and there might be situation where increasing the number of retries do not cover the original problem. So I am a little sceptic how reliable (important for me) will be a solution based on an increased number of retries found by testing. Or am I wrong?

So you mean import telepot and telepot.api (which I have not seen before)? This then would like this - correct?

def telegram_initialize():
    global bot
    import telepot
    import telepot.api

    try:
        telepot.api._pools = {
        'default': urllib3.PoolManager(num_pools=3, maxsize=10, retries=6, timeout=30),
        }
        bot = telepot.Bot(my_telegram_token)

    except:
        log.error("Telepot konnte nicht initialisiert werden.")

[nickoala]

I forgot which version it was, but somewhere between 7.1 and now, I introduced connection pooling, which is implemented in the module telepot.api. #87 happened right after connection pooling was introduced. Version 7.1 (the one you were using) did not have connection pooling, meaning every time it contacts Telegram servers, a brand new HTTP connection is used.

So, it seems:

• Connection pooling: need retries
• Individual connections: no retries needed

Why the difference? I have no idea. I can't think of any changes after 7.1 which may cause such a problem. If I really want to know, I would look more deeply into urllib3. But I have no such motivation.

If you want as much stability as before, you may try to override the function _which_pool() to make it use an individual connection every time:

def always_use_new(req, **user_kw):
    return None

telepot.api._which_pool = always_use_new

I haven't tried this code, but it should not be far off. The point is, you can use the function _which_pool() to dictate which pool to use or to use a new connection.

Another way to achieve more stability is to catch the exception yourself and retry. I understand that's troublesome and is basically shifting the responsibility from the library to the user. That's why I don't expect you to do that, although it is one way out.

Good luck.

[nickoala]

One more thing. Before version 8.0 and connection pooling, I was using Requests. I didn't remember why I chose urllib3 over Requests for connection pooling ..... Maybe Requests did not have proper connection pool then .....

Today I look again, Requests does have it now. Maybe I should give it a try?

[beta48]

I would really appreciate if I could use the newest features of Telegram with your new Telepot versions, because I really like your library. But to be honest, I have a couple of bugs in my own code I have to fix first before digging into your codes. And to be honest I don't really know the concepts of pooling, retries, requests etc. what you explained above.

So I would be very happy if you could find out what reduced the stability after 7.1. If that would end in a bugfix that would of course be the best solution. A tested configuration tutorial could be a solution that would also help.

In any case: Thanks a lot.

[nickoala]

I am actually considering making the underlying HTTP library configurable. That is, it allows you to switch between urllib3 and Requests. As I said, I did not see anything that may cause the instability after 7.1, except the changeover to urllib3 and connection pooling. If Requests proves to be stable, I may actually ditch urllib3 and use Requests solely.

I can't promise when that will be done (and can't guarantee things will be more stable under Requests). For now at least, the easiest path to stability seems to be to revert back to 7.1.

In any case, thanks for using telepot. Since this is all voluntary work, sorry I can't provide much prompt assistance in this respect.

[plaksivayatryapka]

Hello Nick! Great lib, thanx!

Got ProtocolError: ('Connection aborted.', error(104, 'Connection reset by peer'))

1. I get this error for many times but only after script is idle for some time. For example 40 minutes.

2. In one certain script it comes really often, but maybe i updated smth accidentally(. Error occurs maybe in 50% cases, when the script runs this part of code:

   for ... in ... : # here are 3 pics send_photo(chat_id, open(filename1, 'rb'))

   for ... in ... : # 1 pic send_photo(chat_id, open(filename2, 'rb'))

   send_photo(chat_id, open(pic1, 'rb')) send_photo(chat_id, open(pic2, 'rb')) send_message(chat_id, 'some text here') # only this line often leads to error. Not others send_video(chat_id, open(vid1, 'rb')) send_video(chat_id, open(vid2, 'rb'))

So the solution is install telepot 7.1, right?

P.S. added "always_use_new" to my code. I'll test it and tell the results.

[D0m3L]

For me reverting to v7.1 solve urllib3 "idle issue"/connection reset. As I saw two independent TCP connections are established after resposne for first command - I bet one is open for getUpdates and the other one for sending responses. This TCP for sending responses got frozen after a certaing time (like 10+ minutes) and throw an exception mentioned above (e.g. Connection reset by peer or connection timeout) and then establish new tcp pipe but the response (in my case) was never delivered as it was discarded on terminated tcp pipe. My guess would be to add some kind of keep alive/pings for this "sending pipe" between script and Telegram API. I'm not sure how and if configurable in urllib3. The other solution is to keep the "sending pipe" busy by sending "dummy" message every minute but this is an ugly hack for me.

Again - in opinion: "old Requests" >>>>> "urllib3" within connection pooling. As for now I'm staying on v7.1 and looking forward for an option to choose Requests in future releases of your great libs @nickoala!

[nickoala]

Thanks for the info, @D0m3L. Just to let you know, I really can't say when the option to choose between Requests and urllib3 will be done (or at all), because telepot is not a priority in my life right now. In any case, thanks for the support.