Memory leaks suspected

[Badiboy]

1. What version of pyTelegramBotAPI are you using? 3.6.6
2. What OS are you using? Win10 / Ubuntu 18.04
3. What version of python are you using? 3.5.3

I noticed free memory degradation when running bot for a long time. Firstly, I suspect my code; but after some instections and test I came to suspect the bot. :)

I made the simple test application:

from time import sleep
import threading, gc
from telebot import TeleBot

token = "XXX:XXX"
chat_id = 'NNNN'

bot = TeleBot(token)

def garbage_info():
    res = ""
    res += "\nGetObjects len: " + str(len(gc.get_objects()))
    return res

def tester():
    count = 0
    while(True):
        sleep(5)
        count += 1
        msg = "\nTest N{0}".format(count) + garbage_info()
        print(msg)
        bot.send_message(chat_id, msg)

statUpdateThread = threading.Thread(target = tester)
statUpdateThread.start()

bot.remove_webhook()

bot.polling(none_stop = True, timeout = 600)

Starting from 3-4 interation it stabilised and shows the permanent increase of the number of objects, tracked by GC:

Test N3 GetObjects len: 30465 Test N4 GetObjects len: 30467 Test N5 GetObjects len: 30469 Test N6 GetObjects len: 30471 Test N7 GetObjects len: 30473

When I disable the message sending

def tester():
    count = 0
    while(True):
        sleep(5)
        count += 1
        msg = "\nTest N{0}".format(count) + garbage_info()
        print(msg)

the number of tracked objects stops increasing:

Test N3 GetObjects len: 30380 Test N4 GetObjects len: 30380 Test N5 GetObjects len: 30380 Test N6 GetObjects len: 30380 Test N7 GetObjects len: 30380

I tried to force the garbage collection

def tester():
    count = 0
    while(True):
        sleep(5)
        count += 1
        msg = "\nTest N{0}".format(count) + garbage_info()
        print(msg)
        bot.send_message(chat_id, msg)
        gc.collect()

but nothing changes.

Statistics shows, that when bot sends about 10K messages per day memory leaks are about 100Mb per day. All numbers are very rought, because there can be leaks in my code also...

PS: I'm very weak in Python GC, so maybe I made wrong interpretation of it's numbers.

[Badiboy]

I dig deeper unrolling functions one by one... Unitll I remove everything from Telebot... And came to the requests :(

token = "XXX"
chat_id = 'NNN'
proxy = {'https':'ZZZ'}

from time import sleep
import threading, gc, requests

def garbage_info():
    res = ""
    res += "\nGetObjects len: " + str(len(gc.get_objects()))
    return res

def tester():
    count = 0
    while(True):
        sleep(5)
        count += 1
        msg = "\nTest N{0}".format(count) + garbage_info()
        print(msg)

        method_url = r'sendMessage'
        payload = {'chat_id': str(chat_id), 'text': msg}
        request_url = "https://api.telegram.org/bot{0}/{1}".format(token, method_url)
        method_name = 'get'
        session = requests.session()
        session.request(method_name, request_url, params=payload, proxies=proxy)
        gc.collect() #Makes output more clean

tester()

Still leaking...

Test N3 GetObjects len: 29663 Test N4 GetObjects len: 29664 Test N5 GetObjects len: 29665 Test N6 GetObjects len: 29666

Removing session.request(method_name, request_url, params=payload, proxies=proxy) stops leaking...

[Badiboy]

Next step. Going into requests...

        session = requests.session()
        req = requests.Request(
            method=method_name.upper(),
            url=request_url,
            params=payload
        )
        prep = session.prepare_request(req)

Leaks are in the last line... Even no need to actually send anything... _prepare_request_ is too complex for me to unroll. :(

[Badiboy]

The reason of this behaviour was found in "requests" unit cache mechanism.

It works incorrect: it adds a cache record to every call to Telegram API URL (instead of caching it once). But it does not lead to the memory leak, because cache size is limited to 20 and cache is resetting after reaching this limit and the growing number of objects will be decreased back to initial value.

[Skada13]

Как ты решил эту проблему? У меня похожая ситуация с моим ботом на pyTelegramBotAPI, бот работает какое-то время с потреблением 160-200 Мбайт оперативной памяти, но в непонятный для меня момент потребление памяти начинает линейно расти и не останавливается, вплоть до 4 Гигабайт и процесс убивается системой.

[Skada13]

@Badiboy

[Badiboy]

Problem had decreased when upgrading Python (in those days it was 3.6 => 3.8 AFAIR). Plus daily system restart.

Memory leaks in libraries were not confirmed. If you have leaks - most probably you should inspect your code. Regarding the pyTelegramBotAPI - check how you work with "next step" or States (whatever you use).

PS. Pinning someone if he do not reply is not a correct public behaviour.