Closed Badiboy closed 6 years ago
I dig deeper unrolling functions one by one... Unitll I remove everything from Telebot... And came to the requests :(
token = "XXX"
chat_id = 'NNN'
proxy = {'https':'ZZZ'}
from time import sleep
import threading, gc, requests
def garbage_info():
res = ""
res += "\nGetObjects len: " + str(len(gc.get_objects()))
return res
def tester():
count = 0
while(True):
sleep(5)
count += 1
msg = "\nTest N{0}".format(count) + garbage_info()
print(msg)
method_url = r'sendMessage'
payload = {'chat_id': str(chat_id), 'text': msg}
request_url = "https://api.telegram.org/bot{0}/{1}".format(token, method_url)
method_name = 'get'
session = requests.session()
session.request(method_name, request_url, params=payload, proxies=proxy)
gc.collect() #Makes output more clean
tester()
Still leaking...
Test N3 GetObjects len: 29663 Test N4 GetObjects len: 29664 Test N5 GetObjects len: 29665 Test N6 GetObjects len: 29666
Removing
session.request(method_name, request_url, params=payload, proxies=proxy)
stops leaking...
Next step. Going into requests...
session = requests.session()
req = requests.Request(
method=method_name.upper(),
url=request_url,
params=payload
)
prep = session.prepare_request(req)
Leaks are in the last line... Even no need to actually send anything... _prepare_request_ is too complex for me to unroll. :(
The reason of this behaviour was found in "requests" unit cache mechanism.
It works incorrect: it adds a cache record to every call to Telegram API URL (instead of caching it once). But it does not lead to the memory leak, because cache size is limited to 20 and cache is resetting after reaching this limit and the growing number of objects will be decreased back to initial value.
Как ты решил эту проблему? У меня похожая ситуация с моим ботом на pyTelegramBotAPI, бот работает какое-то время с потреблением 160-200 Мбайт оперативной памяти, но в непонятный для меня момент потребление памяти начинает линейно расти и не останавливается, вплоть до 4 Гигабайт и процесс убивается системой.
@Badiboy
Problem had decreased when upgrading Python (in those days it was 3.6 => 3.8 AFAIR). Plus daily system restart.
Memory leaks in libraries were not confirmed. If you have leaks - most probably you should inspect your code. Regarding the pyTelegramBotAPI - check how you work with "next step" or States (whatever you use).
PS. Pinning someone if he do not reply is not a correct public behaviour.
I noticed free memory degradation when running bot for a long time. Firstly, I suspect my code; but after some instections and test I came to suspect the bot. :)
I made the simple test application:
Starting from 3-4 interation it stabilised and shows the permanent increase of the number of objects, tracked by GC:
When I disable the message sending
the number of tracked objects stops increasing:
I tried to force the garbage collection
but nothing changes.
Statistics shows, that when bot sends about 10K messages per day memory leaks are about 100Mb per day. All numbers are very rought, because there can be leaks in my code also...
PS: I'm very weak in Python GC, so maybe I made wrong interpretation of it's numbers.