REST calls memory leak - Githubissues

hawkeye-bot commented 1 year ago

Version of this library.

unicorn_fy: 0.12.2 unicorn_binance_local_depth_cache: 0.7.3 unicorn_binance_rest_api: 1.5.0 unicorn_binance_trailing_stop_loss: not found unicorn_binance_websocket_api: 1.41.0

Solution to Issue cannot be found in the documentation or other Issues and also occurs in the latest version of this library.

[X] I checked the documentation and other Issues. I am using the latest version of this library.

Hardware?

Local server/workstation

Operating System?

macOS

Python version?

Python3.10

Installed packages

Package                           Version
--------------------------------- -----------
aniso8601                         9.0.1
attrs                             22.2.0
bbd-table-parser                  0.0.1
certifi                           2022.12.7
cffi                              1.15.1
charset-normalizer                2.1.1
cheroot                           9.0.0
ciso8601                          2.3.0
click                             8.1.3
colorama                          0.4.6
contourpy                         1.0.6
cryptography                      39.0.0
cycler                            0.11.0
Cython                            0.29.28
dateparser                        1.1.5
engineering-notation              0.8.0
exceptiongroup                    1.1.0
fastenum                          1.0.4
filprofiler                       2023.1.0
flake8                            4.0.1
Flask                             2.1.2
Flask-Cors                        3.0.10
Flask-RESTful                     0.3.9
fonttools                         4.38.0
gitdb                             4.0.10
GitPython                         3.1.27
glob2                             0.7
graphviz                          0.20.1
greenlet                          2.0.1
hjson                             3.0.2
idna                              3.4
influxdb-client                   1.30.0
iniconfig                         2.0.0
itsdangerous                      2.1.2
jaraco.functools                  3.5.2
Jinja2                            3.1.2
joblib                            1.2.0
jsons                             1.6.1
kaleido                           0.2.1
kiwisolver                        1.4.4
kneed                             0.7.0
llvmlite                          0.39.1
Mako                              1.2.4
MarkupSafe                        2.1.1
matplotlib                        3.6.2
mccabe                            0.6.1
more-itertools                    9.0.0
mplfinance                        0.12.9b1
multidict                         6.0.4
numba                             0.56.4
numpy                             1.23.5
objgraph                          3.5.0
orderedattrdict                   1.6.0
packaging                         23.0
pandas                            1.5.1
panwid                            0.3.5
parse                             1.19.0
parse-type                        0.6.0
pid                               3.0.4
Pillow                            9.4.0
pip                               22.3.1
plotly                            5.10.0
pluggy                            1.0.0
polars                            0.15.15
prettytable                       3.4.1
psutil                            5.9.0
py                                1.11.0
pyarrow                           10.0.1
pyasn1                            0.4.8
pyasn1-modules                    0.2.8
pybit                             2.3.1
pycodestyle                       2.8.0
pycparser                         2.21
pyflakes                          2.4.0
Pympler                           1.0.1
pyOpenSSL                         23.0.0
pyparsing                         3.0.9
pytest                            7.2.0
pytest-bdd                        5.0.0
python-dateutil                   2.8.2
pytz                              2022.7
pytz-deprecation-shim             0.1.0.post0
PyYAML                            6.0
raccoon                           3.0.0
regex                             2022.3.2
rel                               0.4.7
requests                          2.28.1
Rx                                3.2.0
scikit-learn                      1.1.3
scipy                             1.9.3
service-identity                  21.1.0
setuptools                        65.6.3
six                               1.16.0
smmap                             5.0.0
sortedcontainers                  2.4.0
SQLAlchemy                        1.4.35
ta                                0.10.2
tabulate                          0.9.0
tenacity                          8.1.0
threadpoolctl                     3.1.0
tk-tools                          0.16.0
tomli                             2.0.1
typish                            1.9.3
tzdata                            2022.7
tzlocal                           4.2
ujson                             5.7.0
unicorn-binance-local-depth-cache 0.7.3
unicorn-binance-rest-api          1.5.0
unicorn-binance-websocket-api     1.41.0
unicorn-fy                        0.12.2
urllib3                           1.26.14
urwid                             2.1.2
urwid-readline                    0.13
urwid-utils                       0.1.2
wcwidth                           0.2.5
websocket-client                  1.3.3
websockets                        10.3
Werkzeug                          2.2.2
yarl                              1.8.1

Logging output

Memory = 32.9609375mb
Memory = 34.0625mb
Memory = 34.31640625mb
Memory = 35.140625mb
Memory = 35.53515625mb
Memory = 35.921875mb
Memory = 36.15625mb
Memory = 36.41796875mb
Memory = 36.9375mb
Memory = 36.953125mb
Memory = 36.953125mb
Memory = 36.953125mb
Memory = 36.953125mb
Memory = 36.953125mb
Memory = 36.953125mb
Memory = 37.046875mb
Memory = 37.27734375mb
Memory = 38.05859375mb
Memory = 38.05859375mb
Memory = 38.1015625mb
Memory = 38.29296875mb
Memory = 38.29296875mb
Memory = 38.29296875mb
Memory = 38.296875mb
Memory = 38.31640625mb
Memory = 38.31640625mb
Memory = 38.31640625mb
Memory = 38.31640625mb
Memory = 38.31640625mb
Memory = 38.31640625mb
Memory = 38.31640625mb
Memory = 38.3203125mb
Memory = 39.11328125mb
Memory = 39.11328125mb
Memory = 39.15625mb
Memory = 39.15625mb
Memory = 39.15625mb
Memory = 39.15625mb
Memory = 39.15625mb
Memory = 39.15625mb

Used endpoint?

concerns all

Issue

I've been having issues with the memory usage in my application, and I've been trying to hunt this down. After quite some searching I've been able to pin it to REST-calls taking place (not sure if it's actually a problem of the underlying requests library). I've constructed a very simple example that shows the memory building up. I can live with the fact that the garbage collector maybe does it's work lazy and it takes some time for the memory usage to stabilise, but adding 7MB in a minute or two with doing nothing but a dumb request and not even storing the result myself is a real problem I think.

Here is the simple piece of code that will show a (small, but continuous) buildup of memory over time:

`import logging import os

import psutil import time from unicorn_binance_rest_api import BinanceRestApiManager

if name == 'main': rest_manager = BinanceRestApiManager()

while True:
    rest_manager.futures_exchange_info()
    print(f'Memory = {psutil.Process(os.getpid()).memory_info().rss / 1024 ** 2}mb')
    time.sleep(1)

`

I have to admit I'm not a deeply experienced python developer, so perhaps I'm overlooking something very simple, but it's not the memory usage pattern I'd expect from this sample code.

oliver-zehentleitner commented 1 year ago

What if you just create a couple requests and then you wait a few seconds and then take a look if there is any garbage?