Possible memory leaks can lead to server crash on dataset export

zhiltsov-max commented 4 years ago

My actions before raising this issue

[ ] Read/searched the docs
[ ] Searched past issues

Expected Behaviour

Current Behaviour

Test on a large video with chunks and cache enabled.

Run the code snippet with python manage.py shell
Hit ctrl+C at some point
check top output

There should not be any extra memory allocated, but after few runs over a video task with 3k frames I have 5-7 gb of allocated memory. Gc tells it knows nothing about these allocations.

import gc
import tracemalloc
tracemalloc.start(10)
try:    
    from cvat.apps.engine.models import Task
    from cvat.apps.engine.frame_provider import FrameProvider
    fp = FrameProvider(Task.objects.get(pk=39).data) # a large video task

    while True:
        for i, f in enumerate(fp.get_frames(out_type=fp.Type.NUMPY_ARRAY)):
            pass

except:    
    del fp
    del f
    snapshot = tracemalloc.take_snapshot()    
    gc.collect()

Env:

Python 3.6.5 ``` absl-py==0.8.1 alabaster==0.7.12 appdirs==1.4.4 asgiref==3.2.10 asn1crypto==0.24.0 astor==0.8.0 astroid==2.4.2 astunparse==1.6.3 atpublic==2.0 attrs==19.3.0 autopep8==1.5.3 av==8.0.2 Babel==2.8.0 bandit==1.6.2 bcrypt==3.1.7 cached-property==1.5.1 cachetools==4.0.0 certifi==2020.6.20 cffi==1.12.3 chardet==3.0.4 click==7.1.2 colorama==0.4.3 commonmark==0.9.1 configobj==5.0.6 coreapi==2.3.3 coreschema==0.0.4 coverage==5.2.1 coveralls==2.1.1 croniter==0.3.30 cryptography==2.7 cycler==0.10.0 dataclasses==0.7 -e git+https://github.com/openvinotoolkit/datumaro.git@3157e749f304635775f10a46b7b15ef09d7b4374#egg=datumaro ddt==1.2.1 decorator==4.4.0 defusedxml==0.6.0 dictdiffer==0.8.1 diskcache==4.1.0 distro==1.5.0 dj-pagination==2.5.0 Django==3.1.1 django-allauth==0.42.0 django-appconf==1.0.4 django-auth-ldap==2.2.0 django-cacheops==5.0.1 django-compressor==2.4 django-cors-headers==3.5.0 django-extensions==3.0.6 django-filter==2.3.0 django-rest-auth==0.9.5 django-revproxy==0.10.0 django-rq==2.3.2 django-sendfile==0.3.11 django-silk==4.0.1 djangorestframework==3.11.1 docker==3.7.3 docker-compose==1.24.1 docker-pycreds==0.4.0 dockerpty==0.4.1 docopt==0.6.2 docutils==0.16 dpath==2.0.1 drf-yasg==1.17.1 dvc==1.7.2 EasyProcess==0.3 editdistance==0.5.3 entrypoint2==0.2.1 fakeredis==1.4.1 ffmpy==0.2.2 flatten-dict==0.3.0 flufl.lock==3.2 funcy==1.14 furl==2.1.0 future==0.18.2 gast==0.3.3 gitdb==4.0.5 gitdb2==3.0.2 GitPython==3.1.3 google-auth==1.11.0 google-auth-oauthlib==0.4.1 google-pasta==0.1.8 gprof2dot==2019.11.30 grandalf==0.6 grpcio==1.24.1 h5py==2.10.0 idna==2.7 imageio==2.6.1 imagesize==1.2.0 imgaug==0.4.0 importlib-metadata==1.7.0 inflection==0.3.1 isort==4.3.21 itypes==1.1.0 Jinja2==2.10.3 joblib==0.16.0 jsonpath-ng==1.5.2 jsonschema==2.6.0 Keras==2.4.2 Keras-Applications==1.0.8 Keras-Preprocessing==1.1.0 kiwisolver==1.1.0 lazy-object-proxy==1.5.1 lxml==4.4.1 Markdown==3.2.2 MarkupSafe==1.1.1 matplotlib==3.3.1 mccabe==0.6.1 mod-wsgi==4.7.1 nanotime==0.5.2 networkx==2.3 nibabel==3.1.1 nose==1.3.7 numpy==1.17.5 oauthlib==3.1.0 opencv-python==4.4.0.42 opencv-python-headless==4.2.0.34 opt-einsum==3.1.0 orderedmultidict==1.0.1 packaging==20.3 paramiko==2.6.0 pascal-voc-writer==0.1.4 pathlib2==2.3.5 pathspec==0.8.0 patool==1.12 pbr==5.4.5 pdf2image==1.14.0 Pillow==7.2.0 pipdeptree==0.13.2 piprot==0.9.11 ply==3.11 protobuf==3.10.0 psycopg2-binary==2.8.5 py-cpuinfo==4.0.0 pyasn1==0.4.7 pyasn1-modules==0.2.7 pycocotools==2.0.0 pycodestyle==2.6.0 pycparser==2.19 pydicom==2.0.0 pydot==1.4.1 Pygments==2.6.1 pygtrie==2.3.2 pyhash==0.9.3 pylint==2.6.0 pylint-django==2.3.0 pylint-plugin-utils==0.6 PyNaCl==1.3.0 pyparsing==2.4.2 pypi-kenlm==0.1.20190403 python-dateutil==2.8.0 python-ldap==3.3.1 python-logstash==0.4.6 python3-openid==3.1.0 pytz==2020.1 pyunpack==0.2.1 PyWavelets==1.0.3 PyYAML==5.3.1 rcssmin==1.0.6 redis==3.5.3 requests==2.24.0 requests-futures==1.0.0 requests-oauthlib==1.2.0 rich==6.2.0 rjsmin==1.1.0 rope==0.17.0 rq==1.5.1 rq-scheduler==0.10.0 rsa==4.0 rstcheck==3.3.1 ruamel.yaml==0.16.5 ruamel.yaml.clib==0.2.0 rules==2.2 scikit-image==0.15.0 scikit-learn==0.23.2 scipy==1.4.1 sentencepiece==0.1.91 Shapely==1.7.1 shortuuid==1.0.1 shtab==1.3.1 six==1.15.0 smmap==3.0.4 smmap2==2.0.5 snakeviz==2.1.0 snowballstemmer==2.0.0 sortedcontainers==2.1.0 Sphinx==3.2.1 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==1.0.3 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.4 sqlparse==0.3.1 stevedore==3.2.0 tabulate==0.8.7 tensorboard==2.2.2 tensorboard-plugin-wit==1.7.0 tensorboardX==1.8 tensorflow==2.2.0 tensorflow-estimator==2.2.0 termcolor==1.1.0 texttable==0.9.1 threadpoolctl==2.1.0 tokenizers==0.8.1 toml==0.10.1 torch==1.5.1 torchvision==0.6.1 tornado==6.0.3 tqdm==4.49.0 typed-ast==1.4.0 typing-extensions==3.7.4.3 uritemplate==3.0.0 urllib3==1.24.3 voluptuous==0.11.7 websocket-client==0.56.0 Werkzeug==1.0.1 wrapt==1.12.1 yamlloader==0.5.5 zc.lockfile==2.0 zipp==3.1.0 ```

Your Environment

Git hash commit (git log -1): f2c84a2653fc601c90bcb84154f0fe7a6be937df
Operating System and version (e.g. Linux, Windows, MacOS): Ubuntu 18.04

Next steps

You may join our Gitter channel for community support.

zhiltsov-max commented 3 years ago

The reason can probably be here: https://github.com/openvinotoolkit/cvat/blob/develop/cvat/apps/engine/media_extractors.py#L325

The container gets created, but never freed.

nmanovic commented 3 years ago

@kirill-sizov , is it something similar to the issue which you are working on?

azhavoro commented 1 year ago

Fixed in #6501

cvat-ai / cvat