celery / billiard

Multiprocessing Pool Extensions
Other
419 stars 252 forks source link

due to circular reference of local variable in Exception #311

Open heedong-jung opened 4 years ago

heedong-jung commented 4 years ago

Memory leak due to circular reference of local variable in Exception.

Python Packages

\ pip freeze Output:

aioprocessing==1.0.1 alabaster==0.7.12 alembic==1.2.1 amqp==2.5.2 aniso8601==6.0.0 astroid==2.3.3 astropy==3.2.3 atomicwrites==1.3.0 attrs==19.3.0 auto-sklearn==0.5.2 Babel==2.7.0 backcall==0.1.0 backports.csv==1.0.7 billiard==3.6.3.0 bleach==3.1.0 boto3==1.9.160 botocore==1.12.160 catboost==0.14.2 celery==4.3.0 celery-progress==0.0.3 certifi==2019.11.28 cffi==1.14.0 chardet==3.0.4 Click==7.0 cloudpickle==1.1.1 config==0.4.2 configparser==3.7.4 ConfigSpace==0.4.7 confuse==1.0.0 coreapi==2.3.3 coreschema==0.0.4 coverage==4.5.4 cycler==0.10.0 Cython==0.29.16 databricks-cli==0.9.0 debug==0.3.2 decorator==4.4.1 defusedxml==0.6.0 diff-match-patch==20181111 dill==0.2.9 Django==2.2.2 django-cors-headers==3.0.2 django-dbconn-retry==0.1.5 django-import-export==1.2.0 django-keycloak==0.1.1 django-rest-framework==0.1.0 django-user==0.2.1 djangorestframework==3.9.4 docker==4.0.1 docutils==0.14 drf-yasg==1.16.0 ds-lime==0.1.1.27 ecdsa==0.13.2 eli5==0.10.1 entrypoints==0.3 enum34==1.1.6 et-xmlfile==1.0.1 fastparquet==0.3.2 Flask==1.0.3 future==0.17.1 gitdb2==2.0.5 GitPython==2.1.11 gorilla==0.3.0 GPy==1.9.9 GPyOpt==1.2.5 graphene==2.1.7 graphene-django==2.5.0 graphene-file-upload==1.2.2 graphql-core==2.2.1 graphql-relay==2.0.0 graphviz==0.11 gunicorn==19.9.0 h5py==2.10.0 hpbandster==0.7.4 htmlmin==0.1.12 idna==2.8 imageio==2.5.0 imagesize==1.1.0 importlib-metadata==0.20 inflection==0.3.1 ipdb==0.13.2 ipython==7.13.0 ipython-genutils==0.2.0 isort==4.3.21 itsdangerous==1.1.0 itypes==1.1.0 jdcal==1.4.1 jedi==0.16.0 Jinja2==2.10.1 jmespath==0.9.4 joblib==0.13.2 jsonschema==3.0.1 jupyter-client==5.3.4 jupyter-core==4.6.1 Keras==2.3.1 Keras-Applications==1.0.8 Keras-Preprocessing==1.1.0 kiwisolver==1.1.0 kombu==4.6.8 lazy-import==0.2.2 lazy-object-proxy==1.4.3 liac-arff==2.4.0 lightgbm==2.2.3 lime==0.1.1.34 llvmlite==0.29.0 lockfile==0.12.2 Mako==1.1.0 MarkupSafe==1.1.1 matplotlib==3.1.0 mccabe==0.6.1 memory-profiler==0.55.0 missingno==0.4.2 mistune==0.8.4 mleap==0.8.1 mlflow==1.2.0 mock==3.0.5 more-itertools==8.2.0 multiprocess==0.70.7 nbconvert==5.6.1 nbformat==4.4.0 nest-asyncio==1.2.3 netifaces==0.10.9 networkx==2.3 nose==1.3.7 nose-exclude==0.5.0 numba==0.45.1 numpy==1.16.4 objgraph==3.4.1 odfpy==1.4.0 openpyxl==2.6.2 packaging==19.0 pandas==0.24.2 pandas-profiling==2.3.0 pandocfilters==1.4.2 paramz==0.9.5 parso==0.6.0 patsy==0.5.1 PDPbox==0.2.0 pexpect==4.8.0 phik==0.9.8 pickleshare==0.7.5 Pillow==6.0.0 plotly==3.10.0 pluggy==0.12.0 prometheus-client==0.7.1 promise==2.2.1 prompt-toolkit==3.0.4 protobuf==3.8.0 psutil==5.6.2 psycopg2==2.7.5 ptyprocess==0.6.0 py==1.8.0 py4j==0.10.7 pyarrow==0.14.1 pyasn1==0.4.5 pycparser==2.20 pydot==1.4.1 pyfiglet==0.8.post1 Pygments==2.4.2 PyHive==0.6.1 pylint==2.4.3 Pympler==0.7 pynisher==0.5.0 pyparsing==2.4.0 pyrfr==0.8.0 Pyro4==4.77 pyrsistent==0.15.3 pysasl==0.5.0 pyspark==2.4.3 pytest==5.1.2 pytest-cov==2.7.1 pytest-django==3.5.1 pytest-mock==1.10.4 pytest-pylint==0.14.1 python-dateutil==2.8.0 python-editor==1.0.4 python-jose==3.0.1 python-keycloak-client==0.2.2 pytz==2019.1 PyWavelets==1.0.3 PyYAML==5.1.1 pyzmq==18.1.1 querystring-parser==1.2.3 rabbitmq==0.2.0 requests==2.22.0 retrying==1.3.3 rsa==4.0 ruamel.yaml==0.15.97 Rx==1.6.1 s3transfer==0.2.0 sasl==0.2.1 scikit-image==0.15.0 scikit-learn==0.19.2 scipy==1.3.0 seaborn==0.9.0 see==1.4.1 Send2Trash==1.5.0 serpent==1.28 simplejson==3.16.0 singledispatch==3.4.0.3 six==1.14.0 skater==1.0.4 smac==0.8.0 smmap2==2.0.5 snowballstemmer==1.2.1 sobol-seq==0.1.2 spark-celery==0.1.1 Sphinx==2.1.0 sphinx-rtd-theme==0.4.3 sphinxcontrib-applehelp==1.0.1 sphinxcontrib-devhelp==1.0.1 sphinxcontrib-htmlhelp==1.0.2 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.2 sphinxcontrib-serializinghtml==1.1.3 SQLAlchemy==1.3.8 sqlparse==0.3.0 statsmodels==0.10.2 tablib==0.13.0 tabulate==0.8.3 testpath==0.4.4 thrift==0.11.0 thrift-sasl==0.3.0 tornado==6.0.3 traitlets==4.3.2 typed-ast==1.4.0 typing==3.6.6 uritemplate==3.0.0 urllib3==1.25.3 vine==1.3.0 wcwidth==0.1.7 webencodings==0.5.1 websocket-client==0.56.0 Werkzeug==0.15.4 wrapt==1.11.2 xgboost==0.90 xlrd==1.2.0 xlwt==1.3.0 yellowbrick==0.9.1 zipp==0.6.0

While the Except statement was in progress, an error occurred that the Garbage Collector could not collect by referring to the f_locals of the frame.

Is it necessary to refer to Tracebackhide in Frame function that copies and utilizes traceback? It doesn't seem to be utilized in practice.