search

loosolab / UROPA

Universal RObust Peak Annotator
https://uropa-manual.readthedocs.io/
MIT License
15 stars 6 forks source link

Logger cannot be pickled #13

Closed njrobins closed 2 years ago

njrobins commented 2 years ago

Thank you very much in advance for your help. I attempted to run UROPA using the following command:

uropa -i UROPA_TF_footprinting.json -p merged_0.05_reannotated

The JSON contains the following:

{ "queries":[ {"feature":"gene", "feature.anchor": "start", "distance":[10000,1000], "strand":"ignore", "direction":"any_direction", "internals":"Yes", "filter.attribute":"None", "attribute.value":"None", "show.attributes":"gene_id" } ], "gtf": "Rattus_norvegicus.mRatBN7.2.105.gtf", "bed": "merged_AllSamples_0.05.bed" }

I have also tried to run UROPA by including individual parameters/inputs through the command line, i.e.:

uropa -b merged_AllSamples_0.05.bed -g Rattus_norvegicus.mRatBN7.2.105.gtf --feature "gene" --feature-anchor "start" --distance 10000 1000 --internals 1 --show-attributes "gene_id" -p "merged_0.05_reannotated"

In both cases, I received the following message sequence, terminating in the error message "logger cannot be pickled":

Traceback (most recent call last): File "/Users/njrobins/enter/bin/uropa", line 8, in sys.exit(main()) File "/Users/njrobins/enter/lib/python3.9/site-packages/uropa/uropa.py", line 134, in main logger.start_logger_queue() #start listening for logging sent to queue File "/Users/njrobins/enter/lib/python3.9/site-packages/uropa/utils.py", line 79, in start_logger_queue self.listener.start() File "/Users/njrobins/enter/lib/python3.9/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) File "/Users/njrobins/enter/lib/python3.9/multiprocessing/context.py", line 224, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "/Users/njrobins/enter/lib/python3.9/multiprocessing/context.py", line 284, in _Popen return Popen(process_obj) File "/Users/njrobins/enter/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in init super().init(process_obj) File "/Users/njrobins/enter/lib/python3.9/multiprocessing/popen_fork.py", line 19, in init self._launch(process_obj) File "/Users/njrobins/enter/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch reduction.dump(process_obj, fp) File "/Users/njrobins/enter/lib/python3.9/multiprocessing/reduction.py", line 60, in dump ForkingPickler(file, protocol).dump(obj) File "/Users/njrobins/enter/lib/python3.9/logging/init.py", line 1739, in reduce raise pickle.PicklingError('logger cannot be pickled') _pickle.PicklingError: logger cannot be pickled

Notably, I encountered the same issue when attempting to run TOBIAS with a BED file that was not re-annotated using UROPA, and I am happy to provide my code and error message if you would find these helpful. Because these are separate packages, I suspect there is an issue with my version of python or a subset of its program files that are utilized by both of these packages, but I have found nothing particularly on-point elsewhere on the web. I am curious if this issue has come up for you or others in the past and what may be done to circumvent it (e.g., updating python or specific dependencies).

Thanks so much again! Nathaniel

msbentsen commented 2 years ago

Hi Nathaniel, Sorry about this issue. Yes, it is very possible that the same modules are used for both packages, but I was not able to reproduce the issue. Can you maybe give a little bit more information on the system you are on? Mac / linux / hpc cluster? Maybe a list of the package versions (e.g. using pip freeze)? That would be really helpful to debug this, thank you!

njrobins commented 2 years ago

Hi Mette,

Thanks so much for your response. I am actually encouraged that you were able to run this successfully on your system. I am using a Mac (MacOS Big Sur v11.3.1). I do have access to an HPC cluster and could try running it through there.

Currently I have UROPA v4.0.2 and TOBIAS v.0.13.2 installed. A complete output from pip freeze is given below:

adjustText==0.7.3 amply==0.1.4 appdirs==1.4.4 appnope @ file:///Users/runner/miniforge3/conda-bld/appnope_1610094673755/work argon2-cffi @ file:///Users/runner/miniforge3/conda-bld/argon2-cffi_1625821234041/work async-generator==1.10 attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1620387926260/work backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1618230623929/work biopython==1.79 bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1626371669429/work boto3==1.21.20 botocore==1.24.20 brotlipy==0.7.0 certifi==2021.10.8 cffi @ file:///opt/concourse/worker/volumes/live/976f8942-f51d-4f0e-7352-2a10f0820d0e/volume/cffi_1625814703974/work chardet @ file:///opt/concourse/worker/volumes/live/7e1102c4-8702-40f2-63d6-f260ce5f85e4/volume/chardet_1607706831384/work conda==4.12.0 conda-package-handling @ file:///opt/concourse/worker/volumes/live/8fb3e065-760b-4a9d-4cd9-aca7fc8baf53/volume/conda-package-handling_1618262145611/work ConfigArgParse==1.5.1 configs==3.0.3 connection-pool==0.0.3 cryptography @ file:///opt/concourse/worker/volumes/live/d5dda287-c0b3-4861-7262-fab05baa64dc/volume/cryptography_1616769284011/work cycler==0.11.0 datrie==0.8.2 debugpy @ file:///Users/runner/miniforge3/conda-bld/debugpy_1626905096664/work decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1621187651333/work defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work dill==0.3.4 docutils==0.17.1 entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1605121927639/work/dist/entrypoints-0.3-py2.py3-none-any.whl filelock==3.0.12 fonttools==4.30.0 gitdb==4.0.7 GitPython==3.1.18 idna @ file:///home/linux1/recipes/ci/idna_1610986105248/work importlib-metadata @ file:///Users/runner/miniforge3/conda-bld/importlib-metadata_1625463744025/work ipykernel @ file:///Users/runner/miniforge3/conda-bld/ipykernel_1626728184263/work/dist/ipykernel-6.0.3-py3-none-any.whl ipython @ file:///Users/runner/miniforge3/conda-bld/ipython_1625027511604/work ipython-genutils==0.2.0 jedi @ file:///Users/runner/miniforge3/conda-bld/jedi_1610146808808/work Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1621419064915/work jmespath==0.10.0 joblib==1.1.0 jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema_1614815863336/work jupyter-client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1615693636836/work jupyter-core @ file:///Users/runner/miniforge3/conda-bld/jupyter_core_1612125283939/work jupyterlab-pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1601375948261/work kiwisolver==1.4.0 kneed==0.7.0 logomaker==0.8 MarkupSafe @ file:///Users/runner/miniforge3/conda-bld/markupsafe_1621455730541/work matplotlib==3.5.1 matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1618935594181/work mistune @ file:///Users/runner/miniforge3/conda-bld/mistune_1624941339085/work MOODS-python==1.9.4.1 multiprocess==0.70.12.2 nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1614336084111/work nbconvert @ file:///Users/runner/miniforge3/conda-bld/nbconvert_1624472871251/work nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1617383142101/work nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1617163391303/work notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1621259862661/work numpy==1.22.3 packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1625323647219/work pandas==1.4.1 pandocfilters==1.4.2 parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1617148930513/work pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1602535608087/work pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work Pillow==9.0.1 prometheus-client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1622586138406/work prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1623977816122/work psutil==5.8.0 ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl PuLP==2.4 pybedtools==0.9.0 pyBigWig==0.3.18 pycosat==0.6.3 pycparser @ file:///tmp/build/80754af9/pycparser_1594388511720/work Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1620245170812/work pyOpenSSL @ file:///tmp/build/80754af9/pyopenssl_1608057966937/work pyparsing==2.4.7 PyPDF2==1.26.0 pyrsistent @ file:///Users/runner/miniforge3/conda-bld/pyrsistent_1624984762901/work pysam==0.18.0 PySocks @ file:///opt/concourse/worker/volumes/live/112288ac-9cb0-4e73-768b-13baf4ca6419/volume/pysocks_1605305820043/work python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work pytz==2021.3 PyYAML==5.4.1 pyzmq @ file:///Users/runner/miniforge3/conda-bld/pyzmq_1622038559383/work ratelimiter==1.2.0.post0 reportlab==3.6.8 requests @ file:///tmp/build/80754af9/requests_1608241421344/work ruamel-yaml-conda @ file:///opt/concourse/worker/volumes/live/e81cf0fe-611a-498e-6e69-a7320057c1ac/volume/ruamel_yaml_1616016689696/work s3transfer==0.5.2 scikit-learn==1.0.2 scipy==1.8.0 seaborn==0.11.2 Send2Trash @ file:///home/conda/feedstock_root/build_artifacts/send2trash_1624366715919/work six @ file:///tmp/build/80754af9/six_1623709665295/work smart-open==5.1.0 smmap==4.0.0 snakemake==5.9.1 statistics==1.0.3.5 stopit==1.1.2 svist4get==1.2.24 tabulate==0.8.9 terminado @ file:///Users/runner/miniforge3/conda-bld/terminado_1623365368681/work testpath @ file:///home/conda/feedstock_root/build_artifacts/testpath_1621261527237/work threadpoolctl==3.1.0 tobias==0.13.2 toposort==1.6 tornado @ file:///Users/runner/miniforge3/conda-bld/tornado_1625488906146/work tqdm @ file:///tmp/build/80754af9/tqdm_1625563689033/work traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1602771532708/work urllib3 @ file:///tmp/build/80754af9/urllib3_1625084269274/work uropa==4.0.2 Wand==0.6.7 wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1600965781394/work webencodings==0.5.1 wrapt==1.12.1 XlsxWriter==3.0.3 zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1625284368454/work

Thank you again!

msbentsen commented 2 years ago

For future reference: This issue was solved by downgrading to python < 3.8 on MacOS systems (see discussion: https://github.com/loosolab/TOBIAS/issues/127#issuecomment-1090290253).