search

arpcard / rgi

Resistance Gene Identifier (RGI). Software to predict resistomes from protein or nucleotide data, including metagenomics data, based on homology and SNP models.
Other
319 stars 76 forks source link

'validate_json: False' when loading the files #230

Closed DrBinZhu closed 11 months ago

DrBinZhu commented 1 year ago

After installing the RGI using 'mamba create -c iuc -c conda-forge -c bioconda -c defaults --no-channel-priority --name RGI rgi', I ran the following commands:

rgi card_annotation -i path_to/card.json > card_annotation.log 2>&1 rgi wildcard_annotation -i wildcard --card_json path_to/card.json -v 4.0.0 > wildcard_annotation.log 2>&1

rgi load \ --card_json path_to/card.json \ --debug --local \ --card_annotation card_database_v3.2.7.fasta \ --card_annotation_all_models card_database_v3.2.7_all.fasta \ --wildcard_annotation wildcard_database_v4.0.0.fasta \ --wildcard_annotation_all_models wildcard_database_v4.0.0_all.fasta \ --wildcard_index path_to/wildcard/index-for-model-sequences.txt \ --wildcard_version 4.0.0 \ --amr_kmers path_to/wildcard/all_amr_61mers.txt \ --kmer_database path_to/wildcard/61_kmer_db.json \ --kmer_size 61

Then the information shows: ... filename: card.json, validate_json: False ... filename: card_reference.fasta, validate_json: False ... every validate_json checks are False.

rgi database --version --local 3.2.7

When running a test using this command: 'rgi main --input_sequence path_to/GCF_002861965.1_ASM286196v1_genomic.fna --output_file a --local --clean', there are bugs showing below:

Traceback (most recent call last): File "", line 1, in File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main exitcode = _main(fd, parent_sentinel) File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/multiprocessing/spawn.py", line 125, in _main prepare(preparation_data) File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare _fixup_main_from_path(data['init_main_from_path']) File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path main_content = runpy.run_path(main_path, File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/runpy.py", line 265, in run_path return _run_module_code(code, init_globals, run_name, File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/runpy.py", line 97, in _run_module_code _run_code(code, mod_globals, init_globals, File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/Users/binzhu/miniconda3/envs/RGI/bin/rgi", line 4, in MainBase() File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/site-packages/app/MainBase.py", line 83, in init getattr(self, args.command)() File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/site-packages/app/MainBase.py", line 88, in main self.main_run(args) File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/site-packages/app/MainBase.py", line 128, in main_run rgi_obj.run() File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/site-packages/app/RGI.py", line 255, in run self.filter_process() File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/site-packages/app/RGI.py", line 387, in filter_process filter_obj.run() File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/site-packages/app/Filter.py", line 174, in run self.process_xml_file() File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/site-packages/app/Filter.py", line 102, in process_xml_file model_thread.start() File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/multiprocessing/process.py", line 121, in start self._popen = self._Popen(self) File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/multiprocessing/context.py", line 224, in _Popen return _default_context.get_context().Process._Popen(process_obj) File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/multiprocessing/context.py", line 284, in _Popen return Popen(process_obj) File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 32, in init super().init(process_obj) File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/multiprocessing/popen_fork.py", line 19, in init self._launch(process_obj) File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/multiprocessing/popen_spawn_posix.py", line 42, in _launch prep_data = spawn.get_preparation_data(process_obj._name) File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/multiprocessing/spawn.py", line 154, in get_preparation_data _check_not_importing_main() File "/Users/binzhu/miniconda3/envs/RGI/lib/python3.8/multiprocessing/spawn.py", line 134, in _check_not_importing_main raise RuntimeError(''' RuntimeError: An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

I'm using macOS Big Sur, version 11.1

Do you have any idea to deal with the problem? Many thanks, Bin

raphenya commented 1 year ago

@DrBinZhu Please use rgi auto_load as it will configure the latest databases needed.

raphenya commented 1 year ago

@DrBinZhu Dang it! I'm getting this error too. I will investigate. Cheers.

raphenya commented 1 year ago

@DrBinZhu Ok, so far, it looks like it MacOS a related issue. I used Monterey and got the same error, but using Ubuntu 18.04.6 LTS I have no issues.

raphenya commented 11 months ago

@DrBinZhu I was able to load databases using new version see #235. Re-open the issue if you still have the same problem. Cheers.

DrBinZhu commented 11 months ago

Thanks, @raphenya. Bin

On Mon, Sep 25, 2023 at 1:02 PM amos @.***> wrote:

@DrBinZhu https://github.com/DrBinZhu I was able to load databases using new version see #235 https://github.com/arpcard/rgi/issues/235. Re-open the issue if you still have the same problem. Cheers.

— Reply to this email directly, view it on GitHub https://github.com/arpcard/rgi/issues/230#issuecomment-1734141947, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM7O2IQ2VO3OBZ2I5RF75VDX4G2L3ANCNFSM6AAAAAAYUBSYQM . You are receiving this because you were mentioned.Message ID: @.***>

peruzzoari commented 7 months ago

Hello, I reopen this issue as I am encountering the same problem as described above by @DrBinZhu, with the same Error message

` This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module:

    if __name__ == '__main__':
        freeze_support()
        ...

The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.`

My machine is a Mac, my processor is osx-arm 64, I also had issues installing Conda environments which I solved by using the following example:

conda create -n rgi_env conda activate rgi_env conda config --env --set subdir osx-64

In this way I was able to install rig which seems to be fine and I also used the autoload function you suggested in #235 The databases seem loaded correctly and rgi database --all -v outputs the list of databases as expected.

However when I try and run the code I get the error mentioned above. Using the --debug mode the job produces temporary files up to the blast annotation step (arg_contigs.fasta.temp.contig.fsa.blastRes.xml) and seems to freeze and have issues with the multiprocessing/threading.

I am not sure how to solve this, therefore I look forward receiving your suggestions! thanks

the packages in my Conda env

Name Version Build Channel

archspec 0.2.2 pyhd8ed1ab_0 conda-forge arrow-cpp 10.0.1 h57ddcff_4_cpu conda-forge aws-c-auth 0.6.21 h268417e_3 conda-forge aws-c-cal 0.5.20 h2f96456_3 conda-forge aws-c-common 0.8.5 hb7f2c08_0 conda-forge aws-c-compression 0.2.16 hfb1f584_0 conda-forge aws-c-event-stream 0.2.17 h38cbd52_1 conda-forge aws-c-http 0.7.0 h4f27208_0 conda-forge aws-c-io 0.13.12 hb3bb6ef_1 conda-forge aws-c-mqtt 0.7.13 h32e11d9_12 conda-forge aws-c-s3 0.2.1 h9eec77e_4 conda-forge aws-c-sdkutils 0.1.7 hfb1f584_0 conda-forge aws-checksums 0.1.14 hfb1f584_0 conda-forge aws-crt-cpp 0.18.16 ha874d0c_8 conda-forge aws-sdk-cpp 1.9.379 hdb1f066_7 conda-forge bamtools 2.5.1 h6151dfb_10 bioconda beautifulsoup4 4.9.3 pyhb0f4dca_0 conda-forge bedtools 2.31.1 h6372da2_0 bioconda biopython 1.78 py38h96a0964_2 conda-forge blast 2.14.1 pl5321h91c44f7_0 bioconda bokeh 3.1.1 pyhd8ed1ab_0 conda-forge bowtie2 2.5.3 py38hbee1746_0 bioconda brotli 1.0.9 hb7f2c08_9 conda-forge brotli-bin 1.0.9 hb7f2c08_9 conda-forge brotlipy 0.7.0 py38hcafd530_1006 conda-forge bwa 0.7.17 h45fc8d7_11 bioconda bzip2 1.0.8 h10d778d_5 conda-forge c-ares 1.26.0 h10d778d_0 conda-forge ca-certificates 2024.2.2 h8857fd0_0 conda-forge certifi 2024.2.2 pyhd8ed1ab_0 conda-forge cffi 1.14.6 py38h9688ba1_0 conda-forge chardet 3.0.4 py38h5347e94_1008 conda-forge charset-normalizer 3.3.2 pypi_0 pypi click 8.1.7 unix_pyh707e725_0 conda-forge cloudpickle 3.0.0 pyhd8ed1ab_0 conda-forge colorama 0.4.6 pyhd8ed1ab_0 conda-forge contourpy 1.1.1 py38h15a1a5b_1 conda-forge cryptography 39.0.0 py38ha6c3189_0 conda-forge curl 7.87.0 haf73cf8_0 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge cytoolz 0.12.3 py38hae2e43d_0 conda-forge dask 2023.5.0 pyhd8ed1ab_0 conda-forge dask-core 2023.5.0 pyhd8ed1ab_0 conda-forge diamond 0.8.36 h94ec1f5_7 bioconda distributed 2023.5.0 pyhd8ed1ab_0 conda-forge entrez-direct 16.2 h193322a_1 bioconda exceptiongroup 1.2.0 pyhd8ed1ab_2 conda-forge filetype 1.2.0 pyhd8ed1ab_0 conda-forge fonttools 4.48.1 py38hae2e43d_0 conda-forge freetype 2.12.1 h60636b9_2 conda-forge fsspec 2024.2.0 pyhca7485f_0 conda-forge gettext 0.21.1 h8a4c099_0 conda-forge gflags 2.2.2 hb1e8313_1004 conda-forge giflib 5.2.1 hb7f2c08_3 conda-forge glog 0.6.0 h8ac2a54_0 conda-forge htslib 1.9 h356306b_9 bioconda icu 73.2 hf5e326d_0 conda-forge idna 2.10 pyh9f0ad1d_0 conda-forge importlib-metadata 7.0.1 pyha770c72_0 conda-forge importlib-resources 6.1.1 pyhd8ed1ab_0 conda-forge importlib_metadata 7.0.1 hd8ed1ab_0 conda-forge importlib_resources 6.1.1 pyhd8ed1ab_0 conda-forge iniconfig 2.0.0 pyhd8ed1ab_0 conda-forge jbig 2.1 h0d85af4_2003 conda-forge jellyfish 2.2.10 h04f5b5a_1 bioconda jemalloc 5.3.0 hf0c8a7f_0 conda-forge jinja2 3.1.3 pyhd8ed1ab_0 conda-forge jpeg 9e hb7f2c08_3 conda-forge kiwisolver 1.4.5 py38h15a1a5b_1 conda-forge kma 1.4.14 h45fc8d7_0 bioconda krb5 1.20.1 h0165f36_0 conda-forge lcms2 2.12 h577c468_0 conda-forge lerc 2.2.1 h046ec9c_0 conda-forge libabseil 20220623.0 cxx17_h844d122_6 conda-forge libarrow 10.0.1 hc216794_4_cpu conda-forge libblas 3.9.0 20_osx64_openblas conda-forge libbrotlicommon 1.0.9 hb7f2c08_9 conda-forge libbrotlidec 1.0.9 hb7f2c08_9 conda-forge libbrotlienc 1.0.9 hb7f2c08_9 conda-forge libcblas 3.9.0 20_osx64_openblas conda-forge libcrc32c 1.1.2 he49afe7_0 conda-forge libcurl 7.87.0 haf73cf8_0 conda-forge libcxx 16.0.6 hd57cbcb_0 conda-forge libdeflate 1.7 h35c211d_5 conda-forge libedit 3.1.20191231 hed1e85f_2 conda-forge libev 4.33 h10d778d_2 conda-forge libevent 2.1.10 h815e4d9_4 conda-forge libffi 3.3 h046ec9c_2 conda-forge libgfortran 5.0.0 13_2_0_h97931a8_2 conda-forge libgfortran5 13.2.0 h2873a65_2 conda-forge libgoogle-cloud 2.5.0 h32e9745_1 conda-forge libgrpc 1.51.1 h834a566_0 conda-forge libhwloc 2.9.3 default_h24e0189_1009 conda-forge libiconv 1.17 hd75f5a5_2 conda-forge libidn2 2.3.7 h10d778d_0 conda-forge libjemalloc 5.3.0 hf0c8a7f_0 conda-forge liblapack 3.9.0 20_osx64_openblas conda-forge libnghttp2 1.51.0 h0dd9d14_0 conda-forge libopenblas 0.3.25 openmp_hfef2a42_0 conda-forge libpng 1.6.42 h92b6c6a_0 conda-forge libprotobuf 3.21.12 h7d26f99_2 conda-forge libssh2 1.10.0 h7535e13_3 conda-forge libthrift 0.16.0 h08c06f4_2 conda-forge libtiff 4.3.0 h1167814_1 conda-forge libunistring 0.9.10 h0d85af4_0 conda-forge libutf8proc 2.8.0 hb7f2c08_0 conda-forge libwebp 1.2.2 h28dabe5_0 conda-forge libwebp-base 1.2.2 h0d85af4_1 conda-forge libxcb 1.13 h0d85af4_1004 conda-forge libxml2 2.12.5 hc0ae0f7_0 conda-forge libxslt 1.1.39 h03b04e6_0 conda-forge libzlib 1.2.13 h8a1eda9_5 conda-forge llvm-openmp 17.0.6 hb6ac08f_0 conda-forge locket 1.0.0 pyhd8ed1ab_0 conda-forge lxml 5.1.0 pypi_0 pypi lz4 4.3.3 py38h5f57667_0 conda-forge lz4-c 1.9.4 hf0c8a7f_0 conda-forge markupsafe 2.1.5 py38hae2e43d_0 conda-forge matplotlib 3.7.3 py38h50d1736_0 conda-forge matplotlib-base 3.7.3 py38hcd1b199_0 conda-forge msgpack-python 1.0.7 py38ha94643f_0 conda-forge munkres 1.1.4 pyh9f0ad1d_0 conda-forge ncbi-vdb 3.0.0 pl5321h9722bc1_0 bioconda ncurses 6.1 h0a44026_1002 conda-forge numpy 1.24.4 py38h9a4a08f_0 conda-forge oligoarrayaux 3.8 h770b8ee_0 bioconda openjpeg 2.5.0 h69f46e4_0 conda-forge openssl 1.1.1w h8a1eda9_0 conda-forge orc 1.8.2 ha9d861c_2 conda-forge packaging 23.2 pyhd8ed1ab_0 conda-forge pandas 2.0.3 py38h78e6021_1 conda-forge parquet-cpp 1.5.1 2 conda-forge partd 1.4.1 pyhd8ed1ab_0 conda-forge patsy 0.5.6 pyhd8ed1ab_0 conda-forge pcre 8.45 he49afe7_0 conda-forge perl 5.32.1 7_h10d778d_perl5 conda-forge perl-archive-tar 2.40 pl5321hdfd78af_0 bioconda perl-carp 1.50 pl5321hd8ed1ab_0 conda-forge perl-common-sense 3.75 pl5321hd8ed1ab_0 conda-forge perl-compress-raw-bzip2 2.201 pl5321h775f41a_0 conda-forge perl-compress-raw-zlib 2.202 pl5321h775f41a_0 conda-forge perl-encode 3.19 pl5321hb7f2c08_0 conda-forge perl-exporter 5.74 pl5321hd8ed1ab_0 conda-forge perl-exporter-tiny 1.002002 pl5321hd8ed1ab_0 conda-forge perl-extutils-makemaker 7.70 pl5321hd8ed1ab_0 conda-forge perl-io-compress 2.201 pl5321h7133b54_2 bioconda perl-io-zlib 1.14 pl5321hdfd78af_0 bioconda perl-json 4.10 pl5321hdfd78af_0 bioconda perl-json-xs 2.34 pl5321hcd10b59_5 bioconda perl-list-moreutils 0.430 pl5321hdfd78af_0 bioconda perl-list-moreutils-xs 0.430 pl5321ha5712d3_1 bioconda perl-parent 0.241 pl5321hd8ed1ab_0 conda-forge perl-pathtools 3.75 pl5321h775f41a_0 conda-forge perl-scalar-list-utils 1.63 pl5321h775f41a_0 conda-forge perl-storable 3.15 pl5321h775f41a_0 conda-forge perl-types-serialiser 1.01 pl5321hdfd78af_0 bioconda pillow 9.1.1 py38h21af888_1 conda-forge pip 24.0 pyhd8ed1ab_0 conda-forge platformdirs 4.2.0 pyhd8ed1ab_0 conda-forge pluggy 1.4.0 pyhd8ed1ab_0 conda-forge pooch 1.8.0 pyhd8ed1ab_0 conda-forge prodigal 2.6.3 h2413b67_7 bioconda psutil 5.9.8 py38hae2e43d_0 conda-forge pthread-stubs 0.4 hc929b4f_1001 conda-forge pyahocorasick 1.4.0 py38h4d0b108_2 conda-forge pyarrow 10.0.1 py38h450cdb6_4_cpu conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pyfaidx 0.8.1.1 pyhdfd78af_0 bioconda pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge pyrodigal 3.3.0 py38h604ca31_0 bioconda pysam 0.16.0.1 py38h4ebacbe_3 bioconda pysocks 1.7.1 pyha2e5f31_6 conda-forge pytest 8.0.0 pyhd8ed1ab_0 conda-forge python 3.8.10 h0e5c897_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-tzdata 2023.4 pyhd8ed1ab_0 conda-forge python_abi 3.8 4_cp38 conda-forge pytz 2024.1 pyhd8ed1ab_0 conda-forge pyvcf3 1.0.3 pyhdfd78af_0 bioconda pyyaml 6.0.1 py38hcafd530_1 conda-forge re2 2022.06.01 hb486fe8_1 conda-forge readline 8.0 hed1e85f_1 conda-forge requests 2.31.0 pypi_0 pypi rgi 6.0.3 pypi_0 pypi samtools 1.9 h8aa4d43_12 bioconda scipy 1.10.1 py38h9cf86d3_3 conda-forge seaborn 0.13.2 hd8ed1ab_0 conda-forge seaborn-base 0.13.2 pyhd8ed1ab_0 conda-forge setuptools 69.0.3 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.10 h225ccf5_0 conda-forge sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge soupsieve 2.5 pyhd8ed1ab_1 conda-forge sqlite 3.40.1 h880c91c_0
statsmodels 0.14.1 py38h7092abf_0 conda-forge tbb 2021.11.0 h7728843_1 conda-forge tblib 3.0.0 pyhd8ed1ab_0 conda-forge tk 8.6.13 h1abcd95_1 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge toolz 0.12.1 pyhd8ed1ab_0 conda-forge tornado 6.3.3 py38hcafd530_1 conda-forge unicodedata2 15.1.0 py38hcafd530_0 conda-forge urllib3 1.25.11 py_0 conda-forge wget 1.20.3 h52ee1ee_1 conda-forge wheel 0.42.0 pyhd8ed1ab_0 conda-forge xorg-libxau 1.0.11 h0dc2134_0 conda-forge xorg-libxdmcp 1.1.3 h35c211d_0 conda-forge xyzservices 2023.10.1 pyhd8ed1ab_0 conda-forge xz 5.2.6 h775f41a_0 conda-forge yaml 0.2.5 h0d85af4_2 conda-forge zict 3.0.0 pyhd8ed1ab_0 conda-forge zipp 3.17.0 pyhd8ed1ab_0 conda-forge zlib 1.2.13 h8a1eda9_5 conda-forge zstd 1.5.5 h829000d_0 conda-forge