search

CAMI-challenge / CAMISIM

CAMISIM: Simulating metagenomes and microbial communities
https://data.cami-challenge.org/participate
Apache License 2.0
166 stars 36 forks source link

NoneType path error when trying to run metagenome_from_profile.py from default_config.ini #191

Open cberta11 opened 1 month ago

cberta11 commented 1 month ago

Hello! I am new to CAMISIM and am running into a bit of a roadblock. I am attempting to run metagenome_from_profile.py using the default_config.ini file using the instructions that you gave in the manuel and in https://github.com/CAMI-challenge/CAMISIM/issues/48, however I keep getting: image Below I have attached the error readout with the --debug flag. I also attached the copy of my default_config.ini file as well as the package list from the mamba env in which I am attempting to run CAMISIM on the chance I have the incorrect version of one of the required packages. Any help would be greatly appreciated!

image

image

Mamba env package list

(camisim-env2) joeberta@JoesLaptop:/mnt/d/camisim/CAMISIM-1.3$ mamba list

packages in environment at /home/joeberta/miniforge-pypy3/envs/camisim-env2:

#

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 2_gnu conda-forge alsa-lib 1.2.12 h4ab18f5_0 conda-forge attr 2.5.1 h166bdaf_1 conda-forge biom-format 2.1.16 py39hd3abc70_1 conda-forge biopython 1.84 py39hd3abc70_0 conda-forge brotli 1.1.0 hd590300_1 conda-forge brotli-bin 1.1.0 hd590300_1 conda-forge bzip2 1.0.8 h4bc722e_7 conda-forge c-ares 1.32.2 h4bc722e_0 conda-forge ca-certificates 2024.7.4 hbcca054_0 conda-forge cached-property 1.5.2 hd8ed1ab_1 conda-forge cached_property 1.5.2 pyha770c72_1 conda-forge cairo 1.18.0 hbb29018_2 conda-forge certifi 2024.7.4 pyhd8ed1ab_0 conda-forge click 8.1.7 unix_pyh707e725_0 conda-forge contourpy 1.2.1 py39h7633fee_0 conda-forge coreutils 8.25 1 bioconda curl 8.8.0 he654da7_1 conda-forge cycler 0.12.1 pyhd8ed1ab_0 conda-forge dbus 1.13.6 h5008d03_3 conda-forge ete3 3.1.3 pyhd8ed1ab_0 conda-forge expat 2.6.2 h59595ed_0 conda-forge font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge font-ttf-inconsolata 3.000 h77eed37_0 conda-forge font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge font-ttf-ubuntu 0.83 h77eed37_2 conda-forge fontconfig 2.14.2 h14ed4e7_0 conda-forge fonts-conda-ecosystem 1 0 conda-forge fonts-conda-forge 1 0 conda-forge fonttools 4.53.1 py39hcd6043d_0 conda-forge freetype 2.12.1 h267a509_2 conda-forge gettext 0.22.5 h59595ed_2 conda-forge gettext-tools 0.22.5 h59595ed_2 conda-forge giflib 5.2.2 hd590300_0 conda-forge glib 2.80.3 h8a4344b_1 conda-forge glib-tools 2.80.3 h73ef956_1 conda-forge graphite2 1.3.13 h59595ed_1003 conda-forge gst-plugins-base 1.24.5 hbaaba92_0 conda-forge gstreamer 1.24.5 haf2f30d_0 conda-forge h5py 3.11.0 nompi_py39h24b94d4_102 conda-forge harfbuzz 8.5.0 hfac3d4d_0 conda-forge hdf5 1.14.3 nompi_hdf9ad27_105 conda-forge icu 73.2 h59595ed_0 conda-forge importlib-resources 6.4.0 pyhd8ed1ab_0 conda-forge importlib_resources 6.4.0 pyhd8ed1ab_0 conda-forge keyutils 1.6.1 h166bdaf_0 conda-forge kiwisolver 1.4.5 py39h7633fee_1 conda-forge krb5 1.21.3 h659f571_0 conda-forge lame 3.100 h166bdaf_1003 conda-forge lcms2 2.16 hb7c19ff_0 conda-forge ld_impl_linux-64 2.40 hf3520f5_7 conda-forge lerc 4.0.0 h27087fc_0 conda-forge libaec 1.1.3 h59595ed_0 conda-forge libasprintf 0.22.5 h661eb56_2 conda-forge libasprintf-devel 0.22.5 h661eb56_2 conda-forge libblas 3.9.0 22_linux64_openblas conda-forge libbrotlicommon 1.1.0 hd590300_1 conda-forge libbrotlidec 1.1.0 hd590300_1 conda-forge libbrotlienc 1.1.0 hd590300_1 conda-forge libcap 2.69 h0f662aa_0 conda-forge libcblas 3.9.0 22_linux64_openblas conda-forge libclang-cpp15 15.0.7 default_h127d8a8_5 conda-forge libclang13 18.1.8 default_h6ae225f_0 conda-forge libcups 2.3.3 h4637d8d_4 conda-forge libcurl 8.8.0 hca28451_1 conda-forge libdeflate 1.20 hd590300_0 conda-forge libedit 3.1.20191231 he28a2e2_2 conda-forge libev 4.33 hd590300_2 conda-forge libevent 2.1.12 hf998b51_1 conda-forge libexpat 2.6.2 h59595ed_0 conda-forge libffi 3.4.2 h7f98852_5 conda-forge libflac 1.4.3 h59595ed_0 conda-forge libgcc 7.2.0 h69d50b8_2 conda-forge libgcc-ng 14.1.0 h77fa898_0 conda-forge libgcrypt 1.11.0 h4ab18f5_0 conda-forge libgettextpo 0.22.5 h59595ed_2 conda-forge libgettextpo-devel 0.22.5 h59595ed_2 conda-forge libgfortran-ng 14.1.0 h69a702a_0 conda-forge libgfortran5 14.1.0 hc5f4f2c_0 conda-forge libglib 2.80.3 h8a4344b_1 conda-forge libgomp 14.1.0 h77fa898_0 conda-forge libgpg-error 1.50 h4f305b6_0 conda-forge libiconv 1.17 hd590300_2 conda-forge libjpeg-turbo 3.0.0 hd590300_1 conda-forge liblapack 3.9.0 22_linux64_openblas conda-forge libllvm15 15.0.7 hb3ce162_4 conda-forge libllvm18 18.1.8 hc9dba70_0 conda-forge libnghttp2 1.58.0 h47da74e_1 conda-forge libnsl 2.0.1 hd590300_0 conda-forge libogg 1.3.5 h4ab18f5_0 conda-forge libopenblas 0.3.27 pthreads_hac2b453_1 conda-forge libopus 1.3.1 h7f98852_1 conda-forge libpng 1.6.43 h2797004_0 conda-forge libpq 16.3 ha72fbe1_0 conda-forge libsndfile 1.2.2 hc60ed4a_1 conda-forge libsqlite 3.46.0 hde9e2c9_0 conda-forge libssh2 1.11.0 h0841786_0 conda-forge libstdcxx-ng 14.1.0 hc0a3c3a_0 conda-forge libsystemd0 255 h3516f8a_1 conda-forge libtiff 4.6.0 h1dd3fc0_3 conda-forge libuuid 2.38.1 h0b41bf4_0 conda-forge libvorbis 1.3.7 h9c3ff4c_0 conda-forge libwebp-base 1.4.0 hd590300_0 conda-forge libxcb 1.16 hd590300_0 conda-forge libxcrypt 4.4.36 hd590300_1 conda-forge libxkbcommon 1.7.0 h2c5496b_1 conda-forge libxml2 2.12.7 h4c95cb1_3 conda-forge libxslt 1.1.39 h76b75d6_0 conda-forge libzlib 1.3.1 h4ab18f5_1 conda-forge lxml 5.2.2 py39h8a3eb29_0 conda-forge lz4-c 1.9.4 hcb278e6_0 conda-forge matplotlib 3.9.1 py39hf3d152e_0 conda-forge matplotlib-base 3.9.1 py39hd75cb13_0 conda-forge mpg123 1.32.6 h59595ed_0 conda-forge munkres 1.0.7 py_1 bioconda mysql-common 8.3.0 hf1915f5_4 conda-forge mysql-libs 8.3.0 hca2cd23_4 conda-forge ncurses 6.5 h59595ed_0 conda-forge nextflow 24.04.3 hdfd78af_0 bioconda nspr 4.35 h27087fc_0 conda-forge nss 3.102 h593d115_0 conda-forge numpy 2.0.0 py39ha0965c0_0 conda-forge openjdk 21.0.0 haa376d0_0 conda-forge openjpeg 2.5.2 h488ebb8_0 conda-forge openssl 3.3.1 h4ab18f5_1 conda-forge packaging 24.1 pyhd8ed1ab_0 conda-forge pandas 2.2.2 py39hfc16268_1 conda-forge pcre2 10.44 h0f59acf_0 conda-forge pillow 10.4.0 py39h16a7006_0 conda-forge pip 24.0 pyhd8ed1ab_0 conda-forge pixman 0.43.2 h59595ed_0 conda-forge ply 3.11 pyhd8ed1ab_2 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge pulseaudio-client 17.0 hb77b528_0 conda-forge pyparsing 3.1.2 pyhd8ed1ab_0 conda-forge pyqt 5.15.9 py39h52134e7_5 conda-forge pyqt5-sip 12.12.2 py39h3d6467e_5 conda-forge python 3.9.19 h0755675_0_cpython conda-forge python-dateutil 2.9.0 pyhd8ed1ab_0 conda-forge python-tzdata 2024.1 pyhd8ed1ab_0 conda-forge python_abi 3.9 4_cp39 conda-forge pytz 2024.1 pyhd8ed1ab_0 conda-forge qhull 2020.2 h434a139_5 conda-forge qt-main 5.15.8 ha2b5568_22 conda-forge readline 8.2 h8228510_1 conda-forge scipy 1.13.1 py39haf93ffa_0 conda-forge setuptools 70.3.0 pyhd8ed1ab_0 conda-forge sip 6.7.12 py39h3d6467e_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge tk 8.6.13 noxft_h4845f30_101 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge tornado 6.4.1 py39hd3abc70_0 conda-forge tzdata 2024a h0c530f3_0 conda-forge unicodedata2 15.1.0 py39hd1e30aa_0 conda-forge wheel 0.43.0 pyhd8ed1ab_1 conda-forge xcb-util 0.4.1 hb711507_2 conda-forge xcb-util-image 0.4.0 hb711507_2 conda-forge xcb-util-keysyms 0.4.1 hb711507_0 conda-forge xcb-util-renderutil 0.3.10 hb711507_0 conda-forge xcb-util-wm 0.4.2 hb711507_0 conda-forge xkeyboard-config 2.42 h4ab18f5_0 conda-forge xorg-fixesproto 5.0 h7f98852_1002 conda-forge xorg-inputproto 2.3.2 h7f98852_1002 conda-forge xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.1.1 hd590300_0 conda-forge xorg-libsm 1.2.4 h7391055_0 conda-forge xorg-libx11 1.8.9 hb711507_1 conda-forge xorg-libxau 1.0.11 hd590300_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h0b41bf4_2 conda-forge xorg-libxfixes 5.0.3 h7f98852_1004 conda-forge xorg-libxi 1.7.10 h7f98852_0 conda-forge xorg-libxrender 0.9.11 hd590300_0 conda-forge xorg-libxt 1.3.0 hd590300_1 conda-forge xorg-libxtst 1.2.3 h7f98852_1002 conda-forge xorg-recordproto 1.14.2 h7f98852_1002 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge xorg-xf86vidmodeproto 2.3.1 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.6 h166bdaf_0 conda-forge zipp 3.19.2 pyhd8ed1ab_0 conda-forge zlib 1.3.1 h4ab18f5_1 conda-forge zstd 1.5.6 ha6fb4c9_0 conda-forge

AlphaSquad commented 1 month ago

Hi, thank you for your interest in CAMISIM! When you want to create a metagenome from profile using the metagenome_from_profile.py script, you are required to provide a profile with the -p option. Since this is not entirely obvious I added the path to a small test BIOM profile as default option which should "fix" your error if you pull the latest version of CAMISIM. Since you seem to be testing, I recommend using the mini_config.ini instead of the default_config.ini

cberta11 commented 1 month ago

Thank you for your quick response! Since originally posting, I tried running metagenomesimulation.py using mini_config.ini as I saw suggested in https://github.com/CAMI-challenge/CAMISIM/issues/69. image However, this raised the "Empty BAM file list in line 106" error as seen in https://github.com/CAMI-challenge/CAMISIM/issues/150 et al. Taking the advice from the submitter in that issue, I created a new mamba envirnonment with basically every dependency I could find mentioned in the user manual and/or issue forum as being necessary: image (No clue if changing from python 3.9 to 3.8 was necessary but that's what that user had and I know v3.10 causes a seperate issue) This appeared to be working, however, I repeatedly saw an error flash by relating to ncursesw: image I saw you mention that this might be an issue in https://github.com/CAMI-challenge/CAMISIM/issues/150, so I wnet ahead and aborted the process to try to proactivly tackle another anticipated issue. Ncursesw does not appear to have a anaconda package, so I just ran: image This appeared to solve the ncursesw library issue, and the "Empty BAM file list in line 106" error was no longer raised during the "Generating gold standard assembly" stage as it had previously and ran all the way through sucessfully. I tried running metagenome_from_profile.py with the mini_config.ini as you just suggested: image This still raises the same "TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType" but I am going to try constructing a biom profile to feed into it since you are saying it is expecting one.

My end goal is to create 16S rRNA metagenomic profiles that mimic species and abundances commonly found in wastewater samples. I saw reference to 16S with CAMISIM being possible (I think in the user manual or the publication?) but my question is do I just need to pull 16S rRNA nucleotide sequences for each species from NCBI or can I still use full genomes from NCBI and just change some sort of setting to only generate reads from the imaginary rRNA "amplicon" region? Also thank you so much for your work, this program is immensly useful!

AlphaSquad commented 1 month ago

Did you pull the latest version of CAMISIM (Since it says "1.3" which is the most recent tagged version, but does not include some later bug fixes)? There is an small example biom profile in the defaults folder (mini.biom) which you can provide with the -p option for testing. Currently you would need to pull your own 16S rRNA sequences out of the genomes, but there will soon be a metatranscriptomic mode as part of "CAMISIM 2"; you can already have a look at the metat branch. In this mode it might be possible to provide gff files only containing the 16S regions and then just simulate these.