search

tseemann / snippy

:scissors: :zap: Rapid haploid variant calling and core genome alignment
GNU General Public License v2.0
474 stars 115 forks source link

Got broken pipeline when running freebayes-parallel #561

Open buihoangphuc412 opened 1 year ago

buihoangphuc412 commented 1 year ago

I got this error like this. Does anyone can help to fix?? I run command on HPC: snippy --R1 $p_trim/VN0467_1_paired.fastq.gz --R2 $p_trim/VN0467_2_paired.fastq.gz --outdir /lustre7/home/buihoangphuc412/Projects/GWAS/variant/VN0467 --prefix VN0467 --ref $p_ref/genome/CHC97.fasta --report --cpus 4 --ram 2 --basequal 15 VN0467.log

echo snippy 4.6.0

cd /lustre7/home/buihoangphuc412

/home/buihoangphuc412/miniconda3/envs/pyseer/bin/snippy --R1 /lustre7/home/buihoangphuc412/Projects/GWAS/trim/VN0467_1_paired.fastq.gz --R2 /lustre7/home/buihoangphuc412/Projects/GWAS/trim/VN0467_2_paired.fastq.gz --outdir /lustre7/home/buihoangphuc412/Projects/GWAS/variant/VN0467 --prefix VN0467 --ref /lustre7/home/buihoangphuc412/Projects/GWAS/ref/genome/CHC97.fasta --report --cpus 4 --ram 2 --basequal 15

samtools faidx reference/ref.fa

bwa index reference/ref.fa

[bwa_index] Pack FASTA... 0.00 sec [bwa_index] Construct BWT for the packed sequence... [bwa_index] 0.25 seconds elapse. [bwa_index] Update BWT... 0.02 sec [bwa_index] Pack forward-only FASTA... 0.00 sec [bwa_index] Construct SA from BWT and Occ... 0.09 sec [main] Version: 0.7.17-r1188 [main] CMD: bwa index reference/ref.fa [main] Real time: 0.420 sec; CPU: 0.368 sec

mkdir -p reference/genomes && cp -f reference/ref.fa reference/genomes/ref.fa

ln -sf reference/ref.fa .

ln -sf reference/ref.fa.fai .

mkdir -p reference/ref && gzip -c reference/ref.gff > reference/ref/genes.gff.gz

bwa mem -Y -M -R '@RG\tID:VN0467\tSM:VN0467' -t 4 reference/ref.fa /lustre7/home/buihoangphuc412/Projects/GWAS/trim/VN0467_1_paired.fastq.gz /lustre7/home/buihoangphuc412/Projects/GWAS/trim/VN0467_2_paired.fastq.gz | samclip --max 10 --ref reference/ref.fa.fai | samtools sort -n -l 0 -T /tmp --threads 1 -m 1000M | samtools fixmate -m --threads 1 - - | samtools sort -l 0 -T /tmp --threads 1 -m 1000M | samtools markdup -T /tmp --threads 1 -r -s - - > VN0467.bam

COMMAND: samtools markdup -T /tmp --threads 1 -r -s - - READ: 907222 WRITTEN: 895475 EXCLUDED: 106405 EXAMINED: 800817 PAIRED: 752182 SINGLE: 48635 DUPLICATE PAIR: 1168 DUPLICATE SINGLE: 10579 DUPLICATE PAIR OPTICAL: 0 DUPLICATE SINGLE OPTICAL: 0 DUPLICATE NON PRIMARY: 0 DUPLICATE NON PRIMARY OPTICAL: 0 DUPLICATE PRIMARY TOTAL: 11747 DUPLICATE TOTAL: 11747 ESTIMATED_LIBRARY_SIZE: 120974295

samtools index VN0467.bam

fasta_generate_regions.py reference/ref.fa.fai 237553 > reference/ref.txt

freebayes-parallel reference/ref.txt 4 -p 2 -P 0 -C 2 -F 0.05 --min-coverage 10 --min-repeat-entropy 1.0 -q 15 -m 60 --strict-vcf -f reference/ref.fa VN0467.bam > VN0467.raw.vcf

vcfstreamsort: symbol lookup error: /lustre7/home/buihoangphuc412/miniconda3/envs/pyseer/bin/../lib/libvcflib.so.1: undefined symbol: wavefront_align vcfuniq: symbol lookup error: /lustre7/home/buihoangphuc412/miniconda3/envs/pyseer/bin/../lib/libvcflib.so.1: undefined symbol: wavefront_align Traceback (most recent call last): File "/lustre7/home/buihoangphuc412/miniconda3/envs/pyseer/bin/vcffirstheader", line 17, in print(line.strip()) BrokenPipeError: [Errno 32] Broken pipe

Isoris commented 1 year ago

When I had problems with freebayes, it was because I didnt have the last version which will use python 3 and not python 2. try to update freebayes and use the newest version of snippy, look at the path

which freebayes which -a freebayes

Maybe this can help?

Isoris commented 1 year ago

your freebayes is too old. snippy too. the newest version had a corrected version of vcffirshheader.

jbrizuelagabaldon commented 1 year ago

Their snippy is the latest version 4.6.0 and freebayes is probably the correct version too, given that I am gettin gthe same error. From what i have learned, vcflib got updated and that breaks snippy.

Try specifiying this when installing with conda: snippy=4.6.0=hdfd78af_3

jbrizuelagabaldon commented 1 year ago

Ignore the previous suggestion that didn't work, at least not for me. I have managed to fix it by using the follwing recipe (in a .yaml file). `channels:

jbrizuelagabaldon commented 1 year ago

Also you need to be careful with bcftools and snpeff. When I had bcftools 1.17 and snpeff 5.0 snippy would detect 0 snps across all samples. I have tested now bcftool 1.15.0 and snpeff 5.1 and it works

jbrizuelagabaldon commented 1 year ago

i have exported the environment list to a recipe that should work in case you are still having this issue. Just rename the file from .txt to .yaml and it should be fine. snippy_that_works.txt

RaverJay commented 1 year ago

i have exported the environment list to a recipe that should work in case you are still having this issue. Just rename the file from .txt to .yaml and it should be fine. snippy_that_works.txt

mamba env create -n snippy -f snippy_that_works.yaml does not solve for me with mambaforge:

Could not solve for environment specs
The following packages are incompatible
├─ c-ares ==1.19.1 hd590300_0 is uninstallable because it conflicts with any installable versions previously reported;
├─ libdb ==6.2.32 h9c3ff4c_0 is uninstallable because it conflicts with any installable versions previously reported;
├─ libdeflate ==1.19 hd590300_0 is uninstallable because it conflicts with any installable versions previously reported;
├─ parallel ==20230722 ha770c72_0 is uninstallable because it conflicts with any installable versions previously reported;
├─ perl-extutils-makemaker ==7.70 pl5321hd8ed1ab_0 is uninstallable because it conflicts with any installable versions previously reported;
├─ perl-module-build ==0.4234 pl5321ha770c72_0 is uninstallable because it conflicts with any installable versions previously reported;
├─ perl-net-ssleay ==1.92 pl5321hf14f497_1 is uninstallable because it conflicts with any installable versions previously reported;
└─ perl-xml-parser ==2.44_01 pl5321hc3e0081_1003 is uninstallable because it conflicts with any installable versions previously reported.

Also tried: mamba create -n snippy snippy==4.6.0 vcflib==1.0.1 bcftools==1.15.0 snpeff==5.1

which builds an environment but errors out during the run

Seems that Snippy is properly in dependency hell by now :(

jbrizuelagabaldon commented 1 year ago

Yes, I have seen that my build doesn't work for everyone...

I also used mamba.

I specified exactly those dependencies and worked for me... My freebayes is 1.3.6, samtools 1.17 and bwa 0.7.17 in case it helps.

I'd suggest playing around with the different versions. My colleague also couldn't make my version worked but managed to install it in the end, will ask for their dependencies.

RaverJay commented 1 year ago

Okay so my above mentioned env actually calls the SNPs when working with a .fa instead of a .gbff (is .gbff supported or does it need .gbk?) and only dies at making the consensus:

### bcftools consensus --sample CM13 -f reference/ref.fa -o snps.consensus.fa snps.vcf.gz

No such sample: CM13

I'm not too proficient with bcftools so no idea why sample isn't specified in my vcf

Adding the deps you just posted didn't fix it sadly

At least these runs yield the snps.tab file that is the main result

Isoris commented 1 year ago

No guys it's because some parts of the pipeline use scrips like VCF first header which are old and expect python 2 syntax.

You have to get the latest versions of each components or to go manually into the file that gives the error and change the syntax so it can work with snippy.

I got the same error a few months ago and I got it solved by uninstalling the packages and reinstalling them.

I think the issue comes from you not using the correct versions or your paths not properly set between your virtual environments.

What is suggest is to try different things.

Create a new conda environment, download up to date snippy (the snippy dependencies will work with python3). You try to uninstall the different packages and reinstall them if necessary. Use pip install when possible.

I think that the issue comes from the fact that some parts of the script like in vcffirstheader are not good. You should use the latest version of each package and install them manually and add them to path by yourself.

You see that the error says that vcffirstheader can't strip the line and that is because it is using an old version of python and nowadays the function is written differently. So you need to update your vcffirstheader or to change this part of the code.

Also for the previous comments please uninstall each package / library of your system and of conda environment. And reinstall them.

Some commands:

To know which version of the software you are using:

which python3

To know all of the versions of a software installed on a system:

which -a python3

To uninstall them:

Sudo apt uninstall. (Or something like that)

To install perl packages please use cpan

Once you have Installed the software you need to add it to your path

You can add it to .bashrc or to .profile

nano ~/.profile

nano ~/.bashrc

You can ask chatgpt for help.

Something like PATH=$:add your path here to the new packages

Then you need to source it to activate it to the current environment

source ~/.profile Or source ~/.bashrc

For the effects to take change.

Also you have to read the logs. For instance for the vcffirstheader error. It is written in the logs that the error is caused by linestrip()

Snippy is fine and working well but what probably happened is that you installed old versions of a dependency on the base environment of conda or on the root environment or that your paths are not properly set, hence the problem.

Also be careful because python and python3 can be both installed and based on the scripts some will call python and some will call python3 which is infact the same. But two different installations from my understanding.

So you have to give the correct path too.

Isoris commented 1 year ago

Okay so my above mentioned env actually calls the SNPs when working with a .fa instead of a .gbff (is .gbff supported or does it need .gbk?) and only dies at making the consensus:

### bcftools consensus --sample CM13 -f reference/ref.fa -o snps.consensus.fa snps.vcf.gz

No such sample: CM13

I'm not too proficient with bcftools so no idea why sample isn't specified in my vcf

Adding the deps you just posted didn't fix it sadly

At least these runs yield the snps.tab file that is the main result

Could you please write the full command that you used for running snippy? And also the version of each packages on the environment?

RaverJay commented 1 year ago

snippy --cpus 60 --outdir CM13 --ref ref.fa --bam CM13.bam

Your wall of text boils down to: "I think the issue comes from you not using the correct versions or your paths not properly set between your virtual environments." Mamba manages both of these things, what it cannot manage or know is if newer versions of dependencies break the functionality.

It is not obvious which program has an incompatible version

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
alsa-lib                  1.2.10               hd590300_0    conda-forge
any2fasta                 0.4.2                hdfd78af_3    bioconda
argtable2                 2.13              h14c3975_1001    conda-forge
bcftools                  1.15                 h0ea216a_2    bioconda
bedtools                  2.31.0               hf5e1c6e_3    bioconda
blast                     2.14.1          pl5321h6f7f691_0    bioconda
blast-legacy              2.2.26               h9ee0642_3    bioconda
bwa                       0.7.17              he4a0461_11    bioconda
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.20.1               hd590300_0    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cairo                     1.16.0            h0c91306_1017    conda-forge
clustalo                  1.2.4                hdbdd923_7    bioconda
clustalw                  2.1                  h4ac6f70_9    bioconda
curl                      8.3.0                hca28451_0    conda-forge
entrez-direct             16.2                 he881be0_1    bioconda
expat                     2.5.0                hcb278e6_1    conda-forge
font-ttf-dejavu-sans-mono 2.37                 hab24e00_0    conda-forge
font-ttf-inconsolata      3.000                h77eed37_0    conda-forge
font-ttf-source-code-pro  2.038                h77eed37_0    conda-forge
font-ttf-ubuntu           0.83                 hab24e00_0    conda-forge
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonts-conda-ecosystem     1                             0    conda-forge
fonts-conda-forge         1                             0    conda-forge
freebayes                 1.3.6                hb0f3ef8_5    bioconda
freetype                  2.12.1               h267a509_2    conda-forge
gettext                   0.21.1               h27087fc_0    conda-forge
giflib                    5.2.1                h0b41bf4_3    conda-forge
graphite2                 1.3.13            h58526e2_1001    conda-forge
gsl                       2.7                  he838d99_0    conda-forge
harfbuzz                  8.2.1                h3d44ed6_0    conda-forge
htslib                    1.17                 h81da01d_2    bioconda
icu                       73.2                 h59595ed_0    conda-forge
k8                        0.2.5                hdcf5f25_4    bioconda
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.2               h659d440_0    conda-forge
lcms2                     2.15                 hb7c19ff_3    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libblas                   3.9.0           18_linux64_openblas    conda-forge
libcblas                  3.9.0           18_linux64_openblas    conda-forge
libcups                   2.3.3                h4637d8d_4    conda-forge
libcurl                   8.3.0                hca28451_0    conda-forge
libdb                     6.2.32               h9c3ff4c_0    conda-forge
libdeflate                1.19                 hd590300_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    7.2.0                h69d50b8_2    conda-forge
libgcc-ng                 13.2.0               h807b86a_2    conda-forge
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            13.2.0               h69a702a_2    conda-forge
libgfortran5              13.2.0               ha4646dd_2    conda-forge
libglib                   2.78.0               hebfc3b9_0    conda-forge
libgomp                   13.2.0               h807b86a_2    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
libidn2                   2.3.4                h166bdaf_0    conda-forge
libjpeg-turbo             3.0.0                hd590300_1    conda-forge
libnghttp2                1.52.0               h61bc06f_0    conda-forge
libnsl                    2.0.0                hd590300_1    conda-forge
libopenblas               0.3.24          pthreads_h413a1c8_0    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libsqlite                 3.43.0               h2797004_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx-ng              13.2.0               h7e041cc_2    conda-forge
libtiff                   4.6.0                ha9c0a0a_2    conda-forge
libunistring              0.9.10               h7f98852_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libwebp-base              1.3.2                hd590300_0    conda-forge
libxcb                    1.15                 h0b41bf4_0    conda-forge
libzlib                   1.2.13               hd590300_5    conda-forge
mafft                     7.520                h031d066_2    bioconda
minimap2                  2.26                 he4a0461_2    bioconda
muscle                    5.1                  h4ac6f70_3    bioconda
ncbi-vdb                  3.0.0           pl5321h87f3376_0    bioconda
ncurses                   6.4                  hcb278e6_0    conda-forge
openjdk                   20.0.2               haa376d0_2    conda-forge
openssl                   3.1.3                hd590300_0    conda-forge
paml                      4.10.6               h031d066_2    bioconda
parallel                  20230922             ha770c72_0    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
perl                      5.32.1          4_hd590300_perl5    conda-forge
perl-algorithm-diff       1.201           pl5321hd8ed1ab_0    conda-forge
perl-archive-tar          2.40            pl5321hdfd78af_0    bioconda
perl-base                 2.23            pl5321hd8ed1ab_0    conda-forge
perl-bio-asn1-entrezgene  1.73            pl5321hdfd78af_3    bioconda
perl-bio-coordinate       1.007001        pl5321hdfd78af_3    bioconda
perl-bio-featureio        1.6.905         pl5321hdfd78af_4    bioconda
perl-bio-samtools         1.43            pl5321he4a0461_4    bioconda
perl-bio-searchio-hmmer   1.7.3           pl5321hdfd78af_0    bioconda
perl-bio-tools-phylo-paml 1.7.3           pl5321hdfd78af_3    bioconda
perl-bio-tools-run-alignment-clustalw 1.7.4           pl5321hdfd78af_3    bioconda
perl-bio-tools-run-alignment-tcoffee 1.7.4           pl5321hdfd78af_5    bioconda
perl-bioperl              1.7.8                hdfd78af_1    bioconda
perl-bioperl-core         1.7.8           pl5321hdfd78af_1    bioconda
perl-bioperl-run          1.007003        pl5321hdfd78af_0    bioconda
perl-business-isbn        3.007           pl5321hd8ed1ab_0    conda-forge
perl-business-isbn-data   20210112.006    pl5321hd8ed1ab_0    conda-forge
perl-carp                 1.50            pl5321hd8ed1ab_0    conda-forge
perl-class-data-inheritable 0.09            pl5321ha770c72_0    conda-forge
perl-common-sense         3.75            pl5321hd8ed1ab_0    conda-forge
perl-compress-raw-bzip2   2.201           pl5321h166bdaf_0    conda-forge
perl-compress-raw-zlib    2.202           pl5321h166bdaf_0    conda-forge
perl-constant             1.33            pl5321hd8ed1ab_0    conda-forge
perl-data-dumper          2.183           pl5321h166bdaf_0    conda-forge
perl-db_file              1.858           pl5321h166bdaf_0    conda-forge
perl-devel-stacktrace     2.04            pl5321ha770c72_0    conda-forge
perl-digest-hmac          1.04            pl5321hdfd78af_0    bioconda
perl-digest-md5           2.58            pl5321h166bdaf_0    conda-forge
perl-encode               3.19            pl5321h166bdaf_0    conda-forge
perl-encode-locale        1.05            pl5321hdfd78af_7    bioconda
perl-exception-class      1.45            pl5321hdfd78af_0    bioconda
perl-exporter             5.74            pl5321hd8ed1ab_0    conda-forge
perl-exporter-tiny        1.002002        pl5321hd8ed1ab_0    conda-forge
perl-extutils-makemaker   7.70            pl5321hd8ed1ab_0    conda-forge
perl-file-listing         6.16            pl5321hdfd78af_0    bioconda
perl-file-slurp-tiny      0.004           pl5321hdfd78af_2    bioconda
perl-file-sort            1.01            pl5321hdfd78af_3    bioconda
perl-file-spec            3.48_01         pl5321hdfd78af_2    bioconda
perl-getopt-long          2.54            pl5321hdfd78af_0    bioconda
perl-html-parser          3.81            pl5321h4ac6f70_1    bioconda
perl-html-tagset          3.20            pl5321hdfd78af_4    bioconda
perl-http-cookies         6.10            pl5321hdfd78af_0    bioconda
perl-http-daemon          6.16            pl5321hdfd78af_0    bioconda
perl-http-date            6.06            pl5321hdfd78af_0    bioconda
perl-http-message         6.36            pl5321hdfd78af_0    bioconda
perl-http-negotiate       6.01            pl5321hdfd78af_4    bioconda
perl-inc-latest           0.500           pl5321ha770c72_0    conda-forge
perl-io-compress          2.201           pl5321hdbdd923_2    bioconda
perl-io-html              1.004           pl5321hdfd78af_0    bioconda
perl-io-socket-ssl        2.075           pl5321hd8ed1ab_0    conda-forge
perl-io-string            1.08            pl5321hdfd78af_4    bioconda
perl-io-tty               1.16            pl5321h166bdaf_0    conda-forge
perl-io-zlib              1.14            pl5321hdfd78af_0    bioconda
perl-ipc-run              20200505.0      pl5321hdfd78af_0    bioconda
perl-json                 4.10            pl5321hdfd78af_0    bioconda
perl-json-xs              2.34            pl5321h4ac6f70_6    bioconda
perl-libwww-perl          6.67            pl5321hdfd78af_0    bioconda
perl-libxml-perl          0.08            pl5321hdfd78af_3    bioconda
perl-list-moreutils       0.430           pl5321hdfd78af_0    bioconda
perl-list-moreutils-xs    0.430           pl5321h031d066_2    bioconda
perl-lwp-mediatypes       6.04            pl5321hdfd78af_1    bioconda
perl-mime-base64          3.16            pl5321h166bdaf_0    conda-forge
perl-module-build         0.4234          pl5321ha770c72_0    conda-forge
perl-net-http             6.22            pl5321hdfd78af_0    bioconda
perl-net-ssleay           1.92            pl5321hf14f497_1    conda-forge
perl-ntlm                 1.09            pl5321hdfd78af_5    bioconda
perl-parent               0.241           pl5321hd8ed1ab_0    conda-forge
perl-pathtools            3.75            pl5321h166bdaf_0    conda-forge
perl-scalar-list-utils    1.63            pl5321h166bdaf_0    conda-forge
perl-socket               2.027           pl5321h031d066_4    bioconda
perl-storable             3.15            pl5321h166bdaf_0    conda-forge
perl-sub-uplevel          0.2800          pl5321h166bdaf_0    conda-forge
perl-test-deep            1.130           pl5321hd8ed1ab_0    conda-forge
perl-test-differences     0.70            pl5321ha770c72_0    conda-forge
perl-test-exception       0.43            pl5321hd8ed1ab_0    conda-forge
perl-test-fatal           0.016           pl5321ha770c72_0    conda-forge
perl-test-most            0.38            pl5321hdfd78af_0    bioconda
perl-test-warn            0.37            pl5321hd8ed1ab_0    conda-forge
perl-test-warnings        0.031           pl5321ha770c72_0    conda-forge
perl-text-diff            1.45            pl5321hd8ed1ab_0    conda-forge
perl-time-local           1.35            pl5321hdfd78af_0    bioconda
perl-timedate             2.33            pl5321hdfd78af_2    bioconda
perl-tree-dag_node        1.32            pl5321hdfd78af_0    bioconda
perl-try-tiny             0.31            pl5321ha770c72_0    conda-forge
perl-types-serialiser     1.01            pl5321hdfd78af_0    bioconda
perl-uri                  5.17            pl5321ha770c72_0    conda-forge
perl-url-encode           0.03            pl5321h9ee0642_0    bioconda
perl-www-robotrules       6.02            pl5321hdfd78af_4    bioconda
perl-xml-dom              1.46            pl5321hdfd78af_1    bioconda
perl-xml-dom-xpath        0.14            pl5321hdfd78af_2    bioconda
perl-xml-parser           2.44_01         pl5321hc3e0081_1003    conda-forge
perl-xml-regexp           0.04            pl5321hdfd78af_3    bioconda
perl-xml-xpathengine      0.14            pl5321hdfd78af_3    bioconda
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pixman                    0.42.2               h59595ed_0    conda-forge
poa                       2.0                  h031d066_5    bioconda
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
python                    3.12.0          hab00c5b_0_cpython    conda-forge
readline                  8.2                  h8228510_1    conda-forge
samclip                   0.4.0                hdfd78af_1    bioconda
samtools                  1.17                 hd87286a_2    bioconda
seqtk                     1.4                  he4a0461_1    bioconda
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
snippy                    4.6.0                hdfd78af_2    bioconda
snp-sites                 2.5.1                he4a0461_4    bioconda
snpeff                    5.1                  hdfd78af_2    bioconda
t-coffee                  12.00.7fb08c2        h26a2512_0    bioconda
tabixpp                   1.1.2                hd68fcf3_1    bioconda
tk                        8.6.13               h2797004_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
vcflib                    1.0.1                hd2e4403_1    bioconda
viennarna                 2.4.7                    py27_2    bioconda
vt                        2015.11.10           he941832_3    bioconda
wget                      1.20.3               ha35d2d1_1    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xorg-fixesproto           5.0               h7f98852_1002    conda-forge
xorg-inputproto           2.3.2             h7f98852_1002    conda-forge
xorg-kbproto              1.0.7             h7f98852_1002    conda-forge
xorg-libice               1.1.1                hd590300_0    conda-forge
xorg-libsm                1.2.4                h7391055_0    conda-forge
xorg-libx11               1.8.6                h8ee46fc_0    conda-forge
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xorg-libxext              1.3.4                h0b41bf4_2    conda-forge
xorg-libxfixes            5.0.3             h7f98852_1004    conda-forge
xorg-libxi                1.7.10               h7f98852_0    conda-forge
xorg-libxrender           0.9.11               hd590300_0    conda-forge
xorg-libxt                1.3.0                hd590300_1    conda-forge
xorg-libxtst              1.2.3             h7f98852_1002    conda-forge
xorg-recordproto          1.14.2            h7f98852_1002    conda-forge
xorg-renderproto          0.11.1            h7f98852_1002    conda-forge
xorg-xextproto            7.3.0             h0b41bf4_1003    conda-forge
xorg-xproto               7.0.31            h7f98852_1007    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.5                hfc55251_0    conda-forge
RaverJay commented 1 year ago

I also suspect .gbff is not supported, but .gbk which it expects is deprecated by NCBI :<

Isoris commented 1 year ago

Yes it is very possible that it is caused with the input files.

Can you try using a .fna as a genomic assembly reference? See if it works.

Also can you do this command please:

snippy version

RaverJay commented 1 year ago

Yes it is very possible that it is caused with the input files.

Can you try using a .fna as a genomic assembly reference? See if it works.

That is a separate problem when using .gbff

As per my command above you can see I already do this, and run into the bcftools consensus error

Also can you do this command please:

snippy version

[15:51:34] This is snippy 4.6.0
[15:51:34] Written by Torsten Seemann
[15:51:34] Obtained from https://github.com/tseemann/snippy
[15:51:34] Detected operating system: linux
[15:51:34] Enabling bundled linux tools.
[15:51:34] Found bwa - /home/sebastian/mambaforge/envs/snippy/bin/bwa
[15:51:34] Found bcftools - /home/sebastian/mambaforge/envs/snippy/bin/bcftools
[15:51:34] Found samtools - /home/sebastian/mambaforge/envs/snippy/bin/samtools
[15:51:34] Found java - /home/sebastian/mambaforge/envs/snippy/bin/java
[15:51:34] Found snpEff - /home/sebastian/mambaforge/envs/snippy/bin/snpEff
[15:51:34] Found samclip - /home/sebastian/mambaforge/envs/snippy/bin/samclip
[15:51:34] Found seqtk - /home/sebastian/mambaforge/envs/snippy/bin/seqtk
[15:51:34] Found parallel - /home/sebastian/mambaforge/envs/snippy/bin/parallel
[15:51:34] Found freebayes - /home/sebastian/mambaforge/envs/snippy/bin/freebayes
[15:51:34] Found freebayes-parallel - /home/sebastian/mambaforge/envs/snippy/bin/freebayes-parallel
[15:51:34] Found fasta_generate_regions.py - /home/sebastian/mambaforge/envs/snippy/bin/fasta_generate_regions.py
[15:51:34] Found vcfstreamsort - /home/sebastian/mambaforge/envs/snippy/bin/vcfstreamsort
[15:51:34] Found vcfuniq - /home/sebastian/mambaforge/envs/snippy/bin/vcfuniq
[15:51:34] Found vcffirstheader - /home/sebastian/mambaforge/envs/snippy/bin/vcffirstheader
[15:51:34] Found gzip - /usr/bin/gzip
[15:51:34] Found vt - /home/sebastian/mambaforge/envs/snippy/bin/vt
[15:51:34] Found snippy-vcf_to_tab - /home/sebastian/mambaforge/envs/snippy/bin/snippy-vcf_to_tab
[15:51:34] Found snippy-vcf_report - /home/sebastian/mambaforge/envs/snippy/bin/snippy-vcf_report
[15:51:34] Checking version: samtools --version is >= 1.7 - ok, have 1.10
[15:51:34] Checking version: bcftools --version is >= 1.7 - ok, have 1.10
[15:51:34] Checking version: freebayes --version is >= 1.1 - ok, have 1.3.6
[15:51:34] Checking version: snpEff -version is >= 4.3 - ok, have 4.3
[15:51:34] Checking version: bwa is >= 0.7.12 - ok, have 0.7.17
[15:51:34] Please supply a reference FASTA/GBK/EMBL file with --reference
Isoris commented 1 year ago

What I suggest you to do is simply to download the snippy binary from GitHub

github.com/tseemann/snippy/releases

Download "source code (tar.gz)"

Then extract it and put the snippy folder in your home folder or anywhere.

then you add snippy executable to your path

For instance if you have the folder snippy-4.6.0 in your home.

You can do

export PATH=$HOME/snippy-4.6.0/bin:$PATH

You have to do it everytime you run a script in the console, or write it at the beginning of your script.

Or you can write this command directly in your ~/.profile

Then do source ~/.profile

The idea is that you already installed most dependencies on your conda environment. But you will change the path for snippy. And because snippy points to its own libraries within its own folder. By using the binary release you can have something that may work.

Isoris commented 1 year ago

Yes it is very possible that it is caused with the input files.

Can you try using a .fna as a genomic assembly reference? See if it works.

That is a separate problem when using .gbff

As per my command above you can see I already do this, and run into the bcftools consensus error

Also can you do this command please:

snippy version

[15:51:34] This is snippy 4.6.0
[15:51:34] Written by Torsten Seemann
[15:51:34] Obtained from https://github.com/tseemann/snippy
[15:51:34] Detected operating system: linux
[15:51:34] Enabling bundled linux tools.
[15:51:34] Found bwa - /home/sebastian/mambaforge/envs/snippy/bin/bwa
[15:51:34] Found bcftools - /home/sebastian/mambaforge/envs/snippy/bin/bcftools
[15:51:34] Found samtools - /home/sebastian/mambaforge/envs/snippy/bin/samtools
[15:51:34] Found java - /home/sebastian/mambaforge/envs/snippy/bin/java
[15:51:34] Found snpEff - /home/sebastian/mambaforge/envs/snippy/bin/snpEff
[15:51:34] Found samclip - /home/sebastian/mambaforge/envs/snippy/bin/samclip
[15:51:34] Found seqtk - /home/sebastian/mambaforge/envs/snippy/bin/seqtk
[15:51:34] Found parallel - /home/sebastian/mambaforge/envs/snippy/bin/parallel
[15:51:34] Found freebayes - /home/sebastian/mambaforge/envs/snippy/bin/freebayes
[15:51:34] Found freebayes-parallel - /home/sebastian/mambaforge/envs/snippy/bin/freebayes-parallel
[15:51:34] Found fasta_generate_regions.py - /home/sebastian/mambaforge/envs/snippy/bin/fasta_generate_regions.py
[15:51:34] Found vcfstreamsort - /home/sebastian/mambaforge/envs/snippy/bin/vcfstreamsort
[15:51:34] Found vcfuniq - /home/sebastian/mambaforge/envs/snippy/bin/vcfuniq
[15:51:34] Found vcffirstheader - /home/sebastian/mambaforge/envs/snippy/bin/vcffirstheader
[15:51:34] Found gzip - /usr/bin/gzip
[15:51:34] Found vt - /home/sebastian/mambaforge/envs/snippy/bin/vt
[15:51:34] Found snippy-vcf_to_tab - /home/sebastian/mambaforge/envs/snippy/bin/snippy-vcf_to_tab
[15:51:34] Found snippy-vcf_report - /home/sebastian/mambaforge/envs/snippy/bin/snippy-vcf_report
[15:51:34] Checking version: samtools --version is >= 1.7 - ok, have 1.10
[15:51:34] Checking version: bcftools --version is >= 1.7 - ok, have 1.10
[15:51:34] Checking version: freebayes --version is >= 1.1 - ok, have 1.3.6
[15:51:34] Checking version: snpEff -version is >= 4.3 - ok, have 4.3
[15:51:34] Checking version: bwa is >= 0.7.12 - ok, have 0.7.17
[15:51:34] Please supply a reference FASTA/GBK/EMBL file with --reference

Can you please write the following in your console:

which -a snippy which -a bcftools which -a samtools

RaverJay commented 1 year ago

Respectfully, that doesn't make sense at all.

That file is exactly the one I am running right now, it's just here: /home/sebastian/mambaforge/envs/snippy/bin/snippy

And it runs the same dependencies again, so nothing changed.

RaverJay commented 1 year ago

Could you just post the versions of the dependencies that you are using?

Isoris commented 1 year ago

If I was you I would try to change your genbank file into a fasta using any2fasta and see if it works. Then run snippy on it.

Here are my versions

https://github.com/tseemann/snippy Torsten Seemann (sntppy) quentin@quentin-B450-AORUS-ELITE:~$ snippy version

[22:41:24] This is snippy 4.6.0 [22:41:24] Written by Torsten Seemann

[22:41:24] Obtained from https://github.com/tseemann/sntppy

[22:41:24] Detected operating system: linux [22:41:24] Enabling bundled linux tools.

[22:41:24] Found bwa /home/quentin/snippy/binaries/linux/bwa

[22:41:24] Found beftools /home/quentin/snippy/binaries/linux/bcftools Found samtools- /home/quentin/snippy/binaries/linux/samtools

[22:41:24] Found java- /home/quentin/Downloads/openjdk-11 linux-x64_bin/jdk-11/bin/java

Found snpEff /home/quentin/snippy/binaries/noarch/snpEff Found samclip /home/quentin/snippy/binaries/noarch/samclip

[22:41:24] Found seqtk - /home/quentin/sntppy/binaries/linux/seqtk Found parallel- /home/quentin/snippy/binaries/noarch/parallel

Found freebayes- /home/quentin/snippy/binaries/linux/freebayes Found freebayes-parallel /home/quentin/snippy/binaries/noarch/freebayes-parallel

[22:41:24] Found fasta_generate_regtons.py /home/quentin/snippy/binaries/noarch/fasta_generate_regions.py ] Found vcfstreamsort /home/quentin/snippy/binaries/linux/vcfstreamsort

[22:41:24] Found vcfuntq /home/quentin/snippy/binaries/linux/vcfuntq

[22:41:24] Found vcffirstheader- /home/quentin/sntppy/binaries/noarch/vcfftrstheader

[22:41:24] Found gztp /usr/bin/gztp

[22:41:24]Found vt /home/quentin/sntppy/binaries/linux/vt

[22:41:24] Found sntppy-vcf_to_tab /home/quentin/snippy/bin/snippy-vcf_to_tab Found snippy-vcf_report /home/quentin/snippy/bin/snippy-vcf_report

[22:41:24] Checking verston: samtools --version ts >= 1.7 ok, have 1.10

[22:41:24] Checking version: bcftools --version ts >= 1.7 ok, have 1.10

[22:41:24] Checking version: freebayes-version is 1.1 ok, have 1.3.1

[22:41:25] Checking version: snpEff version is >= 4.3 ok, have 4.3

[22:41:25] Checking version: bwa ts > 0.7.12 ok, have 0.7.17

[22:41:25] Please supply a reference FASTA/GBK/EMBL file

RaverJay commented 1 year ago

Can you please write the following in your console:

which -a snippy

/home/sebastian/mambaforge/envs/snippy/bin/snippy

which -a bcftools

/home/sebastian/mambaforge/envs/snippy/bin/bcftools
/usr/bin/bcftools
/bin/bcftools
/usr/bin/bcftools
/bin/bcftools

which -a samtools

/home/sebastian/mambaforge/envs/snippy/bin/samtools
/usr/local/bin/samtools
/usr/bin/samtools
/bin/samtools
/usr/bin/samtools
/bin/samtools
Isoris commented 1 year ago

Ok and now do

which -a python which -a python3

which snippy which samtools which bcftools which python which python3

Isoris commented 1 year ago

Maybe your problem is related to this issue..

https://github.com/tseemann/snippy/issues/407

RaverJay commented 1 year ago

Maybe your problem is related to this issue..

407

The bcftools problem was related to this, my mappings had no RG tag, letting snippy map itself fixed it, thanks!

Doing that I also discovered that snippy will fail to run bwa commands when the read data paths have spaces in them, and symlinks don't help because it resolves the paths first...

I can work around that though, and all should work

Conclusion: instead of using conda/mamba, download the release as it has all the finicky dependencies packaged as binaries. Just add pretty much any samtools and bcftools and it should be good to go.

Isoris commented 1 year ago

Can you go into the folder where your vcf is and do something like bcftools view snps.vcf.gz

Also can you try to run snippy but add the option

-rgid "CM13"

Let me know what happens when you add this option to the snippy command

RaverJay commented 1 year ago

Can you go into the folder where your vcf is and do something like bcftools view snps.vcf.gz

Also can you try to run snippy but add the option

-rgid "CM13"

Let me know what happens when you add this option to the snippy command

I tried this, did not help - the tag has to be in the .bam files, which snippy adds during mapping

Isoris commented 1 year ago

Maybe your problem is related to this issue..

407

The bcftools problem was related to this, my mappings had no RG tag, letting snippy map itself fixed it, thanks!

Doing that I also discovered that snippy will fail to run bwa commands when the read data paths have spaces in them, and symlinks don't help because it resolves the paths first...

I can work around that though, and all should work

Conclusion: instead of using conda/mamba, download the release as it has all the finicky dependencies packaged as binaries. Just add pretty much any samtools and bcftools and it should be good to go.

Good job! It is a good practice to use underscores i will send you a link that may help in naming your files

https://github.com/josephryan/RyanLabUnixBestPractices

also sometimes it is possible to use the argument as a string between " " or you could do like this in other situations if your file is named "file with spaces in the name" you can add escape characters: file\ with\ space\ in\ the\ name

export PATH=$HOME/snippy-4.6.0/bin:$PATH snippy --outdir 'SIKU01' --R1 '/mnt/2TB/GENOME_ANALYSIS_QUENTIN_CEAAH_KASETSART/00-DATA_AND_PROGRAMS/10-MY_GENOMES/SRA_FASTQ_READS_DUMPS/SIKU01/SIKU01_R1_TRIM.fastq.gz' --R2 '/mnt/2TB/GENOME_ANALYSIS_QUENTIN_CEAAH_KASETSART/00-DATA_AND_PROGRAMS/10-MY_GENOMES/SRA_FASTQ_READS_DUMPS/SIKU01/SIKU01_R2_TRIM.fastq.gz' --ref SIKU01.fa --cpus 12

Yes using the binary release is the best way to have something stable in most cases. Have a great day.

ZhangZF1102 commented 4 months ago

i have exported the environment list to a recipe that should work in case you are still having this issue. Just rename the file from .txt to .yaml and it should be fine. snippy_that_works.txt

Thank you so much, I think I fix the problem using your method