search

deng-lab / viroprofiler

A containerized bioinformatics pipeline for viral metagenomic data analysis
https://deng-lab.github.io/viroprofiler
MIT License
23 stars 12 forks source link

OCI index found, but accept header does not support OCI indexes #4

Closed liuronghyw closed 1 year ago

liuronghyw commented 1 year ago

Description of the bug

Dear developers: I can't pull the viroprofilter in cluster with UNIX operate system, the command as follows:

export NXF_SINGULARITY_CACHEDIR="./temp"

nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r v0.2 --db "/public/home/**/db/viroprofiler" -without-conda

The error is as follows: image

Would you please give me some suggestions on how to solve this problem.

Command used and terminal output

nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r v0.2 --db "/public/home/hpc4011032/db/viroprofiler"

Relevant files

No response

System information

N E X T F L O W ~ version 22.10.6

HPC local Signalarity CentOS Linux

viroprofiler v0.2

rujinlong commented 1 year ago

Hi @liuronghyw, could you try the following command and then report the output?

singularity pull denglab-viroprofiler-base-v0.2.img docker://denglab/viroprofiler-base:v0.2
liuronghyw commented 1 year ago

Hi, Dr. long. I can pull the image with your command. Since I am not familiar with Nextflow. Would you please tell me how to bulid the pipeline with local image (denglan-viroprofilter-base-v0.2)？ Thank you very much! @rujinlong @deng-lab

rujinlong commented 1 year ago

I guess the original error occurred because I released the new version 10 hours ago, and you executed the Nextflow command simultaneously while the image was not yet successfully created. If you run the Nextflow command now, the error should not occur. Alternatively, you can pull images manually using the following commands, and then execute the Nextflow command:

singularity pull denglab-viroprofiler-base-v0.2.img docker://denglab/viroprofiler-base:v0.2
singularity pull denglab-viroprofiler-vibrant-v0.2.img docker://denglab/viroprofiler-vibrant:v0.2
singularity pull denglab-viroprofiler-binning-v0.2.img docker://denglab/viroprofiler-binning:v0.2
singularity pull denglab-viroprofiler-host-v0.2.img docker://denglab/viroprofiler-host:v0.2
singularity pull denglab-viroprofiler-geneannot-v0.2.img docker://denglab/viroprofiler-geneannot:v0.2
singularity pull denglab-viroprofiler-taxa-v0.2.img docker://denglab/viroprofiler-taxa:v0.2
singularity pull denglab-viroprofiler-abundance-v0.2.img docker://denglab/viroprofiler-abundance:v0.2
singularity pull denglab-viroprofiler-bracken-v0.2.img docker://denglab/viroprofiler-bracken:v0.2
singularity pull denglab-viroprofiler-replicyc-v0.2.img docker://denglab/viroprofiler-replicyc:v0.2
olgais93 commented 1 year ago

I'm having the same issue: 


Error executing process > 'VIROPROFILER:SETUP:DB_VREFSEQ'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name denglab-viroprofiler-base-v0.2.img.pulling.1681391469512 docker://denglab/viroprofiler-base:v0.2 > /dev/null
  status : 1
  message:
    WARNING: pull for Docker Hub is not guaranteed to produce the
    WARNING: same image on repeated pull. Use Singularity Registry
    WARNING: (shub://) to pull exactly equivalent images.
    ERROR MANIFEST_UNKNOWN: OCI index found, but accept header does not support OCI indexes
    ERROR: pulling container failed!
rujinlong commented 1 year ago

What is the version of singularity? Could you try the following command to update the pipeline,

nextflow pull deng-lab/viroprofiler

and then run with the -r main parameter,

nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main
olgais93 commented 1 year ago

Thank you! I did as your said and obtained this error: 

Error executing process > 'VIROPROFILER:SETUP:DB_CHECKV'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name denglab-viroprofiler-base.img.pulling.1681393187226 docker://denglab/viroprofiler-base > /dev/null
  status : 1
  message:
    WARNING: pull for Docker Hub is not guaranteed to produce the
    WARNING: same image on repeated pull. Use Singularity Registry
    WARNING: (shub://) to pull exactly equivalent images.
    ERROR MANIFEST_UNKNOWN: OCI index found, but accept header does not support OCI indexes
    ERROR: pulling container failed!
rujinlong commented 1 year ago

Could you please paste the first two lines of the output here? it should look like the following,

N E X T F L O W  ~  version 22.04.0
Launching `https://github.com/deng-lab/viroprofiler` [soggy_elion] DSL2 - revision: 471c32e0ba [main]
olgais93 commented 1 year ago

Sure!

$ nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main
N E X T F L O W  ~  version 22.10.6
Launching `https://github.com/deng-lab/viroprofiler` [hopeful_borg] DSL2 - revision: 471c32e0ba [main]
rujinlong commented 1 year ago

It's quite strange. The error seems to be caused by singularity. I cannot reproduce this error, but if you can provide additional information, it may help me identify the bug. Could you try as the following,

  1. Pull the image manually using this command.
  2. If it fails, check the singularity version and path using 
    singularity --version
which singularity

    and paste the output here.

olgais93 commented 1 year ago

The first thing failed and gave this error:

$ singularity pull denglab-viroprofiler-base-v0.2.img docker://denglab/viroprofiler-base:v0.2
ERROR: pull is only supported for shub URIs

Singularity version and path:

$ singularity --version
2.4.2-dist
$ which singularity
/home/olga/miniconda3/bin/singularity
rujinlong commented 1 year ago

Ok, this might be the reason. Your singularity version is quite old. Could you update to the newest version? The latest on conda is singularity v3.8.7 and singularityce v3.10.0. If possible, ask your sys-admin to install the newest singularity as root, as explained in another issue https://github.com/darcyabjones/pante/issues/4#issuecomment-546830722

olgais93 commented 1 year ago

I installed a newer version of Singularity. 

$ singularity --version
singularity-ce version 3.11.1-jammy

It is now giving a different error:

$ nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main

WorkflowError:
  Failed to solve the job scheduling problem with pulp. Please report a bug and use --scheduler greedy as a workaround:
  Pulp: Error while trying to execute, use msg=True for more detailscbc
  [2023-04-13 19:49 CRITICAL] Command 'snakemake --snakefile /opt/conda/envs/viroprofiler-virsorter2/lib/python3.10/site-packages/virsorter/rules/setup-retry.smk --directory /home/olga/viroprofiler/virsorter2 --quiet --config Skip_deps_install=False --jobs 1 --rerun-incomplete --latency-wait 600 --nolock  --use-conda --conda-prefix /home/olga/viroprofiler/virsorter2/conda_envs --conda-frontend mamba ' returned non-zero exit status 1.

Work dir:
  /home/olga/work/b9/26db2d6609386b7de7b34fa9ee8433

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

Thank you very much

rujinlong commented 1 year ago

It seems this is caused by snakemake in virsorter2 https://github.com/jiarong/VirSorter2/issues/51 (also see this). I updated the code, please try the following commands,

# update the pipeline
nextflow pull deng-lab/viroprofiler

# delete the old virsorter2 database (in your case, /home/olga/viroprofiler/virsorter2)
rm -rf /path/to/viroprofiler/db/virsorter2

# run the pipeline
nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main
olgais93 commented 1 year ago

Thank you, I did as you said and running nextflow run deng-lab/viroprofiler -profile singularity --mode "setup" -r main worked: 

executor >  local (3)
[05/27b9fd] process > VIROPROFILER:SETUP:DB_CHECKV     [100%] 1 of 1, cached: 1 ✔
[ec/7f85af] process > VIROPROFILER:SETUP:DB_VIBRANT    [100%] 1 of 1, cached: 1 ✔
[fe/c41c20] process > VIROPROFILER:SETUP:DB_VIRSORTER2 [100%] 1 of 1 ✔
[ef/8e9ed3] process > VIROPROFILER:SETUP:DB_DRAM       [100%] 1 of 1 ✔
[df/78f962] process > VIROPROFILER:SETUP:DB_IPHOP      [100%] 1 of 1 ✔
[cb/451b85] process > VIROPROFILER:SETUP:DB_VREFSEQ    [100%] 1 of 1, cached: 1 ✔
-[ViroProfiler] Pipeline completed successfully-

But then running the test failed: $ nextflow run deng-lab/viroprofiler -r main -profile singularity,test

This is the error I got: 

Error executing process > 'VIROPROFILER:VIRSORTER2'

Caused by:
  Process `VIROPROFILER:VIRSORTER2` terminated with an error exit status (1)

Command executed:

  virsorter config --set HMMSEARCH_THREADS=2
  virsorter run --seqname-suffix-off --viral-gene-enrich-off --prep-for-dramv -i putative_vcontigs_pref1.fasta -w out_vs2 --include-groups dsDNAphage --min-length 3000 --min-score 0.5 -j 2 --provirus-off -d /home/olga/viroprofiler/virsorter2 all
  grep '^>' out_vs2/final-viral-combined.fa | sed 's/>//' | sed 's/||.*//' > virus_virsorter2.list
  ln -s out_vs2/for-dramv/final-viral-combined-for-dramv.fa .
  ln -s out_vs2/for-dramv/viral-affi-contigs-for-dramv.tab .
  seqkit fx2tab -n final-viral-combined-for-dramv.fa | sed 's/-cat_/,/g' | csvtk add-header -n Contig,vs2_category > vs2_category.csv

  cat <<-END_VERSIONS > versions.yml
  "VIROPROFILER:VIRSORTER2":
executor >  local (29)
[db/8c0fa0] process > VIROPROFILER:FASTQC (UC30)               [100%] 15 of 15, cached: 15 ✔
[d3/91c5e0] process > VIROPROFILER:FASTP (UC30)                [100%] 15 of 15, cached: 15 ✔
[87/47afb0] process > VIROPROFILER:SPADES (HT24)               [100%] 15 of 15, cached: 14 ✔
[d7/86e681] process > VIROPROFILER:CONTIGLIB                   [100%] 1 of 1 ✔
[ce/a24c4f] process > VIROPROFILER:CHECKV                      [100%] 1 of 1 ✔
[3e/545e1b] process > VIROPROFILER:CONTIGLIB_CLUSTER           [100%] 1 of 1 ✔
[c4/607316] process > VIROPROFILER:GENEPRED4CTG                [100%] 1 of 1 ✔
[46/f2fbf7] process > VIROPROFILER:NRPROT                      [100%] 1 of 1 ✔
[98/6f1211] process > VIROPROFILER:NRGENE                      [100%] 1 of 1 ✔
[02/8154e6] process > VIROPROFILER:CONTIGINDEX                 [100%] 1 of 1 ✔
[6d/32e55f] process > VIROPROFILER:MAPPING2CONTIGS2 (HT24)     [100%] 15 of 15 ✔
[28/e6bfaf] process > VIROPROFILER:ABUNDANCE                   [100%] 1 of 1 ✔
[03/627fa9] process > VIROPROFILER:VIBRANT                     [100%] 1 of 1 ✔
[1c/6f9093] process > VIROPROFILER:DVF                         [100%] 1 of 1 ✔
[84/3d432f] process > VIROPROFILER:VIRCONTIGS_PRE              [100%] 1 of 1 ✔
[42/6be750] process > VIROPROFILER:VIRSORTER2                  [100%] 1 of 1, failed: 1 ✘
[-        ] process > VIROPROFILER:DRAMV                       -
[-        ] process > VIROPROFILER:TAXONOMY_VCONTACT           -
[-        ] process > VIROPROFILER:TAXONOMY_MMSEQS             -
[-        ] process > VIROPROFILER:TAXONOMY_MERGE              -
[f5/24a9a1] process > VIROPROFILER:VIRALHOST_IPHOP             [100%] 1 of 1, failed: 1 ✘
[-        ] process > VIROPROFILER:BACPHLIP                    -
[-        ] process > VIROPROFILER:RESULTS_TSE                 -
[-        ] process > VIROPROFILER:CUSTOM_DUMPSOFTWAREVERSIONS -
[-        ] process > VIROPROFILER:MULTIQC                     -
Pulling Singularity image docker://denglab/viroprofiler-taxa:v0.2 [cache /home/olga/work/singularity/denglab-viroprofiler-taxa-v0.2.img]
Pulling Singularity image docker://denglab/viroprofiler-replicyc:v0.2 [cache /home/olga/work/singularity/denglab-viroprofiler-replicyc-v0.2.img]
Execution cancelled -- Finishing pending tasks before exit

Any ideas?

rujinlong commented 1 year ago

There is a log file in work/42/6be750*/.command.log, could show me the content of that file?

olgais93 commented 1 year ago

In /home/olga/work/f5/24a9a1c9f030cba84ff25d0ee074cc/.command.log, there was this:

Pblm, I could not find directory /home/olga/viroprofiler/iphop/Sept_2021_pub. Maybe the host database directory was not specified with the --db_dir and is not the default "db/" directory ?

rujinlong commented 1 year ago

Your previous setup of the iphop database is not complete. The complete iphop database is ~200G. Please delete the iphop folder, and then run the setup step again, or you can use the --use_iphop false parameter to skip the host prediction step.