Nextflow Failing with Missing Output File

sharra44 commented 2 months ago

Describe the issue

Attempting to run vep with nextflow, running into output file error which terminates process.

System

VEP version: 111
VEP Cache version: 111
Perl version: [5.26.3
OS: Almalinux
tabix installed yes

Full VEP command line

nextflow run -resume -bg nextflow/workflows/run_vep.nf --vcf nextflow/testing.vcf.gz --skip_check 1 --vep_config /home/anno/ensembl-vep/nextflow/vep_config/vep.ini --output_prefix testing-output --outdir /home/anno/ensembl-vep

Full error message

ERROR ~ Error executing process > 'vep:runVEP (1)'

Caused by:
  Missing output file(s) `*.vcf.gz_summary.*` expected by process `vep:runVEP (1)`

Command executed:

  vep -i out.xaab.vcf.gz -o vep-testing-vep-out.xaab.vcf.gz --vcf --compress_output bgzip --format vcf --config vep.ini

  if [[ "tbi" == "tbi" ]]; then
    tabix -p vcf vep-testing-vep-out.xaab.vcf.gz
  else
    tabix -C -p vcf vep-testing-vep-out.xaab.vcf.gz
  fi

Command exit status:
  0

Data files (if applicable)

Input file is a small section of a vcf (around 20,000 lines). Running command from ensembl-vep folder. Output file does not seem to be generated

likhitha-surapaneni commented 2 months ago

Hi @sharra44 ,

Sorry to hear that you are facing issues with running VEP using Nextflow.

Is it possible to share the nextflow log file that got generated?
From the error, it seems like the expected intermediate files did not get generated. Is it possible to inspect the respective work directory to see if there is any .command.err file?

Thanks and regards, Likhitha

sharra44 commented 2 months ago

Here is the log file- nextflow.log Here is a screenshot of the work dir referenced for the failed run: When I tried to view .command.err, I got an empty file

sharra44 commented 2 months ago

Hi - Just wanted to follow up on this, as it is blocking our progress in implementing nextflow. Do you need anything further from me?

likhitha-surapaneni commented 2 months ago

Hi @sharra44,

Apologies for the delay. I was trying to run with the above options but was not able to reproduce. Can I request you to please try running with the test input and let us know if you are seeing the same issue?

Thanks and regards, Likhitha

sharra44 commented 2 months ago

Just tried - similar output nextflow.log

sharra44 commented 2 months ago

For reference - the command I used is nextflow run -resume -bg nextflow/workflows/run_vep.nf --vcf /home/anno/ensembl-vep/nextflow/examples/clinvar-testset/input.vcf.gz --skip_check 1 --vep_config /home/anno/ensembl-vep/nextflow/vep_config/vep.ini The vep.ini file looks like this: cache 1 dir_cache /root/.vep species 'homo_sapiens' format 'vcf' no_stats 1 force_overwrite 1 assembly 'GRCh38' af_gnomadg 1 symbol 1 variant_class 1 fasta /home/anno/fasta/Homo_sapiens_assembly38.fasta check_ref 1 merged 1 buffer_size 5000 offline 1 flag_pick 1 vcf 1

likhitha-surapaneni commented 2 months ago

Hi @sharra44 ,

Thank you for providing vep.ini configuration. Can you please try removing no_stats 1 and retry again? Currently Nextflow module expects statistics files as intermediate files.

Kind regards, Likhitha

sharra44 commented 2 months ago

This seemed to do the trick - thank you!!

likhitha-surapaneni commented 2 months ago

Thank you @sharra44, I am closing the ticket for now. Please feel free to open a new issue if there are any other problems.

Ensembl / ensembl-vep