MSG: ERROR: Could not write to output file

[PlatonB]

Elementary OS 5.1 Docker version 18.09.7, build 2d0083d ensembl-vep latest fcfda10

sudo docker run -t -i -v $HOME/vep_data:/opt/vep/.vep:Z ensemblorg/ensembl-vep ./vep --cache -e --vcf --force_overwrite --no_stats --dir_cache /opt/vep/.vep/ -i /opt/vep/.vep/input/vep_test_input.txt -o /opt/vep/.vep/output/vep_test_output.vcf

-------------------- EXCEPTION --------------------
MSG: ERROR: Could not write to output file /opt/vep/.vep/output/vep_test_output.vcf

STACK Bio::EnsEMBL::VEP::BaseRunner::get_output_file_handle /opt/vep/src/ensembl-vep/modules/Bio/EnsEMBL/VEP/BaseRunner.pm:226
STACK Bio::EnsEMBL::VEP::Runner::run /opt/vep/src/ensembl-vep/modules/Bio/EnsEMBL/VEP/Runner.pm:196
STACK toplevel ./vep:224
Date (localtime)    = Sun Dec  8 21:36:50 2019
Ensembl API version = 98
---------------------------------------------------

[aparton]

Hi,

Thank you for reporting this issue. This issue occurs when the VEP output is directed into a folder that doesn't exist. Making the directory '/opt/vep/.vep/output/' and rerunning your query should fix your issue.

I've had a look at our Docker documentation, and it could be a little clearer regarding the directory structure included within the docker image. I'll take a look at making some improvements to this.

Kind Regards, Andrew

[PlatonB]

There are _/home/platon/vepdata/input/ and _/home/platon/vepdata/output/ folders. As far as I understand, $HOME/vep_data:/opt/vep/.vep:Z should make _/home/platon/vepdata/output/ equivalent to /opt/vep/.vep/output/.

[aparton]

Hi,

Thanks for this. If /opt/vep/.vep/output exists, then I suspect it's a permissions issue that's preventing docker from writing to $HOME/vep_data. Could you please run:

_chmod a+rwx $HOME/vepdata/output

and try again?

Kind Regards, Andrew

[PlatonB]

Now it works. Thanks!

This is not critical, but I would like to inform you that the output file has a "locked" icon.

[aparton]

Hi,

Excellent, I'm glad that it's now working. Thanks for reporting the locked icon, we'll look into it.

I'm going to close this issue now. If you have any more questions, please feel free to reopen it or open a new one.

Kind Regards, Andrew

[lamnvcnsh]

Hi,

Thank you for reporting this issue. This issue occurs when the VEP output is directed into a folder that doesn't exist. Making the directory '/opt/vep/.vep/output/' and rerunning your query should fix your issue.

I've had a look at our Docker documentation, and it could be a little clearer regarding the directory structure included within the docker image. I'll take a look at making some improvements to this.

Kind Regards, Andrew

Dear Andrew,

I faced the same issue with VEP docker latest version (109_03);

As you instruction, I not quite fully understand how to solve this issue. Could you please provide me more information about that.

Docker version 23.0.3, build 3e7cbfd Ubuntu: 20.04 Command: docker run --rm -v /$HOME/vep_data:/data ensemblorg/ensembl-vep vep -i /data/1_S13.vcf --cache --offline -o /data/1_S13_out

==> MSG: ERROR: Could not write to stats file /data/1_S13_out_summary.html

Thank you very much.

Best regards,

[lamnvcnsh]

I can fix this problem with your intruction here: https://asia.ensembl.org/info/docs/tools/vep/script/vep_download.html (Read/Write access from the container);

The first option is not working for me, but the second option by changing permisison of the mount folder is working.

[nuno-agostinho]

Hi @lamnvcnsh!

Glad that fix worked for you. Is everything working as expected after changing the permissions of the mount folder?

Thanks, Nuno

[lamnvcnsh]

Hi @nuno-agostinho

The annotation process is working well now. I am reviewing the annotation result. If having any other issues, I'll let you know. Thank you very much.

Best wishes, Lam Nguyen

[mikal16]

Hi there,

I'm a bit new to working with docker and I am having a similar issue setting up the docker command.

I created a folder call vep_data in my home directory like it shows in this tutorial (https://useast.ensembl.org/info/docs/tools/vep/script/vep_download.html#docker) and placed my vcf, gff and fasta files in there and followed the steps to process the gff and fasta files for this analysis.

This is the command I end up with: docker run -t -i -v $HOME/vep_data:/data:Z ensemblorg/ensembl-vep ./vep -i $HOME/vep_data/out_filtered.mod.vcf -o $HOME/vep_data/output.txt --gff $HOME/vep_data/Sars_cov_2.ASM985889v3.101.gtf.gz -fasta $HOME/vep_data/Sars_cov_2.ASM985889v3.dna.toplevel.fa.gz

However, I'm not really sure here what data is referring to in this command and I get this error when I run this:

When I try running a similar command, but not using docker, I then get this error:

Any help with where I am going wrong here would be really appreciated.

Thanks a bunch in advance,

Mikal

[nuno-agostinho]

Hey Mikal,

Hope you are doing well!

Heare are some issues with your commands:

• ./vep is not available in the working folder of Docker. Simply replace that command with vep.
• The paths to input data are pointing to folders in your host machine that are not available in Docker. You don't need to give the full path when using VEP Docker, as you are already mapping $HOME/vep_data to the working directory in VEP Docker (/data).
• The options -t -i only make sense if you want to run vep in interactive mode. If you simply want vep to run as a command (as I think that is your intention), you should avoid to use these options.
• VEP requires your GTF to have a Tabix-indexed file, so you need to run tabix Sars_cov_2.ASM985889v3.101.gtf.gz before running VEP. More information in VEP Custom annotations.

Can you try first running Tabix on the GTF and then running this command with the fixes mentioned above?

docker run -v $HOME/vep_data:/data:Z ensemblorg/ensembl-vep \
  vep -i out_filtered.mod.vcf \
      -o output.txt \
      --gff Sars_cov_2.ASM985889v3.101.gtf.gz \
      --fasta Sars_cov_2.ASM985889v3.dna.toplevel.fa.gz

Kind regards, Nuno

[mikal16]

Hi Nuno,

Thank you so much for the quick reply! So, I ran this modified code and it seems I still have the "Couldn't find index for file Sars_cov_2.ASM985889v3.101.gtf.gz at /usr/local/lib/x86_64-linux-gnu/perl/5.26.1/Bio/DB/HTS/Tabix.pm line 53." error. I then had the following additional error messages below; I'm thinking maybe these are things I need to be installing into docker, so I'm going to look into that.

I am trying to run the variant effect predictor with my custom genome (SARS-CoV-2) and I've attached the gtf and fasta files here in case that helps.

Any insight into this would be helpful, and thanks so much again for your help and breaking everything down so nicely!

Mikal

[mikal16]

Hi there,

Sorry for the double message, but I troubleshooted some more and actually found that when I was running tabix, I was getting this error: “tabix [E::hts_idx_push] Unsorted positions on sequence #1: 21553 followed by 266”.

I looked into how to fix this issue and came across this fix in a GitHub post https://github.com/samtools/htslib/pull/1457 . So, I fixed the issue by running the following commands to sort and run tabix again.

After this, I ran the vep line again in docker and that fixes the “Couldn't find index for file Sars_cov_2.ASM985889v3.101.gtf.gz at /usr/local/lib/x86_64-linux-gnu/perl/5.26.1/Bio/DB/HTS/Tabix.pm line 53.” error. However, I still get the other errors along with a warning about the start and stop codons being skipped.

Thanks again and let me know if there’s anything else I could provide that would be of use.

Kindest regards,

Mikal

[nuno-agostinho]

Hi @mikal16,

You can ignore the following warnings:

Unknown option: no_update
Ignoring unsupported option 'no_update' found via ENV variable or INI file
Unknown option: no htslib
Ignoring unsupported option 'no_htslib' found via ENV variable or INI file
[Unknown option: no_plugins
Ignoring unsupported option
'no_plugins' found via ENV variable or INI file
[Unknown option: pluginsdir
Ignoring unsupported option 'pluginsdir' found via ENV variable or INI file

We made VEP more verbose than needed for the Docker image. Those warnings will be suppressed by default in the next release of VEP.

Regarding the others:

WARNING: The feature_type start_codon is being skipped
WARNING: The feature_type stop_codon is being skipped

This is because VEP does not support these features from the GTF, but will still parse any other features available. This may be a problem or not, depending on your context. Is your GTF mostly made of feature_type start_codon and stop_codon features?

Regardless of the warnings, you should have an output file. Is the output file missing or wrong?

Thanks!

Cheers, Nuno

[mikal16]

Hi Nuno,

Ahh okay that makes a lot of sense! And yes, the output works, thank you!! I'm sorry, but I have one very last question, but when I ran this analysis using the web interface my output file looked slightly different then what I'm getting now (shown below).

I am pretty sure this is because of the annotation I'm using, and so I was wondering if it be possible with this command to modify it so that I use the same annotation file as the one used by the web interface. I'm pretty sure that perhaps the fasta file would have to remain the same, but I think this could solve my issue, because I really need the gene and amino acid column information.

Thanks very much again.

Kindest regards,

Mikal

[mikal16]

Actually, I think I got this to work, I simply replaced --gtf by --custom and that fixed this. Can't say thank you enough for all the help!

Have a nice day,

Mikal

[nuno-agostinho]

Hey @mikal16,

I am glad that you made it work and I hope that you enjoy using VEP. 😊

If you have any other questions, feel free to open a new issue. Have a great day!

Cheers, Nuno