search

FRED-2 / OptiType

Precision HLA typing from next-generation sequencing data
BSD 3-Clause "New" or "Revised" License
185 stars 75 forks source link

OSError: [Errno 13] Permission denied: '/local' #71

Open ktroule opened 6 years ago

ktroule commented 6 years ago

Hi

I know somebody down here had the same problem.

I'm trying to use optotype in docker: docker run -v /local/pVACtools/:/data/ -t fred2/optitype --input SRR2672972_1.fastq SRR2672972_2.fastq --rna -o /local/pVACtools/Optitype/RNA_control

And I get this:

Traceback (most recent call last):
  File "/usr/local/bin/OptiType/OptiTypePipeline.py", line 235, in <module>
    os.makedirs(args.outdir)        
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 150, in makedirs
    makedirs(head, mode)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/local'

I assume that python 2.7is also in the docker. I've tried to run this as sudo and ik keeps showing the same errors.

Thanks

ju-mu commented 6 years ago

Same here...

b-schubert commented 6 years ago

the output path that you are specifying has to point to /data/

your path /local/pVACtools/ is the host system's path and is not known within Docker. What you are effectively trying to do is to write within the docker container to a virtual folder /local/pVACtools/. However, the only physical folder that has been mounted from your host system points to /data/ within the docker virtual environment.

ju-mu commented 6 years ago

Actually I did this, however the solution in my case was to give 777 permissions to the mounted, local folder. I guess this is all related to the lack of experience with docker, but the next issue I ran into was then

sudo docker run -v /local/directory:/data/ -t fred2/optitype -i x_1.fastq.gz x_2.fastq.gz -d -o /data/

[E::hts_open_format] Failed to open file /data/2018_03_20_15_06_37/2018_03_20_15_06_37_1.bam Traceback (most recent call last): File "/usr/local/bin/OptiType/OptiTypePipeline.py", line 299, in <module> pos, read_details = ht.pysam_to_hdf(bam_paths[0]) File "/usr/local/bin/OptiType/hlatyper.py", line 186, in pysam_to_hdf sam = pysam.AlignmentFile(samfile, sam_or_bam) File "pysam/libcalignmentfile.pyx", line 444, in pysam.libcalignmentfile.AlignmentFile.__cinit__ File "pysam/libcalignmentfile.pyx", line 621, in pysam.libcalignmentfile.AlignmentFile._open IOError: [Errno 2] could not open alignment file/data/2018_03_20_15_06_37/2018_03_20_15_06_37_1.bam: No such file or directory

Any idea?

jo6fu3 commented 6 years ago

I have the same problem...

[weichi@ngsteam Optitype_test]$ docker run -v /home/weichi/working:/Optitype_test/  fred2/optitype -i TVB019ND_sorted.fastq -d -v -o ./Optitype_test/

mapping with 1 threads...

 0:00:00.36 Mapping TVB019ND_sorted.fastq to GEN reference...

 0:00:01.06 Generating binary hit matrix.
[E::hts_open_format] Failed to open file ./Optitype_test/2018_05_04_14_18_47/2018_05_04_14_18_47_1.bam
Traceback (most recent call last):
  File "/usr/local/bin/OptiType/OptiTypePipeline.py", line 358, in <module>
    pos, read_details = ht.pysam_to_hdf(bam_paths[0])
  File "/usr/local/bin/OptiType/hlatyper.py", line 186, in pysam_to_hdf
    sam = pysam.AlignmentFile(samfile, sam_or_bam)
  File "pysam/libcalignmentfile.pyx", line 728, in pysam.libcalignmentfile.AlignmentFile.__cinit__
  File "pysam/libcalignmentfile.pyx", line 918, in pysam.libcalignmentfile.AlignmentFile._open
IOError: [Errno 2] could not open alignment file `./Optitype_test/2018_05_04_14_18_47/2018_05_04_14_18_47_1.bam`: No such file or directory
b-schubert commented 6 years ago

@ju-mu This might be related to gzipped fastq files? could you try to run dockerized optitype on an unzipped file?

@jo6fu3 you are using the wrong internal folder. /Optitype_test does not exist in the docker-VM and is not set as working directory. Please use the predefined /data folder

If you run optitype as follows you should not have any problems:

docker run -v /home/weichi/working:/data -d fred2/optitype -i TVB019ND_sorted.fastq -d -o /da

assuming that TVB019ND_sorted.fastq is un /home/weichi/working

jo6fu3 commented 6 years ago

@b-schubert Thanks for your help. But I got the other problem. I have no idea what happened. 

docker run -v /home/weichi/working/Optitype_test:/data/ fred2/optitype -i TVB019ND_sorted.fastq -d -v -o /data/
/data/seqan-src/include/seqan/basic/basic_exception.h:363 FAILED!  (Uncaught exception of type std::bad_alloc: std::bad_alloc)

stack trace:
  0          [0x8917fd]  /usr/local/bin/razers3()
  1    [0x7fe23b98c6b6]  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8d6b6)
  2    [0x7fe23b98c701]  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8d701)
  3    [0x7fe23b98c919]  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8d919)
  4    [0x7fe23b98cebc]  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8debc)
  5          [0x620458]  void seqan::AppendString_<seqan::Tag<seqan::TagGenerous_> >::append_<seqan::String<seqan::SimpleType<unsigned char, seqan::Dna5Q_>, seqan::Alloc<void> >, seqan::String<seqan::SimpleType<unsigned char, seqan::Dna5Q_>, seqan::Alloc<void> > const>(seqan::String<seqan::SimpleType<unsigned char, seqan::Dna5Q_>, seqan::Alloc<void> >&, seqan::String<seqan::SimpleType<unsigned char, seqan::Dna5Q_>, seqan::Alloc<void> > const&) + 0x88
  6          [0x77bbed]  bool seqan::loadReads<MyFragStoreConfig, seqan::FragmentStoreConfig<MyFragStoreConfig>, seqan::RazerSOptions<seqan::RazerSSpec<false, false> > >(seqan::FragmentStore<MyFragStoreConfig, seqan::FragmentStoreConfig<MyFragStoreConfig> >&, seqan::FormattedFile<seqan::Tag<seqan::TagFastq_>, seqan::Tag<seqan::Input_>, void>&, seqan::RazerSOptions<seqan::RazerSSpec<false, false> >&) + 0x31d
  7          [0x88ddd2]  int mapReads<seqan::RazerSSpec<false, false> >(seqan::StringSet<seqan::String<char, seqan::Alloc<void> >, seqan::Owner<seqan::Tag<seqan::Default_> > >&, seqan::StringSet<seqan::String<char, seqan::Alloc<void> >, seqan::Owner<seqan::Tag<seqan::Default_> > >&, seqan::RazerSOptions<seqan::RazerSSpec<false, false> >&) + 0x532
  8          [0x572e65]  main + 0x845
  9    [0x7fe23abf7830]  __libc_start_main + 0xf0
 10          [0x573d69]  _start + 0x29

Aborted (core dumped)

mapping with 1 threads...

 0:00:03.14 Mapping TVB019ND_sorted.fastq to GEN reference...

 0:11:33.03 Generating binary hit matrix.
[E::hts_open_format] Failed to open file /data/2018_05_07_02_54_47/2018_05_07_02_54_47_1.bam
Traceback (most recent call last):
  File "/usr/local/bin/OptiType/OptiTypePipeline.py", line 358, in <module>
    pos, read_details = ht.pysam_to_hdf(bam_paths[0])
  File "/usr/local/bin/OptiType/hlatyper.py", line 186, in pysam_to_hdf
    sam = pysam.AlignmentFile(samfile, sam_or_bam)
  File "pysam/libcalignmentfile.pyx", line 728, in pysam.libcalignmentfile.AlignmentFile.__cinit__
  File "pysam/libcalignmentfile.pyx", line 918, in pysam.libcalignmentfile.AlignmentFile._open
IOError: [Errno 2] could not open alignment file `/data/2018_05_07_02_54_47/2018_05_07_02_54_47_1.bam`: No such file or directory
jo6fu3 commented 6 years ago

On the other hand, I used BWA to map my fastq to hla_reference genome data/hla_reference_dna.fasta then generate the filtered fastq. But I got the same error message when used this filtered fastq.

b-schubert commented 6 years ago

Hmmm, how did you build the docker container? Did you use the pre-compiled docker container on docker-hub or did you compile it yourself?

jo6fu3 commented 6 years ago

I follow the " Installation via Docker " on this website (https://hub.docker.com/r/fred2/optitype/). docker pull fred2/optitype I think it's pre-compiled docker container on docker-hub.

b-schubert commented 6 years ago

Ohm okay. Your base operating system is Linux a presume? Have you checked you read&write access to your data folder? It has to be write accessible to everybody. Or if that is an security concern than you have to add the user biodocker to the owner group of the folder.

jo6fu3 commented 6 years ago

Yes, my operating system is CentOS Linux release 7.4.1708. I use chmod -R 777 data to change the read&write access, but got the same error message. drwxrwxrwx. 3 weichi sftp_group 33 May 7 23:12 data sudo docker run -v /home/weichi/working/Optitype_test:/data fred2/optitype -i TVB019ND_for_hla.fastq -d -v -o ./data

ps. I don't have the user named biodockerbut have docker in the owner group of the folder.

/data/seqan-src/include/seqan/basic/basic_exception.h:363 FAILED!  (Uncaught exception of type std::bad_alloc: std::bad_alloc)

stack trace:
  0          [0x8917fd]  /usr/local/bin/razers3()
  1    [0x7f24705506b6]  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8d6b6)
  2    [0x7f2470550701]  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8d701)
  3    [0x7f2470550919]  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8d919)
  4    [0x7f2470550ebc]  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8debc)
  5          [0x620458]  void seqan::AppendString_<seqan::Tag<seqan::TagGenerous_> >::append_<seqan::String<seqan::SimpleType<unsigned char, seqan::Dna5Q_>, seqan::Alloc<void> >, seqan::String<seqan::SimpleType<unsigned char, seqan::Dna5Q_>, seqan::Alloc<void> > const>(seqan::String<seqan::SimpleType<unsigned char, seqan::Dna5Q_>, seqan::Alloc<void> >&, seqan::String<seqan::SimpleType<unsigned char, seqan::Dna5Q_>, seqan::Alloc<void> > const&) + 0x88
  6          [0x77bbed]  bool seqan::loadReads<MyFragStoreConfig, seqan::FragmentStoreConfig<MyFragStoreConfig>, seqan::RazerSOptions<seqan::RazerSSpec<false, false> > >(seqan::FragmentStore<MyFragStoreConfig, seqan::FragmentStoreConfig<MyFragStoreConfig> >&, seqan::FormattedFile<seqan::Tag<seqan::TagFastq_>, seqan::Tag<seqan::Input_>, void>&, seqan::RazerSOptions<seqan::RazerSSpec<false, false> >&) + 0x31d
  7          [0x88ddd2]  int mapReads<seqan::RazerSSpec<false, false> >(seqan::StringSet<seqan::String<char, seqan::Alloc<void> >, seqan::Owner<seqan::Tag<seqan::Default_> > >&, seqan::StringSet<seqan::String<char, seqan::Alloc<void> >, seqan::Owner<seqan::Tag<seqan::Default_> > >&, seqan::RazerSOptions<seqan::RazerSSpec<false, false> >&) + 0x532
  8          [0x572e65]  main + 0x845
  9    [0x7f246f7bb830]  __libc_start_main + 0xf0
 10          [0x573d69]  _start + 0x29

Aborted (core dumped)

mapping with 1 threads...

 0:00:02.67 Mapping TVB019ND_for_hla.fastq to GEN reference...

 0:17:01.84 Generating binary hit matrix.
[E::hts_open_format] Failed to open file ./data/2018_05_07_15_43_23/2018_05_07_15_43_23_1.bam
Traceback (most recent call last):
  File "/usr/local/bin/OptiType/OptiTypePipeline.py", line 358, in <module>
    pos, read_details = ht.pysam_to_hdf(bam_paths[0])
  File "/usr/local/bin/OptiType/hlatyper.py", line 186, in pysam_to_hdf
    sam = pysam.AlignmentFile(samfile, sam_or_bam)
  File "pysam/libcalignmentfile.pyx", line 728, in pysam.libcalignmentfile.AlignmentFile.__cinit__
  File "pysam/libcalignmentfile.pyx", line 918, in pysam.libcalignmentfile.AlignmentFile._open
IOError: [Errno 2] could not open alignment file `./data/2018_05_07_15_43_23/2018_05_07_15_43_23_1.bam`: No such file or directory
b-schubert commented 6 years ago

-o ./data

has to be just /data

jo6fu3 commented 6 years ago

I tried this two command -o /data/ and -o /data, but got the same problem.

If I used -o ./data it would create a folder named data and put 2018_05_xx_xx_xx_xx folder under it. If I used -o /data it would not create the data folder, only create the 2018_05_xx_xx_xx_xx folder. But I always got the error message IOError: [Errno 2] could not open alignment file./data/2018_05_07_15_43_23/2018_05_07_15_43_23_1.bam: No such file or directory in the end.

And is it correct to got this message? /data/seqan-src/include/seqan/basic/basic_exception.h:363 FAILED! (Uncaught exception of type std::bad_alloc: std::bad_alloc)

Seluj78 commented 6 years ago

I am getting this error message on jenkins with blueocean:

https://imgur.com/JFGGkZh

https://pastebin.com/eumaEwVU

Any idea ?

racharyaUC commented 6 years ago

Mounting an additional volume for output files still gives this error. Example: docker run -v /local/folder:/data/ -v /another/local/folder:/output/ -t fred2/optitype -i forward.fastq reverse.fastq -d -o /output/ results in the error:

Traceback (most recent call last):
  File "/usr/local/bin/OptiType/OptiTypePipeline.py", line 275, in <module>
    os.makedirs(out_dir)
  File "/usr/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/output/2018_07_31_02_28_19'
tufanchina commented 5 years ago

I had the similar error message IOError: [Errno 2] could not open alignment file ./data/2019_01_23_23_10_57/2019_01_23_23_10_57_1.bam: No such file or directory

I found that it was because the reads file is too large, probably out of the setting for docker, I made a smaller subset fastq file out of the reads file and then it works great.

Hopefully it will help.

ghost commented 4 years ago

I had the same error message, I increased the memory in docker (docker whale --> preferences --> advanced) and it worked.

jingydz commented 4 years ago

Yes, my operating system is CentOS Linux release 7.4.1708. I use chmod -R 777 data to change the read&write access, but got the same error message. drwxrwxrwx. 3 weichi sftp_group 33 May 7 23:12 data sudo docker run -v /home/weichi/working/Optitype_test:/data fred2/optitype -i TVB019ND_for_hla.fastq -d -v -o ./data

ps. I don't have the user named biodockerbut have docker in the owner group of the folder.

/data/seqan-src/include/seqan/basic/basic_exception.h:363 FAILED!  (Uncaught exception of type std::bad_alloc: std::bad_alloc)

stack trace:
  0          [0x8917fd]  /usr/local/bin/razers3()
  1    [0x7f24705506b6]  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8d6b6)
  2    [0x7f2470550701]  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8d701)
  3    [0x7f2470550919]  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8d919)
  4    [0x7f2470550ebc]  /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8debc)
  5          [0x620458]  void seqan::AppendString_<seqan::Tag<seqan::TagGenerous_> >::append_<seqan::String<seqan::SimpleType<unsigned char, seqan::Dna5Q_>, seqan::Alloc<void> >, seqan::String<seqan::SimpleType<unsigned char, seqan::Dna5Q_>, seqan::Alloc<void> > const>(seqan::String<seqan::SimpleType<unsigned char, seqan::Dna5Q_>, seqan::Alloc<void> >&, seqan::String<seqan::SimpleType<unsigned char, seqan::Dna5Q_>, seqan::Alloc<void> > const&) + 0x88
  6          [0x77bbed]  bool seqan::loadReads<MyFragStoreConfig, seqan::FragmentStoreConfig<MyFragStoreConfig>, seqan::RazerSOptions<seqan::RazerSSpec<false, false> > >(seqan::FragmentStore<MyFragStoreConfig, seqan::FragmentStoreConfig<MyFragStoreConfig> >&, seqan::FormattedFile<seqan::Tag<seqan::TagFastq_>, seqan::Tag<seqan::Input_>, void>&, seqan::RazerSOptions<seqan::RazerSSpec<false, false> >&) + 0x31d
  7          [0x88ddd2]  int mapReads<seqan::RazerSSpec<false, false> >(seqan::StringSet<seqan::String<char, seqan::Alloc<void> >, seqan::Owner<seqan::Tag<seqan::Default_> > >&, seqan::StringSet<seqan::String<char, seqan::Alloc<void> >, seqan::Owner<seqan::Tag<seqan::Default_> > >&, seqan::RazerSOptions<seqan::RazerSSpec<false, false> >&) + 0x532
  8          [0x572e65]  main + 0x845
  9    [0x7f246f7bb830]  __libc_start_main + 0xf0
 10          [0x573d69]  _start + 0x29

Aborted (core dumped)

mapping with 1 threads...

 0:00:02.67 Mapping TVB019ND_for_hla.fastq to GEN reference...

 0:17:01.84 Generating binary hit matrix.
[E::hts_open_format] Failed to open file ./data/2018_05_07_15_43_23/2018_05_07_15_43_23_1.bam
Traceback (most recent call last):
  File "/usr/local/bin/OptiType/OptiTypePipeline.py", line 358, in <module>
    pos, read_details = ht.pysam_to_hdf(bam_paths[0])
  File "/usr/local/bin/OptiType/hlatyper.py", line 186, in pysam_to_hdf
    sam = pysam.AlignmentFile(samfile, sam_or_bam)
  File "pysam/libcalignmentfile.pyx", line 728, in pysam.libcalignmentfile.AlignmentFile.__cinit__
  File "pysam/libcalignmentfile.pyx", line 918, in pysam.libcalignmentfile.AlignmentFile._open
IOError: [Errno 2] could not open alignment file `./data/2018_05_07_15_43_23/2018_05_07_15_43_23_1.bam`: No such file or directory

Hi, I had the same error message. Can I ask if you have solved this problem? `/opt/conda/conda-bld/razers3_1592544302093/work/include/seqan/basic/basic_exception.h:363 FAILED! (Uncaught exception of type std::bad_alloc: std::bad_alloc)

stack trace: 0 [0x2ad856d790f4] razers3(+0x5cc0f4) 1 [0x2ad857605f69] /home/zhangjj/software/miniconda3/envs/optitype/bin/../lib/libstdc++.so.6(+0xacf69) 2 [0x2ad857605fab] /home/zhangjj/software/miniconda3/envs/optitype/bin/../lib/libstdc++.so.6(+0xacfab) 3 [0x2ad857606194] __cxarethrow + 0 4 [0x2ad85760656a] operator new(unsigned long) + 0x52 5 [0x2ad856a7351f] razers3(+0x2c651f) 6 [0x2ad856bd7876] bool seqan::loadReads<MyFragStoreConfig, seqan::FragmentStoreConfig, seqan::RazerSOptions<seqan::RazerSSpec<false, false> > >(seqan::FragmentStore<MyFragStoreConfig, seqan::FragmentStoreConfig >&, seqan::FormattedFile<seqan::Tag<seqan::TagFastq>, seqan::Tag, void>&, seqan::RazerSOptions<seqan::RazerSSpec<false, false> >&) + 0x2a6 7 [0x2ad856d78478] int mapReads<seqan::RazerSSpec<false, false> >(seqan::StringSet<seqan::String<char, seqan::Alloc >, seqan::Owner<seqan::Tag > >&, seqan::StringSet<seqan::String<char, seqan::Alloc >, seqan::Owner<seqan::Tag > >&, seqan::RazerSOptions<seqan::RazerSSpec<false, false> >&) + 0x6f8 8 [0x2ad856a212f9] main + 0x849 9 [0x2ad857bd0d5d] __libc_start_main + 0xfd 10 [0x2ad856a219d6] razers3(+0x2749d6)

Aborted (core dumped)`

Magisterh1 commented 2 years ago

I had the same permission errors (Ubuntu server). I did the following and so far optitype is running: sudo adduser biodocker sudo chown -R :users /work/Projects sudo chmod -R 777 /work/Projects sudo docker run -v /work/Projects:/data -t fred2/optitype -i EA005_001_1.fq EA005_001_2.fq -d -o /data

Update: Optitype finished without errors :)