search

nanoporetech / medaka

Sequence correction provided by ONT Research
https://nanoporetech.com
Other
410 stars 74 forks source link

Singularity Error from converted docker container (h5py: OSError: Unable to open file (file signature not found)) #398

Closed jguhlin closed 1 year ago

jguhlin commented 1 year ago

Built container using:

singularity build medaka_gpu.sif docker://ontresearch/medaka:latest

As docker is not allowed on most research instances. mini_align step works fine, but running the consensus step I get the following error.

❯ singularity exec --nv -B .:/data medaka_gpu.sif medaka consensus --model r103_hac_g507  /data/reads.bam /data/GCF_004027225.2_bStrHab1.2.pri_genomic.fna
[09:20:20 - Predict] Setting tensorflow inter/intra-op threads to 1/1.
[09:20:20 - Predict] Processing region(s): NC_044277.2:0-155644563 NC_044278.2:0-123023803 NC_044279.2:0-89082643 NC_046358.1:0-86460390 NC_044281.2:0-83240647 NC_044282.2:0-76431904 NC_044283.2:0-70131234 NC_0442
84.2:0-63488934 NC_044285.2:0-44431488 NC_046359.1:0-43764472 NC_046360.1:0-39187110 NC_044288.2:0-27433492 NC_044289.2:0-16476160 NW_022651054.1:0-12224398 NC_044290.2:0-12884122 NC_044291.2:0-12307978 NC_044292.
2:0-8419051 NC_044293.2:0-7441144 NC_044294.2:0-6734548 NC_044295.2:0-5789228 NC_044296.2:0-5645757 NC_044297.2:0-5492648 NC_044298.2:0-2907026 NC_044299.2:0-2785283 NC_044301.2:0-37425726 NC_044302.2:0-101869369
NW_022651055.1:0-556942 NW_022651056.1:0-378074 NW_022651057.1:0-211696 NW_022651058.1:0-33736 NW_022651059.1:0-357906 NW_022651060.1:0-45507 NW_022651061.1:0-20747 NW_022651062.1:0-60270 NW_022651063.1:0-24535 NW
_022651064.1:0-33252 NW_022651065.1:0-34148 NW_022651066.1:0-151271 NW_022651067.1:0-79880 NW_022651068.1:0-28315 NW_022651069.1:0-52918 NW_022651070.1:0-86693 NW_022651071.1:0-222505 NW_022651072.1:0-77094 NW_022
651073.1:0-33863 NW_022651074.1:0-21488 NW_022651075.1:0-24155 NW_022651076.1:0-48637 NW_022651077.1:0-46904 NW_022651078.1:0-114670 NW_022651079.1:0-31560 NW_022651080.1:0-56214 NW_022651081.1:0-38872 NW_02265108
2.1:0-90314 NW_022651083.1:0-253776 NW_022651084.1:0-24179 NW_022651085.1:0-70586 NW_022651086.1:0-36894 NW_022651087.1:0-76117 NW_022651088.1:0-47232 NW_022651089.1:0-49887 NW_022651090.1:0-57529 NW_022651091.1:0
-44814 NW_022651092.1:0-44271 NW_022651093.1:0-21581 NW_022651094.1:0-27098 NW_022651095.1:0-59057 NW_022651096.1:0-20789 NW_022651097.1:0-62886 NW_022651098.1:0-48461 NW_022651099.1:0-73844 NW_022651100.1:0-47485
 NW_022651101.1:0-22520 NW_022651102.1:0-54083 NW_022651103.1:0-35551 NW_022651104.1:0-151516 NW_022651105.1:0-684923 NW_022651106.1:0-348875 NW_022651107.1:0-32151 NW_022651108.1:0-286977 NW_022651109.1:0-229176
NW_022651110.1:0-784362 NW_022651111.1:0-234768 NW_022651112.1:0-460413 NW_022651113.1:0-35802 NW_022651114.1:0-49631 NW_022651115.1:0-164872 NW_022651116.1:0-241945 NW_022651117.1:0-28210
[09:20:20 - Predict] Using model: /usr/local/lib/python3.8/dist-packages/medaka/data/r103_hac_g507_model.tar.gz.
[09:20:20 - MdlStrTF] ModelStoreTF exception <class 'OSError'>
Traceback (most recent call last):
  File "/usr/local/bin/medaka", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/medaka/medaka.py", line 724, in main
    args.func(args)
  File "/usr/local/lib/python3.8/dist-packages/medaka/prediction.py", line 127, in predict
    model_store.copy_meta(args.output)
  File "/usr/local/lib/python3.8/dist-packages/medaka/datastore.py", line 181, in copy_meta
    with DataStore(hdf, 'a') as ds:
  File "/usr/local/lib/python3.8/dist-packages/medaka/datastore.py", line 208, in __init__
    self.fh = h5py.File(self.filename, self.mode)
  File "/usr/local/lib/python3.8/dist-packages/h5py/_hl/files.py", line 533, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/usr/local/lib/python3.8/dist-packages/h5py/_hl/files.py", line 238, in make_fid
    fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
OSError: Unable to open file (file signature not found)

The model definitely exists:

❯ singularity exec medaka_gpu.sif ls /usr/local/lib/python3.8/dist-packages/medaka/data/r103_hac_g507_model.tar.gz
/usr/local/lib/python3.8/dist-packages/medaka/data/r103_hac_g507_model.tar.gz
❯ singularity exec medaka_gpu.sif medaka tools list_models
Available: r103_fast_g507, r103_fast_snp_g507, r103_fast_variant_g507, r103_hac_g507, r103_hac_snp_g507, r103_hac_variant_g507, r103_min_high_g345, r103_min_high_g360, r103_prom_high_g360, r103_prom_snp_g3210, r10
3_prom_variant_g3210, r103_sup_g507, r103_sup_snp_g507, r103_sup_variant_g507, r1041_e82_260bps_fast_g632, r1041_e82_260bps_fast_variant_g632, r1041_e82_260bps_hac_g632, r1041_e82_260bps_hac_variant_g632, r1041_e8
2_260bps_sup_g632, r1041_e82_260bps_sup_variant_g632, r1041_e82_400bps_fast_g615, r1041_e82_400bps_fast_g632, r1041_e82_400bps_fast_variant_g615, r1041_e82_400bps_fast_variant_g632, r1041_e82_400bps_hac_g615, r104
1_e82_400bps_hac_g632, r1041_e82_400bps_hac_variant_g615, r1041_e82_400bps_hac_variant_g632, r1041_e82_400bps_sup_g615, r1041_e82_400bps_sup_variant_g615, r104_e81_fast_g5015, r104_e81_fast_variant_g5015, r104_e81
_hac_g5015, r104_e81_hac_variant_g5015, r104_e81_sup_g5015, r104_e81_sup_g610, r104_e81_sup_variant_g610, r10_min_high_g303, r10_min_high_g340, r941_e81_fast_g514, r941_e81_fast_variant_g514, r941_e81_hac_g514, r9
41_e81_hac_variant_g514, r941_e81_sup_g514, r941_e81_sup_variant_g514, r941_min_fast_g303, r941_min_fast_g507, r941_min_fast_snp_g507, r941_min_fast_variant_g507, r941_min_hac_g507, r941_min_hac_snp_g507, r941_min
_hac_variant_g507, r941_min_high_g303, r941_min_high_g330, r941_min_high_g340_rle, r941_min_high_g344, r941_min_high_g351, r941_min_high_g360, r941_min_sup_g507, r941_min_sup_snp_g507, r941_min_sup_variant_g507, r
941_prom_fast_g303, r941_prom_fast_g507, r941_prom_fast_snp_g507, r941_prom_fast_variant_g507, r941_prom_hac_g507, r941_prom_hac_snp_g507, r941_prom_hac_variant_g507, r941_prom_high_g303, r941_prom_high_g330, r941
_prom_high_g344, r941_prom_high_g360, r941_prom_high_g4011, r941_prom_snp_g303, r941_prom_snp_g322, r941_prom_snp_g360, r941_prom_sup_g507, r941_prom_sup_snp_g507, r941_prom_sup_variant_g507, r941_prom_variant_g30
3, r941_prom_variant_g322, r941_prom_variant_g360, r941_sup_plant_g610, r941_sup_plant_variant_g610
Default consensus:  r941_min_hac_g507
Default variant:  r941_min_hac_variant_g507

Installing via conda isn't an option (workshop), so a container seems to be the best option. Any ideas?

cjw85 commented 1 year ago

I'm confused, if you can use singularity presumably you can build your own native singularity image using either the PyPI or conda packages?

jguhlin commented 1 year ago

Not sure what there is to be confused about. Would much prefer to use the official container if that is an option, as it is a workshop.

jguhlin commented 1 year ago

Doesn't look like the conda version supports the GPU either:

https://github.com/nanoporetech/medaka/issues/165#issuecomment-634003709

jguhlin commented 1 year ago

We've been able to resolve the issue with a built-from-source module.

There is an error in the README, Medaka does not use TensorFlow 2.2, it uses 2.7. Which has different requirements for CUDA & related libs.

https://github.com/nanoporetech/medaka/blob/master/requirements.txt#L8

It would still be awesome to have a working docker to singularity pipeline working to save time (Conda is still solving the environment, going on 20 minutes now, even though it looks like it won't work with the GPU anyways, so won't work for us). I'll probably futz around with it a bit more, one org can use singularity but not conda, and will report if I get anywhere. Feel free to close this issue.