search

biocorecrg / MOP2

Master of Pores 2
https://biocorecrg.github.io/MOP2/docs/
MIT License
23 stars 7 forks source link

Error executing process > 'compore_polish_flow:NANOPOLISH_EVENTALIGN:index (mod)' #37

Closed assetdaniyarov closed 1 year ago

assetdaniyarov commented 1 year ago

Good day, colleagues,

Have an error. 

ubuntu@ubuntu:~/MOP2/mop_mod$ /home/ubuntu/nextflow run /home/ubuntu/MOP2/mop_mod/mop_mod.nf -with-singularity -profile local
N E X T F L O W  ~  version 22.10.2
Launching `/home/ubuntu/MOP2/mop_mod/mop_mod.nf` [ecstatic_feynman] DSL2 - revision: 83320fa996

╔╦╗╔═╗╔═╗  ╔╦╗┌─┐┌┬┐
║║║║ ║╠═╝  ║║║│ │ ││
╩ ╩╚═╝╩    ╩ ╩└─┘─┴┘

====================================================
BIOCORE@CRG Master of Pores 2. Detection of RNA modification - N F  ~  version 2.0
====================================================

*****************   Input files    *******************
input_path                              : /home/ubuntu/MOP2/mop_preprocess/output
comparison                              : /home/ubuntu/MOP2/mop_mod/comparison.tsv

********** reference has to be the genome *************
reference                               : /home/ubuntu/MOP2/data/GRCh38_latest_rna.fna.gz
output                                  : /home/ubuntu/MOP2/mop_mod/output

pars_tools              : /home/ubuntu/MOP2/mop_mod/tools_opt.tsv

************************* Flows *******************************
epinano                                 : YES
nanocompore                             : YES
tombo_lsc                               : YES
tombo_msc                               : YES

email                                   : 

Skipping the email

executor >  local (1)
[-        ] process > checkRef                                            [  0%] 0 of 1
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
executor >  local (2)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [  0%] 0 of 1
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
executor >  local (2)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [  0%] 0 of 1
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
executor >  local (3)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [100%] 1 of 1 ✔
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
[-        ] process > epinano_flow:indexReference                         -
executor >  local (3)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [100%] 1 of 1 ✔
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
[-        ] process > epinano_flow:indexReference                         -
executor >  local (4)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [100%] 1 of 1 ✔
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
[-        ] process > epinano_flow:indexReference                         -
executor >  local (4)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [100%] 1 of 1 ✔
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
[-        ] process > epinano_flow:indexReference                         -
executor >  local (4)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [100%] 1 of 1 ✔
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
[-        ] process > epinano_flow:indexReference                         -
executor >  local (4)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [100%] 1 of 1 ✔
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
[-        ] process > epinano_flow:indexReference                         -
[-        ] process > epinano_flow:EPINANO_CALC_VAR_FREQUENCIES           -
[-        ] process > epinano_flow:joinEpinanoRes                         -
[-        ] process > epinano_flow:makeEpinanoPlots_ins                   -
[-        ] process > epinano_flow:makeEpinanoPlots_mis                   -
[-        ] process > epinano_flow:makeEpinanoPlots_del                   -
[1f/508eec] process > compore_polish_flow:getChromInfo (reference.fa)     [100%] 1 of 1 ✔
[-        ] process > compore_polish_flow:NANOPOLISH_EVENTALIGN:index     -
[-        ] process > compore_polish_flow:NANOPOLISH_EVENTALIGN:eventa... -
[-        ] process > compore_polish_flow:NANOPOLISH_EVENTALIGN:eventa... -
[-        ] process > compore_polish_flow:mean_per_pos                    -
[-        ] process > compore_polish_flow:concat_mean_per_pos             -
[-        ] process > compore_polish_flow:concat_csv_files                -
[-        ] process > compore_polish_flow:NANOCOMPORE_SAMPLE_COMPARE:s... -
[-        ] process > tombo_common_flow:multiToSingleFast5                -
[-        ] process > tombo_common_flow:TOMBO_RESQUIGGLE_RNA:resquiggl... -
[d7/1632d4] process > getChromInfo (reference.fa)                         [100%] 1 of 1 ✔
[-        ] process > tombo_msc_flow:TOMBO_GET_MODIFICATION_MSC:getMod... -
[-        ] process > bedGraphToWig_msc                                   -
[-        ] process > tombo_lsc_flow:TOMBO_GET_MODIFICATION_LSC:getMod... -
[-        ] process > bedGraphToWig_lsc                                   -
[-        ] process > wigToBigWig                                         -
[-        ] process > mergeTomboWigsPlus                                  -
[-        ] process > mergeTomboWigsMinus                                 -
[-        ] process > EPINANO_VER:getVersion                              -
[-        ] process > NANOPOLISH_VER:getVersion                           -
[29/5e20d5] process > NANOCOMPORE_VER:getVersion                          [100%] 1 of 1 ✔
[-        ] process > TOMBO_VER:getVersion                                -

Error executing process > 'compore_polish_flow:NANOPOLISH_EVENTALIGN:index (mod)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name biocorecrg-mopnanopolish-0.2.img.pulling.1669106082892 docker://biocorecrg/mopnanopolish:0.2 > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:1e74cf3ea8b17b29912c91cc0ea80b5d3aed0ffe25aa2c8d3d4b24c1454f7052
    Copying blob sha256:e9092151f7a2491c0d2b42f59c1b8e2bf859ac6e53321eba7bb1017b2b3fe43f
    Copying blob sha256:276898f99f7b162c4834f6e13978f98bfd5f377635cdc1416d1fe27d38b47820
    Copying blob sha256:a521508fa11e33da609b8f0480253ef148b68b8beb56c0e409d0d207fda5a363
    Copying blob sha256:61af738b4846d9ef2e9eea025fb44288da5b8c76e8289a4f9eb62a2beb58626e
    Copying blob sha256:e518f22d2fe88cb8e3a813783655e7615b90d288ebd5e683e43b14c7769f69de
    Copying blob sha256:a808f8bcd824acac4d82f9468a5b093eddc46a12468f99bab3c6191e9dd429ac
    Copying blob sha256:7f4f7d6e8dc6d3bc92ecb3f182a4177c46653fcf4a4e3eaecdebdebd6eebcf5d
    Copying blob sha256:3762fddd8cf345cbdd4413f62bf112b2a194c330803a1290e4582c26a357a2a6
    Copying blob sha256:f5624c74d9d1c71e06ad4ca643f473e837b6696d5790a1c8c6c34fdbd1c5a52d
    Copying blob sha256:047916eec6087fc7002822733b41d8b4f76e06cbfee36edee420f8e19730f858
    Copying blob sha256:b0c85be337e40513d2c7cdd5760f24f66aa9f4e5b092f8de6ee10fbdd8c2a3bf
executor >  local (4)
[ee/87a03a] process > checkRef (Checking GRCh38_latest_rna.fna.gz)        [100%] 1 of 1 ✔
[-        ] process > epinano_flow:splitReference                         -
[-        ] process > epinano_flow:splitBams                              -
[-        ] process > epinano_flow:indexReference                         -
[-        ] process > epinano_flow:EPINANO_CALC_VAR_FREQUENCIES           -
[-        ] process > epinano_flow:joinEpinanoRes                         -
[-        ] process > epinano_flow:makeEpinanoPlots_ins                   -
[-        ] process > epinano_flow:makeEpinanoPlots_mis                   -
[-        ] process > epinano_flow:makeEpinanoPlots_del                   -
[1f/508eec] process > compore_polish_flow:getChromInfo (reference.fa)     [100%] 1 of 1 ✔
[-        ] process > compore_polish_flow:NANOPOLISH_EVENTALIGN:index     -
[-        ] process > compore_polish_flow:NANOPOLISH_EVENTALIGN:eventa... -
[-        ] process > compore_polish_flow:NANOPOLISH_EVENTALIGN:eventa... -
[-        ] process > compore_polish_flow:mean_per_pos                    -
[-        ] process > compore_polish_flow:concat_mean_per_pos             -
[-        ] process > compore_polish_flow:concat_csv_files                -
[-        ] process > compore_polish_flow:NANOCOMPORE_SAMPLE_COMPARE:s... -
[-        ] process > tombo_common_flow:multiToSingleFast5                -
[-        ] process > tombo_common_flow:TOMBO_RESQUIGGLE_RNA:resquiggl... -
[d7/1632d4] process > getChromInfo (reference.fa)                         [100%] 1 of 1 ✔
[-        ] process > tombo_msc_flow:TOMBO_GET_MODIFICATION_MSC:getMod... -
[-        ] process > bedGraphToWig_msc                                   -
[-        ] process > tombo_lsc_flow:TOMBO_GET_MODIFICATION_LSC:getMod... -
[-        ] process > bedGraphToWig_lsc                                   -
[-        ] process > wigToBigWig                                         -
[-        ] process > mergeTomboWigsPlus                                  -
[-        ] process > mergeTomboWigsMinus                                 -
[-        ] process > EPINANO_VER:getVersion                              -
[-        ] process > NANOPOLISH_VER:getVersion                           -
[29/5e20d5] process > NANOCOMPORE_VER:getVersion                          [100%] 1 of 1 ✔
[-        ] process > TOMBO_VER:getVersion                                -
Error executing process > 'compore_polish_flow:NANOPOLISH_EVENTALIGN:index (mod)'

Caused by:
  Failed to pull singularity image
  command: singularity pull  --name biocorecrg-mopnanopolish-0.2.img.pulling.1669106082892 docker://biocorecrg/mopnanopolish:0.2 > /dev/null
  status : 255
  message:
    INFO:    Converting OCI blobs to SIF format
    INFO:    Starting build...
    Getting image source signatures
    Copying blob sha256:1e74cf3ea8b17b29912c91cc0ea80b5d3aed0ffe25aa2c8d3d4b24c1454f7052
    Copying blob sha256:e9092151f7a2491c0d2b42f59c1b8e2bf859ac6e53321eba7bb1017b2b3fe43f
    Copying blob sha256:276898f99f7b162c4834f6e13978f98bfd5f377635cdc1416d1fe27d38b47820
    Copying blob sha256:a521508fa11e33da609b8f0480253ef148b68b8beb56c0e409d0d207fda5a363
    Copying blob sha256:61af738b4846d9ef2e9eea025fb44288da5b8c76e8289a4f9eb62a2beb58626e
    Copying blob sha256:e518f22d2fe88cb8e3a813783655e7615b90d288ebd5e683e43b14c7769f69de
    Copying blob sha256:a808f8bcd824acac4d82f9468a5b093eddc46a12468f99bab3c6191e9dd429ac
    Copying blob sha256:7f4f7d6e8dc6d3bc92ecb3f182a4177c46653fcf4a4e3eaecdebdebd6eebcf5d
    Copying blob sha256:3762fddd8cf345cbdd4413f62bf112b2a194c330803a1290e4582c26a357a2a6
    Copying blob sha256:f5624c74d9d1c71e06ad4ca643f473e837b6696d5790a1c8c6c34fdbd1c5a52d
    Copying blob sha256:047916eec6087fc7002822733b41d8b4f76e06cbfee36edee420f8e19730f858
    Copying blob sha256:b0c85be337e40513d2c7cdd5760f24f66aa9f4e5b092f8de6ee10fbdd8c2a3bf
    Copying blob sha256:d5f76c9d11f315a83cab863e8042317dc05769b7bfa9603f7f09428fcd0b0978
    Copying blob sha256:7a9fddbdb3ca2e07ce37064c5ea6b42c0da570572c6c2580a153e2a6ced82b9d
    Copying blob sha256:07f993c72f4b1c62324d5d7ce799cef664e8b08c5df1e85c19f5dc4944db92c4
    Copying blob sha256:9385a7d7f89642d91499350880e6ab2a42bd31620bdce18cc307a10f8037475a
    Copying blob sha256:d766cc541a45a81ec4823b4522ee7f961b2d2ec98584774410ad992cade39fd9
    Copying blob sha256:ff4390cdd1a14ee619901d589ed9c24a2e79a58f42edb97ee27aa941357efa6d
    Copying blob sha256:dbd74f6531c6809f24ba5cf13f151dc94174a9848cbe1d2daacc3702d764a708
    Copying blob sha256:193e5a425e6baba6bb4d219a908e6e0e94fa4b69f2624446fb7240bc8adb38f6
    Copying blob sha256:a4d7254676fb442fa47db40c1b7e31bb0a953b3b52729a3b219916d06cbdd11e
    Copying blob sha256:682c399a9a404d74ee2fb669352db06c0583ba5b9a76f83e9bed0b590dbf7114
    Copying blob sha256:fa9f9b526b080e6f5c15fc671c80d3b7c3b84ddb57b4fd1db7f27d0368413d7d
    Copying blob sha256:7e98e8e2e69a1baea5a8e4df1fc657903304ed9cd8a8f84f98b24317bfe9c475
    Copying blob sha256:2f764843722a65951befd0232dc7c453142a11911667cce6e419ac3dff10fa26
    Copying blob sha256:23f4f9e84f759a26b8f6619ce5dfc8dc3bf1fd5409c7fb49e0aa69461a27483e
    Copying blob sha256:af7d7f43af91468f02bd674b094241b28dbd771b43e2ece3443910f4466f9a96
    Copying blob sha256:2ed7e2ef039b630733aa3cee0eaa454f5e98fa69d3c1d3f81fc0f57a4207030b
    Copying blob sha256:5cde6efba43bb788e3f92dca391ee09820293f0f3a906980a478e5db35468934
    Copying blob sha256:4a9b777eaf0f68990621a1bd66ff627211828315d1390632e70bc7511e247b1e
    Copying blob sha256:ecf6594c79b29b52d168274ad3abdb26d80e653abb7872cbebb9199807bd4c80
    Copying blob sha256:4ab6764dfb64dc73c156b165684a9ec8f06dc3034b08fc3a0b752fa911e55b67
    Copying blob sha256:75ae47fa2f8d70cc832e3dbb3bb63c62fde987cfd31b26cc657dda76948a9ac4
    Copying blob sha256:2c865baeaed15854e3093e5951773c1e9738c1bd40096c6132a4ff94e28950a9
    Copying blob sha256:9f302ca66cbf87680453ef9cb0f6c220730cf2473455dfce4e609947e095af82
    Copying blob sha256:70a97fbc76dd0537bcba4ae7b643a17d70fb7cc61daeaf5e98a13a754b2eb75f
    Copying blob sha256:ddf4e48a4ee60b47435494e1deeb8a333ed9ea4c8910b9fd4eed2c168617f46f
    Copying blob sha256:516e19bd299e96b54c7a1adf62f396ec543a19dfc09fb73383f52a277081eaf6
    Copying blob sha256:43cb3fe5c4aeae6df98c09fb0e43c8a92956f56349011d0d3d7063546a95221b
    Copying blob sha256:571b01b3ddffa820ebee3b1d97115356e039339699aabea91448c4c591070732
    Copying blob sha256:6f2eeb8410b2970ee873f72a1b0a935020363ef73bc490532725d5c539e5dfc9
    Copying config sha256:ec012d2e1e8779518bc0840db99d65a9b8ba084662ccbb9f457c09dbb6de27b2
    Writing manifest to image destination
    Storing signatures
    FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: conveyor failed to get: no descriptor found for reference "440b2c8927bbe2d4e64f19fdf5c304bad7fd7f9c70802cb25b07ec77d9243cf0"

ubuntu@ubuntu:~/MOP2/mop_mod$

params.config.test

params {
    input_path         = "/home/ubuntu/MOP2/mop_preprocess/output"
    comparison         = "$baseDir/comparison.tsv"

    reference           = "/home/ubuntu/MOP2/data/GRCh38_latest_rna.fna.gz"

    output             = "$baseDir/output"

    pars_tools         = "$baseDir/tools_opt.tsv"

    // flows 
    epinano       = "YES"
    nanocompore   = "YES"
    tombo_lsc     = "YES"
    tombo_msc     = "YES"

    // epinano plots
    epinano_plots = "YES"

    email              = ""
}

local.config

process {
    executor = 'local'
    cpus = 23
    memory = '30GB'    
    cache='lenient'
    container = 'biocorecrg/mopprepr:0.7'
    containerOptions = { workflow.containerEngine == "docker" ? '-u $(id -u):$(id -g) --platform amd64': null}
    withLabel: big_cpus_ignore {
        errorStrategy = 'ignore'

    }
    withLabel: basecall_gpus {
        maxForks = 1
        containerOptions = { workflow.containerEngine == "singularity" ? '--nv':
           ( workflow.containerEngine == "docker" ? '-u $(id -u):$(id -g) --gpus all': null ) } 
    }
}
AndreaYCT commented 1 year ago

Hi, I encountered similar error as your and solved by manually singularity pull image.

lucacozzuto commented 1 year ago

I think dockerhub has now a rate limit. https://docs.docker.com/docker-hub/download-rate-limit/#:~:text=Docker%20Hub%20limits%20the%20number,pulls%20per%206%20hour%20period. We are studying how to improve the things using Nextflow wave... https://www.nextflow.io/docs/latest/wave.html