epi2me-labs / wf-human-variation

Other
86 stars 41 forks source link

nf-core failure to attain singularity containers #168

Closed bcardus closed 1 month ago

bcardus commented 2 months ago

Operating System

CentOS 7

Other Linux

CentOS Linux 7 (Core)

Workflow Version

v2.1.0

Workflow Execution

Command line (Cluster)

Other workflow execution

No response

EPI2ME Version

No response

CLI command run

[hzs682@rescomp2 hzs682]$ nf-core download epi2me-labs/wf-human-variation --force

? Select release / branch: v2.1.0 [release] ? Include the nf-core's default institutional configuration files into the download? Yes

In addition to the pipeline code, this tool can download software containers. ? Download software container images: singularity

If you are working on the same system where you will run Nextflow, you can amend the downloaded images to the ones in the$NXF_SINGULARITY_CACHEDIR folder, Nextflow will automatically find them.However if you will transfer the downloaded files to a different system then they should be copied to the target folder. ? Copy singularity images from $NXF_SINGULARITY_CACHEDIR to the target folder or amend new images to the cache? co py

If transferring the downloaded files to another system, it can be convenient to have everything compressed in a single file. This is not recommended when downloading Singularity images, as it can take a long time and saves very little space. ? Choose compression type: none

Workflow Execution - CLI Execution Profile

None

What happened?

I am trying to use nf-core to download the wf-human-variation workflow onto a cluster entry node so that I can run it on a compute node which does not have internet access. I input the above code and then run through the options below it to select the version, container images etc. However, when it tries to pull the singularity images, it fails and crashes. It seems that the command tries to look at an address in quay.io for the singularity image but can't find it, and when I go to that address on my browser there is nothing there either. When I run the individual command to pull the singularity image, as suggested in the error report, I get another error saying it failed to get the checksum and I am not authorised to access it. I did find docker images in different locations when searching manually e.g. at a different quay.io address and at a Docker hub address, but I don't know how to change this in the command. I am also not sure why it is pulling Docker images when I have asked for Singularity images.

Relevant log output

[hzs682@rescomp2 hzs682]$ nf-core download epi2me-labs/wf-human-variation --force

                                          ,--./,-.
          ___     __   __   __   ___     /,-._.--~\
    |\ | |__  __ /  ` /  \ |__) |__         }  {
    | \| |       \__, \__/ |  \ |___     \`-._,-`-,
                                          `._,._,'

    nf-core/tools version 2.10 - https://nf-co.re
    There is a new version of nf-core/tools available! (2.13.1)

? Select release / branch: v2.1.0  [release]
? Include the nf-core's default institutional configuration files into the download? Yes

In addition to the pipeline code, this tool can download software containers.
? Download software container images: singularity

If you are working on the same system where you will run Nextflow, you can amend the downloaded images to the ones
in the$NXF_SINGULARITY_CACHEDIR folder, Nextflow will automatically find them.However if you will transfer the 
downloaded files to a different system then they should be copied to the target folder.
? Copy singularity images from $NXF_SINGULARITY_CACHEDIR to the target folder or amend new images to the cache? co
py

If transferring the downloaded files to another system, it can be convenient to have everything compressed in a 
single file.
This is not recommended when downloading Singularity images, as it can take a long time and saves very little 
space.
? Choose compression type: none
WARNING  Deleting existing output directory: 'epi2me-labs-wf-human-variation_v2.1.0'                              
INFO     Saving 'epi2me-labs/wf-human-variation'                                                                  
          Pipeline revision: 'v2.1.0'                                                                             
          Use containers: 'singularity'                                                                           
          Container library: 'quay.io'                                                                            
          Using $NXF_SINGULARITY_CACHEDIR': /well/thompson/users/hzs682/singularity-cashdir'                      
          Output directory: 'epi2me-labs-wf-human-variation_v2.1.0'                                               
          Include default institutional configuration: 'True'                                                     
INFO     Downloading centralised configs from GitHub                                                              
INFO     Downloading workflow files from GitHub                                                                   
INFO     Processing workflow revision v2.1.0, found 9 container images in total.                                  
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% • 0/9 completed
ontresearch/modkit:shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ERROR    "ontresearch/modkit:shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb" failed for unclear reasons.             
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% • 0/9 completedINFO     Pulling of "ontresearch/modkit:shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb" failed.                      
         Please troubleshoot the command                                                                          
         "singularity pull --name                                                                                 
         /well/thompson/users/hzs682/singularity-cashdir/ontresearch-modkit-shaeedb131a939d3eea2f9bd4dbecec805c0fa
         20bdb.img docker://quay.io/ontresearch/modkit:shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb" manually.     

ERROR    "ontresearch/modkit:shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb" failed for unclear reasons.             

ERROR    Pulling of "ontresearch/modkit:shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb" failed.                      
         Please troubleshoot the command                                                                          
         "singularity pull --name                                                                                 
         /well/thompson/users/hzs682/singularity-cashdir/ontresearch-modkit-shaeedb131a939d3eea2f9bd4dbecec805c0fa
         20bdb.img docker://quay.io/ontresearch/modkit:shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb" manually.     

Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   0% • 0/9 completed
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/nf_core/dow │
│ nload.py:1066 in get_singularity_images                                                          │
│                                                                                                  │
│   1063 │   │   │   │   │   │   # Iteration happens over a copy of self.container_library[:], as  │
│   1064 │   │   │   │   │   │   for library in self.container_library[:]:                         │
│   1065 │   │   │   │   │   │   │   try:                                                          │
│ ❱ 1066 │   │   │   │   │   │   │   │   self.singularity_pull_image(*container, library, progres  │
│   1067 │   │   │   │   │   │   │   │   # Pulling the image was successful, no ContainerError wa  │
│   1068 │   │   │   │   │   │   │   │   break                                                     │
│   1069 │   │   │   │   │   │   │   except ContainerError.ImageExists as e:                       │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/nf_core/dow │
│ nload.py:1285 in singularity_pull_image                                                          │
│                                                                                                  │
│   1282 │   │   │   # something went wrong with the container retrieval                           │
│   1283 │   │   │   if any("FATAL: " in line for line in lines):                                  │
│   1284 │   │   │   │   progress.remove_task(task)                                                │
│ ❱ 1285 │   │   │   │   raise ContainerError(                                                     │
│   1286 │   │   │   │   │   container=container,                                                  │
│   1287 │   │   │   │   │   registry=library,                                                     │
│   1288 │   │   │   │   │   address=address,                                                      │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/nf_core/dow │
│ nload.py:1616 in __init__                                                                        │
│                                                                                                  │
│   1613 │   │   log.debug(f'Failed command:\n{" ".join(singularity_command)}')                    │
│   1614 │   │   log.debug(f'Singularity error messages:\n{"".join(error_msg)}')                   │
│   1615 │   │                                                                                     │
│ ❱ 1616 │   │   raise self.error_type                                                             │
│   1617 │                                                                                         │
│   1618 │   class RegistryNotFound(ConnectionRefusedError):                                       │
│   1619 │   │   """The specified registry does not resolve to a valid IP address"""               │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
OtherError: ('[bold red]"ontresearch/modkit:shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb" failed for unclear 
reasons.[/]\n', 'Pulling of "ontresearch/modkit:shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb" failed.\nPlease 
troubleshoot the command \n"singularity pull --name 
/well/thompson/users/hzs682/singularity-cashdir/ontresearch-modkit-shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb.img
docker://quay.io/ontresearch/modkit:shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb" manually.\n', ContainerError())

During handling of the above exception, another exception occurred:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/bin/nf-core:8 in <module>                │
│                                                                                                  │
│   5 from nf_core.__main__ import run_nf_core                                                     │
│   6 if __name__ == '__main__':                                                                   │
│   7 │   sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])                         │
│ ❱ 8 │   sys.exit(run_nf_core())                                                                  │
│   9                                                                                              │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/nf_core/__m │
│ ain__.py:111 in run_nf_core                                                                      │
│                                                                                                  │
│    108 │   │   │   log.debug(f"Could not check latest version: {e}")                             │
│    109 │   │   stderr.print("\n")                                                                │
│    110 │   # Launch the click cli                                                                │
│ ❱  111 │   nf_core_cli(auto_envvar_prefix="NFCORE")                                              │
│    112                                                                                           │
│    113                                                                                           │
│    114 @click.group(context_settings=dict(help_option_names=["-h", "--help"]))                   │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/Python/3.10.8-GCCcore-12.2.0/lib/python3.10/site-packages/click/ │
│ core.py:1130 in __call__                                                                         │
│                                                                                                  │
│   1127 │                                                                                         │
│   1128 │   def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any:                           │
│   1129 │   │   """Alias for :meth:`main`."""                                                     │
│ ❱ 1130 │   │   return self.main(*args, **kwargs)                                                 │
│   1131                                                                                           │
│   1132                                                                                           │
│   1133 class Command(BaseCommand):                                                               │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/rich_click/ │
│ rich_command.py:126 in main                                                                      │
│                                                                                                  │
│   123 │   │   try:                                                                               │
│   124 │   │   │   try:                                                                           │
│   125 │   │   │   │   with self.make_context(prog_name, args, **extra) as ctx:                   │
│ ❱ 126 │   │   │   │   │   rv = self.invoke(ctx)                                                  │
│   127 │   │   │   │   │   if not standalone_mode:                                                │
│   128 │   │   │   │   │   │   return rv                                                          │
│   129 │   │   │   │   │   # it's not safe to `ctx.exit(rv)` here!                                │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/Python/3.10.8-GCCcore-12.2.0/lib/python3.10/site-packages/click/ │
│ core.py:1657 in invoke                                                                           │
│                                                                                                  │
│   1654 │   │   │   │   super().invoke(ctx)                                                       │
│   1655 │   │   │   │   sub_ctx = cmd.make_context(cmd_name, args, parent=ctx)                    │
│   1656 │   │   │   │   with sub_ctx:                                                             │
│ ❱ 1657 │   │   │   │   │   return _process_result(sub_ctx.command.invoke(sub_ctx))               │
│   1658 │   │                                                                                     │
│   1659 │   │   # In chain mode we create the contexts step by step, but after the                │
│   1660 │   │   # base command has been invoked.  Because at that point we do not                 │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/Python/3.10.8-GCCcore-12.2.0/lib/python3.10/site-packages/click/ │
│ core.py:1404 in invoke                                                                           │
│                                                                                                  │
│   1401 │   │   │   echo(style(message, fg="red"), err=True)                                      │
│   1402 │   │                                                                                     │
│   1403 │   │   if self.callback is not None:                                                     │
│ ❱ 1404 │   │   │   return ctx.invoke(self.callback, **ctx.params)                                │
│   1405 │                                                                                         │
│   1406 │   def shell_complete(self, ctx: Context, incomplete: str) -> t.List["CompletionItem"]:  │
│   1407 │   │   """Return a list of completions for the incomplete value. Looks                   │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/Python/3.10.8-GCCcore-12.2.0/lib/python3.10/site-packages/click/ │
│ core.py:760 in invoke                                                                            │
│                                                                                                  │
│    757 │   │                                                                                     │
│    758 │   │   with augment_usage_errors(__self):                                                │
│    759 │   │   │   with ctx:                                                                     │
│ ❱  760 │   │   │   │   return __callback(*args, **kwargs)                                        │
│    761 │                                                                                         │
│    762 │   def forward(                                                                          │
│    763 │   │   __self, __cmd: "Command", *args: t.Any, **kwargs: t.Any  # noqa: B902             │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/nf_core/__m │
│ ain__.py:344 in download                                                                         │
│                                                                                                  │
│    341 │   │   container_cache_index,                                                            │
│    342 │   │   parallel_downloads,                                                               │
│    343 │   )                                                                                     │
│ ❱  344 │   dl.download_workflow()                                                                │
│    345                                                                                           │
│    346                                                                                           │
│    347 # nf-core licences                                                                        │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/nf_core/dow │
│ nload.py:234 in download_workflow                                                                │
│                                                                                                  │
│    231 │   │   if self.tower:                                                                    │
│    232 │   │   │   self.download_workflow_tower()                                                │
│    233 │   │   else:                                                                             │
│ ❱  234 │   │   │   self.download_workflow_static()                                               │
│    235 │                                                                                         │
│    236 │   def download_workflow_static(self):                                                   │
│    237 │   │   """Downloads a nf-core workflow from GitHub to the local file system in a self-c  │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/nf_core/dow │
│ nload.py:261 in download_workflow_static                                                         │
│                                                                                                  │
│    258 │   │   │   │   self.find_container_images(os.path.join(self.outdir, revision_dirname))   │
│    259 │   │   │   │                                                                             │
│    260 │   │   │   │   try:                                                                      │
│ ❱  261 │   │   │   │   │   self.get_singularity_images(current_revision=item[0])                 │
│    262 │   │   │   │   except OSError as e:                                                      │
│    263 │   │   │   │   │   raise DownloadError(f"[red]{e}[/]") from e                            │
│    264                                                                                           │
│                                                                                                  │
│ /apps/eb/2022b/skylake/software/nf-core/2.10-foss-2022b/lib/python3.10/site-packages/nf_core/dow │
│ nload.py:1095 in get_singularity_images                                                          │
│                                                                                                  │
│   1092 │   │   │   │   │   │   │   │   # Try other registries                                    │
│   1093 │   │   │   │   │   │   │   │   log.error(e.message)                                      │
│   1094 │   │   │   │   │   │   │   │   log.error(e.helpmessage)                                  │
│ ❱ 1095 │   │   │   │   │   │   │   │   if e.error_log.absoluteURI:                               │
│   1096 │   │   │   │   │   │   │   │   │   break  # there no point in trying other registries i  │
│   1097 │   │   │   │   │   │   │   │   else:                                                     │
│   1098 │   │   │   │   │   │   │   │   │   continue                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'ContainerError' object has no attribute 'absoluteURI'
[hzs682@rescomp2 hzs682]$ singularity pull --name /well/thompson/users/hzs682/singularity-cashdir/ontresearch-modk
it-shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb.img docker://quay.io/ontresearch/modkit:shaeedb131a939d3eea2f9bd4db
ecec805c0fa20bdb
FATAL:   While making image from oci registry: error fetching image to cache: failed to get checksum for docker://quay.io/ontresearch/modkit:shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb: reading manifest shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb in quay.io/ontresearch/modkit: unauthorized: access to the requested resource is not authorized

Application activity log entry

No response

Were you able to successfully run the latest version of the workflow with the demo data?

no

Other demo data information

No response

SamStudio8 commented 2 months ago

Hi @bcardus, our Docker images are not hosted on quay.io. I have not used the nf-core download command but from a cursory glance at the docs it looks like you might need to supply -l docker.io to override the registry. For any further support on the nf-core tools I would ask the nf-core community.

If you want to simply pull our images with Singularity, you should be able to accomplish this by removing the reference to quay.io:

$ singularity pull --name ontresearch-modkit-shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb.img docker://ontresearch/modkit:shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb

Note the command will first pull the Docker image in order to build the Singularity image, as we do not host a registry of native Singularity images.

bcardus commented 2 months ago

Hi Sam,

Thank you for the pointers and apologies if I submitted this to the wrong github page.

The modifier -l docker.io solved the registry issue noted above.

For anyone who finds this thread in the future, I still got different errors after that (creating the SIF file failed for unclear reasons). However, I was able to pull the images from Singularity one by one as you suggested. I had some issues with running out of disc space as I could not figure out how to set the cache to the right folder, and in the end I used the --disable-cache option. This all worked in the end, and I managed to run the workflow.

Thank you again!

Best wishes, Bea

From: Sam Nicholls @.> Sent: Wednesday, April 3, 2024 10:22 PM To: epi2me-labs/wf-human-variation @.> Cc: Cardus, Beatrix (RTH) OUH @.>; Mention @.> Subject: Re: [epi2me-labs/wf-human-variation] nf-core failure to attain singularity containers (Issue #168)

Hi @bcardushttps://github.com/bcardus, our Docker images are not hosted on quay.io. I have not used the nf-core download command but from a cursory glance at the docshttps://nf-co.re/tools#downloading-pipelines-for-offline-use it looks like you might need to supply -l docker.io to override the registry. For any further support on the nf-core tools I would ask the nf-core communityhttps://nf-co.re/join.

If you want to simply pull our images with Singularity, you should be able to accomplish this by removing the reference to quay.io:

$ singularity pull --name ontresearch-modkit-shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb.img docker://ontresearch/modkit:shaeedb131a939d3eea2f9bd4dbecec805c0fa20bdb

Note the command will first pull the Docker image in order to build the Singularity image, as we do not host a registry of native Singularity images.

- Reply to this email directly, view it on GitHubhttps://github.com/epi2me-labs/wf-human-variation/issues/168#issuecomment-2035616631, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A4KFTDOXHQ4SZ66MCKNVYMLY3RXGPAVCNFSM6AAAAABFWB2GDGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZVGYYTMNRTGE. You are receiving this because you were mentioned.Message ID: @.***>