search

singularityhub / singularityhub.github.io

Container tools for scientific computing! Docs at https://singularityhub.github.io/singularityhub-docs
https://singularityhub.github.io
68 stars 9 forks source link

Building Singularity image killed after 2 hours sharp (Error: 137) #207

Closed matmu closed 4 years ago

matmu commented 4 years ago

Links

Version of Singularity

Cloud builder singularity 2-5-100GB-private

Behavior when Building Locally

Builds correctly

Error on Singularity Hub

Return value of 137 after 2h sharp.

What do you think is going on?

Time limit of 2h is exceeded. I have outsourced the time consuming steps to the Docker image on https://hub.docker.com. Nevertheless the 2h are not enough. What else can I do? Can I push my local image to singularity-hub?

vsoch commented 4 years ago

You have a 2 hour build limit, and here are some ideas!

matmu commented 4 years ago

@vsoch thanks for your answer. Are these suggestions meant for making the docker image available on singularity-hub.org? If yes, could you give me more specific information on how to do this? E.g. how can I "build a Docker base image and skip time compiling in my build"?

Regarding your first suggestion, I know that I can build a local Singularity image with

singularity build nanopore.simg docker://matmu/nanopore:latest

Is there a way to push it to singularity-hub.org?

vsoch commented 4 years ago

No, Singularity Hub does not allow pulling images. If you want I can offer to help with an automated build for Docker Hub, so you could always pull to Singularity.

matmu commented 4 years ago

@vsoch that would be great!

vsoch commented 4 years ago

okay, looks like you already have Dockerfiles! https://github.com/matmu/vep/. It's still before 6am here and I'm going to go back to sleep, but if you want to make an account on Quay.io or Docker Hub and then add DOCKER_USER and DOCKER_PASS (and for Quay this would be a robot username and token, that is my recommendation) I'll open a PR later today with a GitHub workflow to build and deploy.

vsoch commented 4 years ago

And to be clear you don't need to share these secrets with me, just add them to your repository encrypted secrets and tell me the key that you chose so I can reference them in the recipe. See https://help.github.com/en/actions/configuring-and-managing-workflows/creating-and-storing-encrypted-secrets.

matmu commented 4 years ago

@vsoch Sorry to bother you so early. I have created the encrypted secrets. It's not clear to me however, which key you need. Can you help me with that.

Also, could you use the Singularity recipes unter https://github.com/matmu/vep/ for the automated build? I have added some additional runscripts in there.

vsoch commented 4 years ago

We are going to be building Docker containers for Docker Hub, so please update the Dockerfiles there for deployment. I'll put together a GitHub workflow soon and then use DOCKER_USER and DOCKER_PASS, which I assume you've set as secrets.

vsoch commented 4 years ago

@matmu what is the name of the registry repository and your username that you want to push to?

vsoch commented 4 years ago

I'll need to know at least which registry, and the name of the username/container to open a PR with this first draft. Thanks!

matmu commented 4 years ago

@vsoch I hope, I get you right: I would like to push my images from docker://matmu/vep to shub://matmu/vep by using the recipe files Singularity.* at https://github.com/matmu/vep/. I have set the secrets DOCKER_USER and DOCKER_PASS for my repo at https://github.com/matmu/vep/.

vsoch commented 4 years ago

Sorry I think there is some misunderstanding here - I can create an automated build on GitHub that will build the Docker.* recipes in the repository and push to a Docker Registry and then be available via:

singularity pull docker://<repository>/<container>

You are still not allowed to push to Singularity Hub :0) Let me know if this is not clear...

vsoch commented 4 years ago

hey @matmu - just wanted to share with you that Docker just added an official GitHub Action:

https://github.com/marketplace/actions/build-and-push-docker-images

So this should make it very easy to copy the recipe there and create an automated build to Docker Hub for your Docker recipes! You can then pull those containers to singularity as I showed above. Please ping me if you want any help or review with this work, good luck!

matmu commented 4 years ago

Hey @vsoch,

I apologize for my late reply. You could manage to create my own Github workflow for building and publishing. Unfortunately, Github-hosted runners are also restricted in size but I managed to free 40Gb which is enough for building.

All the best!