Return value 137 when building on the Hub

[agilly]

Links

• container collection: hmgu-itg/burden_testing
• GitHub repository or recipe file: hmgu-itg/burden_testing/Singularity

Version of Singularity

Locally, 3.5.2

Behavior when Building Locally

Builds successfully.

Error on Singularity Hub

[...]
Adding deffile section labels to container
Adding runscript
Adding runscript help
Finalizing Singularity container
Calculating final size for metadata...
Skipping checks
Singularity container built: /root/build
Cleaning up...
Building Singularity image...
Return value of 137.
Killed: Tue Apr 7 11:33:35 UTC 2020.

What do you think is going on?

Not sure. The build process is quite long due to the number of dependencies to download, but this doesn't look like a timeout. Another possibility would be size, 15Gb of data are downloaded during build. Unless it's an error I missed that gets propagated, but then again, build succeeds locally and I can't see anything in the log (https://singularity-hub.org/containers/12687/log).

[vsoch]

The killed signal says that the build is over the time or memory limits, and thus is killed.

[agilly]

Thanks Vanessa, which do you think it is? I can try and make build shorter, but I'm not sure how to limit memory consumption at build time.

[vsoch]

You can tell by looking at the start and ending time stamp for the log - if it’s exactly 2 hours it was time. Of course time is related to size and downloading, as that takes time. Have you tried building on Docker Hub or Quay and pulling to Singularity?

[agilly]

I have a couple ideas to make it build faster, I'll try that first before I convert the recipe to a dockerfile. So 2 hours is the build time limit, what's the memory cap?

Another quick question, there is no way to see the build logs live, is there?

[vsoch]

n1-standard-2 - see here for the stats! https://cloud.google.com/compute/docs/machine-types

[vsoch]

And no, unfortunately logs are known to the server after the build (or when it fails) via sending to storage. If there were oodles of more interest, money, or time for Singularity Hub I could have thought of a live way (seems that all CI services have that) but since it’s a small operation this has worked ok. :)

[agilly]

I see. Well, FWIW I think the whole Singularity ecosystem is an important and useful effort, and hosting containers on the hub for easy access is very convenient. The container in question is actually for analysis by multiple scientific groups as part of a scientific consortium. With this setup, they just have to install singularity, pull the image and run. All libraries and prerequisites are included, and we have reproducibility basically forever, which is quite invaluable.

[ramou]

Amen to that.

On Tue, Apr 7, 2020 at 11:51 AM Arthur Gilly notifications@github.com wrote:

I see. Well, FWIW I think the whole Singularity ecosystem is an important and useful effort, and hosting containers on the hub for easy access is very convenient. The container in question is actually for analysis by multiple scientific groups as part of a scientific consortium. With this setup, they just have to install singularity, pull the image and run. All libraries and prerequisites are included, and we have reproducibility basically forever, which is quite invaluable.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

-- Stuart Thiel, P. Eng.

[agilly]

Unfortunately even after doing some optimisations on the Singularity file, it still seems to build in >2h on the cloud (much faster locally). You mentioned docker, so I used spython with a docker writer to produce a Dockerfile. Had to make manual changes (e.g. cd to WORKDIR) but it eventually built, and was pushed to docker hub.

Still wanted to have an image on shub for user convenience, however even a simple recipe:

Bootstrap: docker
From: agilly/burden_testing

fails to build on shub (same issue apparently, timeout. Logs are in https://singularity-hub.org/containers/12704/log). AFAIU I can't do a singularity push [local_sif] shub://my_collection right? Is there another way this could work that I haven't thought of?

[vsoch]

Can you please post the log here so it’s not behind a login for future inspection? You can use a details html tag if it’s very big.

[agilly]

Sure, here they are:

Start Time: Thu Apr 9 03:40:15 UTC 2020.
Cloning into '/tmp/tmpio8z_mni'...
warning: redirecting to https://github.com/hmgu-itg/burden_testing.git/
Already on 'master'
Your branch is up to date with 'origin/master'.
Using container recipe deffile: Singularity.via.Docker
Building into existing container: /root/build
Using container recipe deffile: Singularity.via.Docker
Sanitizing environment
Adding base Singularity environment to container
Docker image path: index.docker.io/agilly/burden_testing:latest
Cache folder set to /root/.singularity/docker
[1/23] ||----------------------------------| 0.0% [1/23] |=|---------------------------------| 4.3% [2/23] |===|-------------------------------| 8.7% [3/23] |====|------------------------------| 13.0% [4/23] |======|----------------------------| 17.4% [5/23] |=======|---------------------------| 21.7% [6/23] |=========|-------------------------| 26.1% [7/23] |==========|------------------------| 30.4% [8/23] |============|----------------------| 34.8% [9/23] |=============|---------------------| 39.1% [10/23] |===============|-------------------| 43.5% [11/23] |================|------------------| 47.8% [12/23] |==================|----------------| 52.2% [13/23] |===================|---------------| 56.5% [14/23] |=====================|-------------| 60.9% [15/23] |======================|------------| 65.2% [16/23] |========================|----------| 69.6% [17/23] |=========================|---------| 73.9% [18/23] |===========================|-------| 78.3% [19/23] |============================|------| 82.6% [20/23] |==============================|----| 87.0% [21/23] |===============================|---| 91.3% [22/23] |=================================|-| 95.7% [23/23] |===================================| 100.0%
Exploding layer: sha256:84ed7d2f608f8a65d944b40132a0333069302d24e9e51a6d6b338888e8fd0a6b.tar.gz
Exploding layer: sha256:be2bf1c4a48dae92d5a1b8aa319c8767fa491316fb80da52de80378264599a16.tar.gz
Exploding layer: sha256:a5bdc630309340a3154f37e17c00a61c28c476107656e8d6744d2ba9af980058.tar.gz
Exploding layer: sha256:e9055237d68d011bb90d49096b637b3b6c5c7251f52e0f2a2a44148aec1181dc.tar.gz
Exploding layer: sha256:b688b47d6616f20d8788a96cfc6c7a800ca2f6d6f515fb629ff980c5f843c12c.tar.gz
Exploding layer: sha256:be37daaee5721fdbc1eb574d67102c850dfb406b7996b102f23a5aadb9d80c99.tar.gz
Exploding layer: sha256:3b91f5137004a3991e5c293202dbfd5e9728141df6d6ec9073e1b4cdba10be63.tar.gz
Exploding layer: sha256:ccda314280a8e5439c93085d006457dce19a652d03c26f2b0be164fd616d43b9.tar.gz
Exploding layer: sha256:c19df1074b4e435f403ee926c130bff595b4ff356104acf204735ef0e44c660e.tar.gz
Exploding layer: sha256:d34d9bd785716497bb4c66b491d7f8a4312c589cd5067af350aff654ec9cc662.tar.gz
Exploding layer: sha256:6fe759254024f322a2e3eb5d4852a05545f5fb6e586c88b19327c1997af7e1d9.tar.gz
Exploding layer: sha256:4927bcc413327867e1df1844b725f6e95b7729ea2179c9dc7dcaba7202f431e6.tar.gz
Exploding layer: sha256:8e8290d51a99eb1513e651d4c3c5eea8037e4303c0612b1418bc47719b586672.tar.gz
Exploding layer: sha256:c68b12e6a5df3a1c6d60a489cbd46b103d0c2f270dff9f35a54d84c5db71a7c1.tar.gz
Exploding layer: sha256:27d97ba92c4dee3b259f7dd1a2a3af4af91989f54e22b6fc72d1e71bb5ed607c.tar.gz
Exploding layer: sha256:69c2c2dafcc400984af2e09fd91fa9100b0cc41728702a300b785c7779c50b4f.tar.gz
Exploding layer: sha256:ecdd9c6b067cb5e7722ad28656be162cf34c2638123361ae920e6d4a0a3dc7bc.tar.gz
Exploding layer: sha256:bf4350f23d12f7e0add54ef662fe82090a9a432e928a28e627da9eb52ddf0ff9.tar.gz
Exploding layer: sha256:ad30d751442b2b3eee7b454e965e4cf1cfbc66efdbbb56c045d72547f14dc3f5.tar.gz
Exploding layer: sha256:37d57dd5e4b915e432c174b13890600fc0218edf1dc06070f628ac8e9eeb49d4.tar.gz
Exploding layer: sha256:311d533e31d4897c1aea9f0462b375fcbe5063c3877bd9d8d7b0144dbb5671d7.tar.gz
Exploding layer: sha256:894ac1a8310610471f80204ce54a17129e7c06802362d2d9ad48a6087ae9c940.tar.gz
Exploding layer: sha256:d4f0bee8de97cf692ae8b139f95adde73ab240a50ba10bd74c27b58f1188719d.tar.gz
Exploding layer: sha256:3d5bd387145dd9dee5c9f1f8de3713af4ad864bec12c549e78893b635c833aa6.tar.gz
Finalizing Singularity container
Calculating final size for metadata...
Skipping checks
Singularity container built: /root/build
Cleaning up...
Building Singularity image...
Return value of 137.
Killed: Thu Apr 9 05:40:16 UTC 2020.

[agilly]

And attached is the former log which attempted to build everything on shub directly (that one is long, too long even for <details>).

build.log

[vsoch]

Wow! So the container is really so big that it takes a full two hours just to pull layers and try to create the sif - you can see the time stamp cut at 2 hours with the kill message to a T. 14.5GB! That might be the biggest that I’ve seen attempted to build:

https://hub.docker.com/r/agilly/burden_testing/tags and actually we can’t even be sure it would ever finish that, the machine has only ~7.5 GB of RAM - my computer often freezes building large images (much smaller than that) and it has double the memory so I suspect this just isn’t going to work. I can offer some comfort that a singularity build from Docker is doing almost exactly the same thing, but extracting the layers from Docker Hub before building an image. I’m afraid I don’t have any suggestions because Singularity Hub doesn’t support images that large, we could talk about ways to make the image smaller (e.g are you serving data directly from it that could be bound instead?) but other than that I’d need to allow larger (more expensive) instances and longer build time, which I can’t do. I’m really sorry @agilly your image is just massive!

[agilly]

I was afraid you were going to say that 😅 The image is massive because of one single large data download, which is performed at install time by a crucial piece of software in there (ensembl VEP). I had already thought about ways to make the user download it and then mount it, but it's a bit tricky because changes are then needed in multiple bits of code. Until we decide to embark in that direction, I will stick to a docker hub image for now.