search

singularityhub / singularityhub.github.io

Container tools for scientific computing! Docs at https://singularityhub.github.io/singularityhub-docs
https://singularityhub.github.io
68 stars 9 forks source link

building gatk4: git-lfs rate limits exceeded #131

Closed IHautaI closed 5 years ago

IHautaI commented 6 years ago

Link to Container Collection Log, Build, or Collection (in that order)

Building a container with everything necessary for the GATK4 preprocessing pipeline ( in SCIF, as opposed to their docker image ) for use with nextflow

https://www.singularity-hub.org/containers/2801/log

Behavior when Building Locally

Builds just fine locally.

Error on Singularity Hub

git-lfs rate limit exceeded ( near bottom of the log ).

What do you think is going on?

No idea. Not sure why it would exceed the rate limit on singularity-hub but not locally.

I'm doing the localJar build and I'm not pulling the test files, just the ones necessary for building: https://github.com/broadinstitute/gatk#building

vsoch commented 6 years ago

When you clone the repo locally, I assume you are logged in with your Github? It could be that there is some default quota, and as an anonymous user you exceed it on this clone. I am in the middle of traveling but I'll take a look at this as soon as I get back to my computer! As a sanity check try doing the clone where you are an anonymous user.

vsoch commented 6 years ago

Ah, looking at the log I think it's likely you ran out of disk space. How big is the entire download?

IHautaI commented 6 years ago

Thanks for the quick response! I'm checking on the anonymous user build right now. The final size of the image is 2.6G, not sure on how big the GATK clone is when building.

IHautaI commented 6 years ago

It doesn't appear to be github login. Total gatk size was around 890M after git-lfs ran.

vsoch commented 6 years ago

Okay, it looks like the error code is 429:

429 - The user has hit a rate limit with the server. Though the API does not specify any rate limits, implementors are encouraged to set some for availability reasons.

And there is this additional (unrelated?) error:

debconf: delaying package configuration, since apt-utils is not installed
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory)

The other thing that sticks out is the "smudge error" and I've never seen this before (albeit I don't use LFS). I would try various ways of doing the checkout, or adding debug/trace to using lfs?

https://github.com/git-lfs/git-lfs/issues/2652

You could also try getting the files another way?

vsoch commented 5 years ago

Do you still have an issue here? Closing, as this is an issue with git-lfs.