Build dockerfile with Quay

[suujia]

https://quay.io/repository/suujia/orca/build/89d6336c-b862-4b64-864d-0f50086b7f42

I combined orca into one dockerfile and built it on Quay. Needs debugging.

[suujia]

Build failed at pip3 install htseq

reproduced error on linuxbrew/linuxbrew docker image (after installing dependency numpy):

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-jt0ih3r0/htseq/
Cache entry deserialization failed, entry ignored

same error appeared in singularity image build with htseq

raise DistutilsError("Setup script exited with %s" % (v.args[0],))
distutils.errors.DistutilsError: Setup script exited with error: command 'cc' failed with exit status 1

[sjackman]

I've opened a new issue for htseq: https://github.com/bcgsc/orca/issues/60

[suujia]

I did another build on quay with one full dockerfile after commenting out htseq (failed on jdk):

Cannot write: No space left on device
PMcurl: (22) The requested URL returned error: 404 Not Found
7/9/2018, 5:33:49 PMError: Failed to download resource "jdk" Download failed: http://download.oracle.com/otn-pub/java/jdk/9.0.1+11/jdk-9.0.1_linux-x64_bin.tar.gz

[sjackman]

Cannot write: No space left on device

Quay.io Support reports that they have a 40 GB disk space limit. It looks as though we're exceeding that limit. Could you please post the URL to the build log with the above error?

[suujia]

https://quay.io/repository/suujia/orca/build/de86e6d0-95fe-47ed-b985-08347f20ab0a

This is the build log for the one big Orca Dockerfile that failed with no space left on device, on jdk.

Oh we can even pull singularity image from quay dockerfile. (Requires some configurations) https://github.com/singularityware/singularity/issues/749#issuecomment-309934407

==== hmm this is all I found in documentation was on quay enterprise (though we are not using enterprise)

MAXIMUM_LAYER_SIZE [string]: Maximum allowed size of an image layer. Defaults to 20G.
Pattern: ^[0-9]+(G|M)$
Example: 100G

I can try building a slightly smaller image in case it defaults to 20G

[sjackman]

Oh we can even pull singularity image from quay dockerfile.

That's cool!

[suujia]

Yeah, its failing quite early on: here

Wrote only 9216 of 10240 bytes                (the prior build) 

http://blog.devtable.com/

For this reason we now limit each build step in the Dockerfile to 20 minutes. If any individual
step goes over 20 minutes, it will be killed and the build will be marked as an error. If you 
have a build step that can not possibly be split up to fit within 20 minutes, please send us 
a note at support@quay.io with a description of your use case.

[sjackman]

More specific link at http://blog.devtable.com/2014/04/do-you-want-to-build-image.html

Ugh. That's not very helpful. You can put each brew install command on its own line. So change

RUN brew install \
a5 \
abacas \

to

RUN brew install a5
RUN brew install abacas

You'll probably exceed the Docker limit of 255 layers in the image though. Could still be a good experiment to try. We can put Quay.io onto the back burner for now though.

[suujia]

Ok I am rebuilding a big chunk of it here: https://quay.io/repository/suujia/orca/build/427a360b-a987-4bf8-9c4f-0aa6f6094041 (updated link)

Yeah, even splitting it up the error occurs at jdk:

Cannot write: No space left on device

[sjackman]

I've confirmed with Quay.io support that the time limit is twenty minutes per command, and there's no way of increasing the timeout.

[sjackman]

Quay.io has a really helpful direct chat support team. Go to https://quay.io and click on How can we help you? in the bottom right corner. You can ask them about the Cannot write: No space left on device error message. My understanding is that the disk limit is 40 GB, which should be plenty. Point them to the failed build log.

[sjackman]

Rather than one formula per RUN line, I'd suggest grouping them into batches of ~20 minutes each, so perhaps 10 or so formulae per batch. Best practices with Docker are to keep to a minimum the number of layers in the image (each RUN command creates one layer).

[suujia]

Quays support

I apologize, I just got an information that the size limit for building images is 10Gb. 

In the mean time, you can use a third party builder (like CircleCi or build it on premise somewhere) 
and then push the image to Quay. That should work, provided that the network link between Quay 
and your computer or the third party is stable enough so it won't time out ahead of schedule.
Other than that, you can perhaps try and reduce assets of the container, if possible.

Hmm, these CI tools have even lower memory or timeout I believe (circle CI 4gb). I'm rebuilding the image on Quay with half the number of tools just to test: https://quay.io/repository/suujia/orca/build/0e17bba1-c541-4953-9491-067c6b50b200

tar: gcc/5.5.0_4/libexec/gcc/x86_64-unknown-linux-gnu/5.5.0/cc1plus: Wrote only 5120 of 10240 bytes
tar: gcc-5.5.0/ABOUT-NLS: Wrote only 1024 of 10240 bytes
No space left on device

[sjackman]

Cool. Thanks for confirming. If you removed just the largest packages, you may be able to fit a reduces version of the ORCA image in 10 GB.

Here's the largest packages:

$ du -csh /home/linuxbrew/.linuxbrew/Cellar/* | sort -h | tail   
428M    /home/linuxbrew/.linuxbrew/Cellar/mysql
464M    /home/linuxbrew/.linuxbrew/Cellar/amos
491M    /home/linuxbrew/.linuxbrew/Cellar/mitofy
523M    /home/linuxbrew/.linuxbrew/Cellar/prokka
558M    /home/linuxbrew/.linuxbrew/Cellar/jdk
651M    /home/linuxbrew/.linuxbrew/Cellar/metaphlan
679M    /home/linuxbrew/.linuxbrew/Cellar/blast
1.6G    /home/linuxbrew/.linuxbrew/Cellar/repeatmasker
4.2G    /home/linuxbrew/.linuxbrew/Cellar/llvm
19G total

Those 9 add up to ~9.5 GB.

[sjackman]

It looks as though Quay.io is not an ideal replacement for Docker Hub due to these resource limits: 20 minutes per command, 10 GB disk. We can put this issue to rest I think. Thanks for your work on it!

[suujia]

Ok sounds good! 👍