Automated build

[nthiery]

It would be nice to have dockerhub build automatically our images. This would save us from having to maintain the infrastructure to do those builds on a regular basis. As a nice side effect, the docker image description on dockerhub would be automatically filled from the README (see #5)

The show stopper may bethat there are resource limits that dockerhub impose on the build process. See: http://stackoverflow.com/questions/34440753/docker-hub-timeout-in-automated-build. We may not fit in them.

[saraedum]

We can work around the resource limit by using linked repositories. It's the officially proposed workaround. I am trying to setup a prototype.

[saraedum]

This is more limited than I thought. It appears that you can only do this when you only care about a tag such as latest. In more general setups this does not seem to work (linked repositories ignore rebuild rules that use a regex.)

[saraedum]

It's a bit counter to the way a Dockerfile usually works, but starting compilation from a sage/ directory containing all the temporary files of a previous build, compilation finishes in well under one hour. I wrote a multi-stage Dockerfile which has a switch for such compilation from a previous run at https://github.com/saraedum/sage. I still have to try it out with dockerhub but it could do the trick. Note that we could copy over ccache like that as well (see #21.)

[saraedum]

So dockerhub runs a quite old version of docker, so I run into errors. It seems that they are not really developing dockerhub anymore. And docker cloud has no free autobuilds for open source projects anymore.

[saraedum]

Ok, I now have a working version (using CircleCI for the build) which pushes the images and the READMEs automatically at https://github.com/saraedum/sage/tree/circleci which builds to https://hub.docker.com/r/saraedum/sagemath/ and https://hub.docker.com/r/saraedum/sagemath-dev/.

[saraedum]

@nthiery, @embray: Are you zulip, so we could discuss this there?

[embray]

No, but I could try to be. But I am leaving in ~10 minutes. Give me chance to look over what you've been doing. It sounds impressive!

[embray]

I see you've completely reworked the Dockerfile using multi-stage builds. That feature didn't exist when I first started working on it, but it looks like it will make things simpler than my Makefile. I'm still not 100% sure how I feel about some of the fine details but I'm going to take a closer look tomorrow.

[embray]

I haven't used Circle-CI yet personally, though I've seen a few other projects that do. What do they offer that makes it such that there's enough time to actually perform the builds?

[saraedum]

They don't have time limits actually, i.e., you can set the timeout. They just limit open source projects to four concurrent builds. I think I misread something here. Not sure if you can get over the two hours limit.

[saraedum]

The current Dockerfile runs in less than one hour actually. The limiting factor of docker hub was that I ran out of space (inodes?) and that their version of docker is quite old (surprisingly.)

[saraedum]

About multi-stage builds: yes, it's somewhat new and makes the builder pattern mostly unnecessary. I like the idea of having everything in one Dockerfile that actually works, so people can just type docker build . to try it out. Of course, I copied many of the tricky parts from the existing Dockerfiles and I also don't think that my approach is perfect :)

[saraedum]

I am also not sure whether I forgot to copy something over. My dev image is ~4GB~ 3GB and the non-dev is just 800 MB (compressed.)

[saraedum]

From 8.1 to 8.2.beta3 things have changed in the build system it seems, so CircleCI is now essentially rebuilding all of Sage's dependencies. It won't manage to do that on one core in 120 minutes. :( I am not sure what we would want to do about things like that. We could build and push manually in cases like that but that's quite annoying. An alternative would be Google Cloud's container builder which has no limits. It's free for the first 120 minutes per day and after that costs about $0.20 per hour. The main problem is that we don't get publicly visible logs I think and it's harder for people to set up themselves. Another option is GitlabCI (which I like anyway.) I can set the runtime to 600 minutes per job and it does not complain and they say that there are no limits for open source projects (at the moment.)

[saraedum]

I'll try to transfer the circleCI build to gitlab and see if that's more convenient. They give us two cores, 50GB of disk, 2GB of RAM, and a recent docker.

[saraedum]

The CircleCI build actually finished after about 4 hours. Apparently they increased their time limit to five hours: https://discuss.circleci.com/t/maximum-build-time-for-builds/18383

[saraedum]

So the CircleCI build works after all. Anyway, I started working on a Gitlab CI version here: https://gitlab.com/saraedum/sage/tree/gitlabci (click the round icon next to the commit hash to see the build process.)

[embray]

Just to be clear, there's nothing different in the Dockerbuild itself between your gitlab and circleci builds right? It's just the configuration of the CI service itself?

(I actually didn't realize Gitlab was offering a CI service now)

[embray]

I'm also making good progress on improving Sage's ability to use system packages for many of its dependencies, so that should also squash build time where possible. Obviously this can still get tricky. We want to make sure we choose versions of packages that work well with Sage (and that, in turn, could impact what distro(s) we use as the base image).

[saraedum]

@embray: Yes, there should be no difference between CircleCI and GitLab CI, they run the same docker commands to build the images.

[saraedum]

The runners for GitLab CI seems to be a bit underpowered:

ERROR: Job failed: execution took longer than 10h0m0s seconds

Probably we want to use paid (but cheap) runners here…or stick with Circle CI but Circle CI also barely finished in time. Or just increase the build time limit to a couple of days… I'll document the options and let users choose their favorite approach.

[saraedum]

I created a ticket at https://trac.sagemath.org/ticket/24655#ticket for this as I would like to include the Dockerfile and recipes into the main repository.

[saraedum]

Btw., I added a README that you can see here https://hub.docker.com/r/saraedum/sagemath/.

[saraedum]

GitLabCI is not really underpowed. CircleCI usually is giving us 32 cores to work with. No wonder it builds much faster there… /proc/cpuinfo reports 32 CPUs but we only get 2vCPUs. It's still much faster probably because we get more computing power if other users don't use theirs completely.