Add CircleCI as an alternative to Travis (Use docker for CI)

[jgkamat]

We've been having issues with Travis due to:

• The speed of the builds. Travis firstly takes 10 minutes to start up and then we spend a majority of our time getting the system to install the packages we need.
• Travis uses Ubuntu 12.04. We could use docker to emulate 14.04 (it will use the older kernel though), but Travis dosen't support using root inside of docker containers, and we can't sacrifice sudo access at this point.

If we switch to CircleCi, with docker, it will solve these two problems of our CI, by

• Giving us access to 14.04 base machines (and any future systems, should we switch)
• Letting us 'cache' dockerimages. The mechanism to do this seems kind of strange, and this needs to be investigated more, but worst case, we can cache these images on the public docker hub and download them from there during build. Letting us 'cache' dockerimages. The mechanism to do this seems kind of strange, and this needs to be investigated more, but worst case, we can cache these images on the public docker hub and download them from there during build.

The only downside of moving to CircleCi is that CircleCi has a limit of 4 builds running at a time.

Caching builds is going to be a little tricky. My proposed solution for this is to make a sha hash of the util folder/ubuntu-setup and associated files and tag all 'baseimages' using that sha. That way, if we update ubuntu-setup, we will push a new updated 'baseimage'. In addition, when we actually do CI, we can tag images with the git sha of the commit, and push the compiled versions with those tags to the dockerhub, so we can have a full history of all our compiled containers.

tl,dr;

• [x] Run CI in docker on CircleCi (keep our travis builds running as is for now).
• [x] Cache ubuntu-setup images on the docker hub, and tag them with a sha.
• Cache the last run ubuntu-setup image on circleCI servers, to reduce download time (optional) Not worth it.
• [x] Transfer ccache data to docker containers
• Tag built images with their commit sha and push them to the dockerhub. Will be done in automated build
• [x] Move the updated documentation script to circle ci using their secrets setup.
• [x] Add proper artifact links to build outputs.

In the future, we can look into using docker as an optional build platform.

[jgkamat]

The foundation for this seems to be in place now, I would like to move this over to a branch on this repo to harden it. Could you enable builds on circleci for this repository? Also, should we create a robojackets dockerhub account and use that to push images to (That would make more sense imo). We would need to add the password/email of that account to the environmental variables of the build as well. See this @justbuchanan

[justbuchanan]

Awesome! Yeah go ahead and make a branch on here for it. Once I find the password for the robojackets email account, I'll setup an account on dockerhub for this. I just enabled CircleCI for this repo (I think), although it's not doing much since it can't find the config file.

[jgkamat]

Cool, just did that. I also think its a good idea to keep travis for style checking (I think that lets us have 2 seperate checks). Its failing right now on the login step, let me know if you need any help with setting up the dockerhub account!

[justbuchanan]

Yeah I'm waiting on Matt for the email password, then I'll put the login info into circle. One thing we could do that might be nice is setting up build artifacts: https://circleci.com/docs/build-artifacts. We could have an info.txt or something that lists which things were run and whether they passed or failed:

• [Passed] Build Firmware
• [Passed] Build soccer, etc
• [Passed] Checkstyle
• [Passed] Firmware Unit Tests
• [Passed] Soccer Unit Tests

And maybe a log file for each so we can debug any issues. Keeping travis around would be ok, but there are several things we test and I'm not sure if having a style check vs. an everything else check really provides that much. It might be nicer to keep it all in one system.

[justbuchanan]

I wonder how tough it would be to use GitHub's commit status API to show a separate check/x for each of the things we test for in circle. https://github.com/blog/1227-commit-status-api. It looks like there might be a way to do this, then provide a custom url that takes you to the corresponding log in circle's build artifacts.

[justbuchanan]

This actually turned out to be super easy! Try running this (except with your username and changing the description, etc), then look on the "branches" page for the repo and click on the status icon.

curl -u justbuchanan -X POST https://api.github.com/repos/robojackets/robocup-software/statuses/7cce87584d04ce436869930375759afb239b85b2 -H "Content-Type: application/json" -d '{"state":"pending", "description": "test status", "context": "justin/my-terminal", "target_url": "http://robojackets.org"}'

[justbuchanan]

I think we can get a pretty sweet setup if we make our circleci scripts post a status for each thing we test, then set the status's url param so it links to the log we put into the build artifacts folder.

[jgkamat]

Woah thats pretty cool, I didn't know it would be that easy! It would probably be trivial (minus authentication) to add a couple wrappers to the make commands that would always pass, but fail checks on here if needed. Travis probably isn't necessary anymore. :stuck_out_tongue_closed_eyes:

Adding the artifacts probably would be a little difficult, but definetly not impossible (with the build actually happening in containers), do we still want that with these multiple checks?

[justbuchanan]

I think for authentication we can use GitHub's app specific passwords/tokens. If you go to Settings -> Personal access tokens -> Generage new token, you can make one with specific permissions granted to it. Then we can just put this in the environment vars in circle and it works the same as a password.

For the build artifacts, I think it would be cool if the status item in GitHub linked back to the corresponding log in the artifacts. I think the only tough part would be generating the URL of the artifact from its file path, etc.

[jgkamat]

Hmm, we'll have to find out how those build artifacts are linked in order to find out what the url would be, but I don't think they would make that too hard.

I saw the access tokens bit, but a problem with that is that if we pass it to a docker layer, it will be accessible from that layer in the build. We can work around this by running 'docker run ./makewithapi $APITOKEN, instead of building an image directly, or by just not publishing the built continuous image dockerfiles. even if someone got a hold of our key, all they could do is change the status of our CI, so I doubt this is a major problem.

[jgkamat]

Also, when we create our dockerhub account, we might want to create an organization, instead of a real account (just like github). Organizations are being released in a few days. https://hub-beta.docker.com/organizations/add/

I could go ahead and make it now if we want (using my personal account as an admin).

[justbuchanan]

Sounds good to me - an organization sounds better than a regular account anyways. Can you go ahead and make and add me as an admin as well?

[jgkamat]

Sure thing, I just made it here, let me know your docker hub handle and I'll add you to owners!

[justbuchanan]

Cool, my username is "justbuchanan".

[jgkamat]

just got you, you should be added now! (You can try to push to robojackets/test to check that everything is working)

[justbuchanan]

Thanks! I'm at "Hello, world" level with Docker and Docker Hub, so I get the concept, but don't have many details figured out haha.

A few questions on how this all will work:

• Do we need to add this repo as an "Automated Build" on the docker hub or are we handling all of the builds through circle and pushing them to the hub so we don't need Docker hub to do it for us?
• I don't see a git clone in the Dockerfiles - is this a mistake or does it get baked into the image because it's all in the same GitHub repo?
• Can we pass the API_TOKEN for GitHub auth to a built docker image at test time on circle and avoid having to bake the token into a built image? That should solve security issues with the token, right?

[jgkamat]

No problem! :stuck_out_tongue:

1. I'm planning to just add a normal docker repo (one that you can push too), and let circleCi push to it. That way we can get the flexibility that circleci allows us (pushing api tokens, sha checks, etc) rather than just blindly building a dockerfile. Automated build works well for creating images with a set script of a app, but not so great for CI.
2. The git clone is 'replaced' with the COPY step, which will copy the current repo into the container. That way, the correct state is brought into the repo, and we are less dependent on internet connections (which can flake).
3. If we do this, we would want to avoid publishing the completed test image (the image created after the tests run), as the environmental variable will still be accessible. We can solve this security problem by not pushing/publishing this 'completed test' image, or by replacing the RUN command for tests with a manual docker run command, which will pass the hash directly to the script, and it never gets saved as an environmental variable in docker. I'm leaning with the latter, in case we want to publish our built images. (This is what i'm working on right now)

Let me know if you have any other questions! :smile:

[justbuchanan]

Cool, makes sense. I guess there's not much reason to push completed the completed test image since there's a different one per git commit. One thing that might be nice is if the base image has already had make run inside of it (even if it's on a different commit or maybe always what's on the master branch) so that most of the build will be cached by ccache. Not sure what the best way to do this would be though.

[justbuchanan]

Also, could we mount the circle BUILD_ARTIFACTS directory as a docker volume, then have it put logs there as they're generated from the test outputs?

[jgkamat]

Hmm, that would conflict in weird ways with the way that I'm doing it right now. The easiest way to do that is to cache whenever the ubuntu-setup scripts change, but that might not be frequent enough. The problem is that if a branch changes the ubuntu-setup scripts, I don't want to rerun ubuntu-setup more than once, (as that would take a while), so maybe we could export the last run ccache and use that for the next build (coping it back into the container when needed)?

As for mounting the build_artifacts directory, that should be easy enough. Do we want to save the entire run directory for the build_artifacts or just a logs dir?

[justbuchanan]

Yeah I don't see a clean way to do it either. I think the way you have it setup where it just caches an image for each unique set of setup scripts and package lists is fine. The installation accounts for most of the time we currently spend on travis anyways.

I don't think there's much point in saving the output binaries right now (although there may be in the future). The main thing I think would be cool is if we saved a separate log file for "make", "make robot2015", "make test-cpp", etc to the build artifacts and linked to them from the GitHub status so we can easily debug issues.

[jgkamat]

Hmm, do you just mean capturing the output of the commands? That should be easy to do, the hard part probably would be getting those links.

[justbuchanan]

Yeah If we can just redirect stdout and stderr to a file for each of those commands, I think we'd be good to go.

Just ran a test (you can see it in the builds list in circle) and it looks like the URL for artifacts is:

https://circle-artifacts.com/gh/RoboJackets/robocup-software/$CIRCLE_BUILD_NUM/artifacts/0$CIRCLE_ARTIFACTS/filename

[jgkamat]

Oh, that makes things pretty easy then, I'll work on getting that format into the urls for the github api calls!

[jgkamat]

I'm going to switch to a slightly different model for this, instead of including robocup-software and compiled files, the docker containers will just link to the files already on your machine. This both ensures that its easy to use ccache for these builds, and speeds up the creation of images (as we don't need to copy 900mb of data inside every build). The downsides to this is that we cannot create runnable images from this process (as the required files are not actually in the container). It makes more sense performance wise to do this as a side-process (probably as an automated build on the dockerhub, as it is right now).

[jgkamat]

This is definitely worth it. As of now Travis build time/first time circle (no dependencies in dockerhub): ~20 min? Circle with repo inside container: ~10 min Circle with repo outside of container: ~7 min Circle with ccache on previous build: ~3 min

If we cache the dependencies then it will increase build time if we need to regenerate/pull the dependencies dockerimage, but decrease it if we use the cached image. I don't think this is worth it at this point, it's pretty good as is.

Only things left to do is documentation updates and publishing the log files.

[justbuchanan]

That sounds awesome, nice work! That's a huge improvement over Travis. I've wasted plenty of time refreshing the travis build page to see if something passed.

As far as documentation goes, there's a little bit of info on the travis setup on the "Testing & Continuous Integration" page in the doc folder. You could probably just replace that with some info on how the new setup works.

[jgkamat]

Yup, I'm pretty sure I know how it works (I actually implemented it on my own repositories!) :smile:

I just need to properly integrate it and make sure it works! I'll probably get around to this later this week...

[jgkamat]

Looks like the artifact links are working (once the build finishes of course). I'm going to touch up a bit more tomorrow, but it's looking like everything is good (minus the style checking, which always passes)

[justbuchanan]

Sounds great! Go ahead and open a PR whenever you're ready.

Btw, one quick thing that might be nice to implement is to set the status to "pending" for each of our checks as soon as the script starts, then have them change to pass or fail once the test is run.

[jgkamat]

also, this happened, but the deployment of artifacts isn't supported on travis (and travis still takes 2+min to start with this setup), so I think its better if we use circle.

[justbuchanan]

Hahaha perfect timing. Screw Travis they took too long to get this feature put together, let's stick with Circle.