Docker image build for servers, Jenkins release build trigger

[qwc]

Hi!

I created a docker build repository and would like to ask if you could put a docker build trigger inside your Jenkins release-build.

I would also like to add some branches like 'unstable' or other tweaks to build multiple docker images for testing purposes.

Basically I started this https://github.com/qwc/docker-terasology repo quite some time ago, but lost track of versions due to other tasks, so I thought about asking for a build trigger so I just do not have to keep track of current changes, except a dependency fails. Maybe I am able configure a warning email inside docker when a build fails. :)

This is the docker repository: https://registry.hub.docker.com/u/qwick/terasology/ The repo description and surroundings are still in a draft state, but to get it really running I want to automate the process with the triggers, when this is running I plan to beautify the repo itself. :)

So, tell me your thoughts.

[msteiger]

Sounds interesting. I'm not sure what exactly you would need. Just send a notification ping your way or is it neccessary to run the script from your repository from within Jenkins?

I don't think that running docker on Jenkins (slaves) is possible, but maybe @Cervator (our DevOps specialist) might know better.

[Cervator]

Totally could with the right size, one of the DigitalOcean images comes pre-baked with Docker set up and everything :-)

Super neat topic and so much we could play with, just arghtime

[qwc]

Nah, a build on your jenkins server is not necessary, but possible. :)

I need you to do only one thing: to call a specific URL everytime a build succeeds, here some examples:

        # Trigger all tags/branchs for this automated build.
        $ curl -H "Content-Type: application/json" --data '{"build": true}' -X POST https://registry.hub.docker.com/u/qwick/terasology/trigger/<trigger-hash>/

        # Trigger by docker tag name
        $ curl -H "Content-Type: application/json" --data '{"docker_tag_name": "master"}' -X POST https://registry.hub.docker.com/u/qwick/terasology/trigger/<trigger-hash>/

        # Trigger by Source branch named staging
        $ curl -H "Content-Type: application/json" --data '{"source_type": "Branch", "source_name": "staging"}' -X POST https://registry.hub.docker.com/u/qwick/terasology/trigger/<trigger-hash>/

        # Trigger by Source tag named v1.1
        $ curl -H "Content-Type: application/json" --data '{"source_type": "Tag", "source_name": "v1.1"}' -X POST https://registry.hub.docker.com/u/qwick/terasology/trigger/<trigger-hash>/

(I've removed the trigger hash, so no external user is able to trigger builds atm.)

Through this docker API we could build every type of successful build: release, testing, nightly and different types if necessary: server (which I built the repo for), client (yes you can use docker even for GUI programs) and other which did not yet come to my mind.

Btw. this api looks like it's not so old, when I first started the terasology docker repo this menu option was not available at the docker system, or I've overlooked it completly... :D

The only thing for which you would need to contact me through a pull request would be to change dependencies in the Dockerfile or maybe I'm able to add you to the docker-repo (not the github one!) too so you could manage the builds, tags and branches.

@Cervator Yes, time, that's why I lost track of the terasology releases between the creation of the Dockerfile and my post here. And the reason I want to automate that further. ;)

[edit] Oh btw. already configured docker to send me an email if a build fails, so that problem's gone. :)

[Cervator]

Cool stuff, that doesn't seem too bad :-)

I understand Artifactory also supports serving as a Docker registry, and in turn Bintray as well. We're thinking about setting up our formal binary distribution that way (in addition to GitHub as a secondary and Jenkins as tertiary). Any interest in helping out with that? :D

[qwc]

Oh, interesting that Artifactory is able to host docker repositories. Didn't see that coming. :D

For now I just wanted to automate the builds of my docker repo, but when you want to take the approach to a whole new level it's your decision to do that.

Well of course I can help a bit, Docker itself is quite easy, easier than I expected myself, when I started tinkering with it. I'm sure you will learn how to use it efficiently quite fast. ;)

But for Artifactory docker repo hosting you need the Pro or SaaS version, do you get that for the open-source project terasology? o.O

[msteiger]

Hmm .. there's a plugin for Jenkins that sends out notifications, but not too configurable.

Maybe we'd really need to add a post-build step and use curl instead? Sounds like a cheap hack and also won't work on Windows machines..

[qwc]

Who needs windows machines these days? ;) As long as the output of your build is platform independed, which is provided through using java.

Docker in the end is a linux (bsd too? o.O) only thing, so...

Oops, I have to revise my statement. Docker IS possible on Windows. :D But you are not able to dockerize windows applications, you're forced to use linux containers. See: http://docs.docker.com/installation/windows/

[msteiger]

I meant that the machine that runs Jenkins could be a Windows machine, so curl wouldn't work. That said, my main concern is about running "arbitrary" shell scripts on Jenkins.

There's a Docker plugin for Jenkins - does it provide what you need?

[qwc]

Reading the description of the plugin you mentioned, it's more for the other way around, that a dockerhub repo hook triggers a Jenkins build (which may also build a docker image and uploading it). I need the opposite of that for my current approach. That Jenkins triggers a docker image build at dockerhub.

We could also write a small java program, which is executed by jenkins and which sends the required data to dockerhub. Advantage: It runs on windows and doesn't need a linux tool. And you do not have to worry about docker on windows, because dockerhub builds the image not your jenkins. Of course not suitable when you want to build the docker image yourself on your build server. ;) For that you need docker installed locally, whatever OS you're using.

Is your OS, where Jenkins runs, changing often?

[msteiger]

A quick search on google resulted in docker-build-publish-plugin. What do you think of it?

[qwc]

That looks very promising.

But with that plugin you will build the docker image on your jenkins server and not on dockerhub. Which results in quite a bit traffic for the base images and only a little load for plugging the parts together.

And you need to configure a dockerhub account with all credentials.

[Cervator]

Wouldn't we still publish the image to the registry? Yeah I don't think we'd want to host it in Jenkins but it would be nice to have a proper process in there to create it. Account shouldn't be too much hassle :-)

But yeah, it is more work, sure, I get that. Web trigger is easy for current setup. Should we just add a single trigger to the end of the DistroOmegaRelease job for now? Then when somebody has time and energy to mess with Jenkins we can get fancier there later.

[qwc]

Of course you would still publish the image, dockerhub is the central point in the docker universe.

Yeah, that was my goal with this issue! ;)

I'll send you the trigger links and json data for the curl calls during the weekend (can't predict when I'll have time for managing the dockerhub repo/project).

Regarding the docker-build-publish-plugin: I have my own little server where I want to install a jenkins (through docker, hehehe) anyway, so I may have time the upcoming weeks for playing around with this plugin.

[qwc]

To continue this path, let me know when you have time to set up the triggers. :)

[msteiger]

Triggers set up in a dedicated new Docker job: http://jenkins.terasology.org/job/DistroOmegaReleaseDocker/

[qwc]

After I've set up a own jenkins instance I've also put a job up which monitors your build page for build number changes to build the image tagged as 'latest'. The only thing I'm still missing is a properly tagged image with the version as a tag. Thought about parsing the github release page of the project to get the version tag without checking out the complete repository. But I've put that aside because it's too errorprone when you're tagging the successful build later than my job runs (which is daily ~midnight UTC).

If you have the git tag available at your "DistroOmegaReleaseDocker" job, you could change the source_name to "master" and the docker_tag to the current version tag.

[Cervator]

@qwc would the version file at http://jenkins.terasology.org/job/Terasology/lastSuccessfulBuild/artifact/build/resources/main/org/terasology/version/ help?

That's generated for every engine build, admittedly, while the download goal is the Omega zip with all modules added. That usually becomes available a few minutes after the version file is updated.

There is some logic in our Launcher project that goes and scrapes some info out of the paired Jenkins jobs to make a series of stable/unstable builds available in a drop-down, like https://github.com/MovingBlocks/TerasologyLauncher/blob/develop/src/main/java/org/terasology/launcher/game/TerasologyGameVersions.java#L86

[qwc]

Yep, that should work. Thanks for the hint! :+1:

I'll test that as soon as I'm back at home. ;) When this job works too, you could delete your docker job because I'm then providing a solution for all image build possibilities by myself.

[qwc]

Well, after months with no time at all for things like terasology, etc. I've finally managed to set up a versioned build of the image. Sorry for the delay, in the end it's all hobby. ;)

So, the "latest"-build is built by dockerhub through a build-trigger from my own jenkins instance which checks your "versionInfo.properties" from your stable build periodically ( once daily, midnight). Another job, also checking the TerasologyStable build, builds the image on my server and tags it with the current version extracted from the "versionInfo.properties", the resulting tag is: $engineVersion-$displayVersion

At the time of writing this is "1.2.1-alpha".

So that I've now managed to set this all up (1 hour of playing around...) I would like you to disable the "DistroOmegaReleaseDocker" build job on your jenkins, because this is now completly handled through my own jenkins instance. :)

Already more than 175 pulls of the image, hopefully more happy users in the future. Sooo, thanks for your support in making this film ... errr... docker image. If anything goes wrong (with the image), or when there are further questions, don't hesitate to ask. :)

[qwc]

And last but not least I've also created a build job to build the nightly Omega release. docker tag "dev-latest" or versioned with (at the moment): "dev-1.2.2-SNAPSHOT" see: https://hub.docker.com/r/qwick/terasology/tags/

Happy headless server debugging for those of you who might use linux and docker for developtment. :)

[Cervator]

Thanks again for sticking with this so long! We're all here for the hobby so we understand things take time. Heck this one only took a little longer than a year to finish! :D

I've added some info to our Playing.md covering this approach, and will consider that the commit that'll finish this issue :-)

Disabled our Jenkins job for now and also added you to the credits for good measure!

175 pulls already? Where are all these users hiding :8ball:

I wonder if we should try to default this server's config to use JoshariasSurvival, since it has a fair bit more content than the default Core.

[qwc]

Cool, thanks. All the important information is covered by the doc, nice! :) Yeah, more than a year. Well, better late than never. xD