rsyslog docker development containers

[rgerhards]

Not really an issue, just some info. My current playground: https://github.com/rgerhards/rsyslog-docker

Might be worth mentioning. I am currently after the low-hanging fruit, setting up containers for building rsyslog. Useful for CI, BTW (we finally seem to get at least some coverage for Ubuntu devel :-)).

[deoren]

Thanks!

[halfer]

@deoren: I totally recommend learning Docker, it is surprisingly easy to get into. The docs are excellent, and they contain a simple example to get you started.

Docker really is just a nice wrapper around Linux "cgroups" and "namespaces" that allow code to be isolated in containers, much like FreeBSD's jails. It's a modern interface around a fairly old concept. Since containers do not have to contain much, they are faster to stop and start compared to traditional full-fat virtualisation. Their RAM requirement is impressively small as well.

The other main beginner tip is related to getting used to Dockerfiles. Container file systems are built in layers, somewhat analogous to version control commits, which means:

• each statement in a Dockerfile that writes something will write a new layer
• when re-generating a Dockerfile, if a statement has not changed, then the cached layer can be used instead, speeding up the build hugely

It's easy to get started - just sudo apt-get install docker.io and try the build command I have supplied, or follow the Docker tutorial. Ping me here if you need a hand!

[deoren]

@halfer Awesome, thank you for the encouragement and the offer of help, I really appreciate it!

[rgerhards]

@halfer Thanks for words! Just to note, I have create a project on the rsyslog side for docker: https://github.com/rsyslog/rsyslog/projects/5

From my initial playing with it over the weekend, I think I already see a number of potentially smaller but beneficial changes that could go into rsyslog to provide aid to better build containers. I will also see that we build a first-class container, featuring all rsyslog has to offer in a simple to use setup (github issue not yet created ;-)). I would definitely appreciate your advise and help on it!

[halfer]

Some other general thoughts:

• In the Dockerfile I have made use of Alpine Linux, based on BusyBox. It is a very compact version of Linux, which on its own weighs in at an impressive ~4M. Once everything is installed, my image comes in at a lowly 18.3M, which is still significantly better than basing it on full-fat distros. However, you are most welcome to tweak this based on your OS preference (bear in mind though that users in containerised environments do like a small image).
• I've used the repo version for Alpine, but if you wanted to create a build tracking your master (or latest release) then you would need to do an in-container compile build. In turn that would require the installation of build tools, which can increase the image size.
• If you go down the compile route, there are a number of tricks you can use to install the build tools, do the compilation, and then remove/purge the build tools in one go. Remember that one a command has increased the size of an image, you cannot reduce it in a subsequent command. However, RUN commands can be as long as you like!
• Once you are more comfortable with Docker, there is a newer feature in which you can create an intermediate image (containing build tools) and then create a final image just from its outputs (binaries). The intermediate image is then thrown away, see more here.
• I have needed an init process because I wanted an intermediate command (/root/run.sh) and this won't pass on signals very well to Rsyslog. However if you don't need this, make Rsyslog your command (so it will always be PID 1) and then you can drop dumb-init.

That's all for now, will add anything else if it occurs to me.

[deoren]

@halfer thanks for the tips. It is tidbits like that which would have taken me a long time to figure out on my own (well, with Google's help anyway). ;)

[rgerhards]

@halfer thanks again for the info. I have played the last couple of days with the docker project. I have taken a somewhat different approach in regard to current packages: I took some time to learn Alpine packaging, which is surprisingly nice and easy. So I now have a two step process:

1. create APKs for things that I am missing
2. use these APKs and apply them to a minimalist Alpine image

That way you do not need to do any cleanup at all. I will also try to submit my patches to Alpine, so that for others it is easier to get needed rsyslog features. My current unstable alpine rsyslog repository resides at http://build.rsyslog.com/alpine/3.7/unstable, just FYI.

It's actually a three step process, as I used another container for creating the build environment for step 1. The current state resides at https://github.com/rsyslog/rsyslog-docker and the homepage README has a description of my bootstrap process.

Any comments or new issues on that project are appreciated. I am now working my way up from the base. One thing that turns out I need is an Ubuntu image that I can use to simplify my testing of rsyslog container features . The full cycle of edit-compile-build_tarball-build_APK-build_appliance is too time consuming to be productive. So I need some thing were I can map in the binary right after compile.

Of course, all is still in infancy!

Note that I already auto-detect if rsyslog is running inside a container, so that I can adjust defaults and "other things" (whatever this may be): https://github.com/rsyslog/rsyslog/pull/2373

Your postings have been very helpful in getting to todays state. Please continue to let the thoughts flow!