Source Code to Dockerfiles

[logan-connolly]

Hi @spurin, first want to say thanks a lot for such an awesome course on ansible. I have particulary enjoyed the lab environment for testing playbooks. I think getting your hands dirty is the best way to learn this tool.

Therefore, I was wondering if you also open-source the Dockerfiles that are used for the ubuntu-c, ubuntu, and centos images? I had a look around, but could not manage to find anything.

I think it would be an excellent reference for someone trying to create a sandbox environment with different OSes, dependencies, etc. If not, were there any references that you found useful?

[spurin]

Hi @logan-connolly,

Thanks for reaching out on this and it's great to also see support from @carlosdoliveira on your post.

Some good news on this, I'm going to be creating content external to the course (hosted on YouTube) and this very topic, is one that I will be covering early on.

At the moment, the content for this in full transparency ... is a bit of a mis-mash. It would span multiple repositories and I've not shared it yet as it needs a really good tidy up. There's also aspects of this that really need explanations and comments, hence why I'm going with a supporting video as an approach rather than just releasing.

Will keep this issue open and will update this, as progress is made.

In the interim, it's worth you having a look at ttyd (https://github.com/tsl0922/ttyd) to create the web based terminals. I used this for the lab and packaged it in a container with systemd. It's a brilliant project and is very simple to use, essentially, you run the binary specifying a port, and an executable (for example bash) and this sets up the web terminal for you :-). If you have a unix system you can quickly try this out by downloading a binary from the release section on the repository.

Outside of that, I used systemd in a container which is a bit of a hassle (and can be difficult to troubleshoot, especially where systemd varies on the parent system), it does however have benefits that were essential. For example, on the course it was important that the likes of systemctl worked as expected so that we could manage services with Ansible as you would in a full VM. If you don't need this for your lab environment, I strongly recommend using supervisord in your container images instead. It's a lot easier to manage and requires less permissions to run. There are heaps of examples for supervisord with Docker.

I hope for now that this information helps and will be revisiting this with a detailed video, soon!

Best Regards

James Spurin

[logan-connolly]

@spurin thank you so much for the detailed response! This is an excellent starting point and I am really looking forward to digging into ttyd and playing around with different system management tools.

Also looking forward to the new videos coming out. I'll be sure to subscribe to your YouTube channel.

Thanks again for all the hard work you put into your courses and community; it's really above and beyond.

[rkujawa]

Hi all, I just wanted to throw my 3 cents into this discussion.

There is also an important aspect related to security, that you may be overlooking here. While using containers for this course makes the whole lab environment much lighter and easier to deploy, it is also not without problems. As @spurin has probably realized while preparing the lab materials, preparing containerized environment simulating full-fledged VMs is not trivial. The lab containers use privileged Docker mode, which grants them extremely elevated permissions in the host OS.

Since source of Dockerfiles is missing, no one can reliably audit the images (at least without serious reverse engineering), or rebuild them. Course attendees are essentially downloading untrusted binary programs from the internet and running them with root privileges. No one in their right mind should do that.

I understand that @spurin has no ill intentions, but these days on the internet we should not trust anyone. Especially, we should not teach people to do such naive things. This has come to my attention, since a colleague of mine has acquired a course by @spurin and asked for my help with setting up the environment.

Btw. the course itself looks really nice!

[spurin]

Hi @rkujawa

Thanks for your kind words and appreciate you throwing your 3 cents in!

It's been my intention for a while to share the source and processes for the lab images.

Whilst this may sound trivial, I built these images so they are stacked for reusability. The end result is made up of 3 base images before it gets to the source image (i.e. Ubuntu or CentOS). Therefore, it's more effort to document/share and explain the end to end setup.

I have also had doing a fair amount of ongoing work and maintenance with them over the last year and a half and a personal goal has been closing out the difficulties of cgroups v2 not working as expected before I go through the steps referred above.

If you look through the issues page you'll see that the release-candidate branch now resolves this!

Lastly, I think that this whole journey with making these images will be some fantastic content in itself and therefore I want to put some creative efforts into it. There's two issues I encountered during this voyage that took me months to resolve and I think people will find them super interesting.

Those who follow me on LinkedIn will see that in the last few months, I made the move to working on DiveInto full time. Am currently working on the 3rd edition of the course which is mostly audio/visual enhancements alongside fixes/updates and I'm currently on chapter 6 out of 10.

When this is complete, this will be my next target!

To add to this, if anyone in the interim feels strongly about this, then there is also the https://github.com/spurin/diveintoansible-lab-gcp-cloudshell option which is a one click Google cloudshell option. You don’t need to trust the image and with this, you can have a lab running outside your infra, in a couple of clicks.

I hope the above context helps 👍

[spurin]

Hi All,

Hope you're doing well and are having a fantastic new year!

Keeping to my promise, I'm now in the process of adding the source to the containers. I'm coinciding this with an updated release of the labs for the course.

There should be 3 or more repositories for this when it's complete (as the Dockerfiles are stacked, building upon each-other). I'm just in the process of the first one which will be the base, the linux distribution + systemd.

You can see a ubuntu example here - https://github.com/spurin/container-systemd/blob/ubuntu_20.04/README.md

Further updates to follow on this thread

[spurin]

Further update, these are progressing, still more to do but, here's the foundation:

Core systemd images

https://github.com/spurin/container-systemd

Builds upon the image above and adds SSHD and ttyd, user management and fixes to /bin/login (needed for CentOS systems with ttyd)

https://github.com/spurin/container-systemd-sshd-ttyd

More to come 🚀

[spurin]

Am very pleased to let you followers of this issue know that the sources for the lab images are now complete! 🚀 🥳

Here's the last component that builds upon the previous two container images that I shared above -

https://github.com/spurin/diveintoansible-images

I've taken this opportunity to refactor/cleanup/comment the Dockerfiles and some of the contents used in the images, i.e. helper scripts. Ansible in this one is also updated to the latest which is currently 5.2.0, lastly I've also included my build scripts in the repository should you wish to use them yourselves. Where external precompiled binaries are used (for example ttyd), source locations are commented in the Dockerfile allowing all to verify that sources haven't been interfered with and have been accessed directly from the ttyd build sources.

With these being new images and a new version of Ansible, I'll be sending a course announcement for testing and these images will then roll out as the default in the new couple of weeks.

I appreciate everyone's patience with this and hope you're able to make use of these efforts in your own projects! If you do, please let me know.

@rkujawa as you rightly pointed out with your 3 cents (that I appreciate 😀), it is important when running containers that use privileged escalation that the sources can be verified and, if desired can be built independently. Given your area of expertise, it would be appreciated if you could cast your eyes over the sources I've shared and to confirm that this covers off the concerns that you raised.

Best Regards

James Spurin

[spurin]

tagging @logan-connolly @claudiospizzi @carlosdoliveira @bsahane @rkujawa @s2005 for reference

[rkujawa]

Thanks, will take a look!

[spurin]

@logan-connolly - Reading back through our discussion, I previously mentioned back in the August 2021 reply that I recommended to look at supervisord instead of systemd. Whilst this is still quite good advice in general, it's worth knowing that I had a eureka moment that brought stability to the use of these images across all platforms/os variations.

This change was running systemd in legacy mode on Debian based systems (i.e. Ubuntu). The CentOS systems always seemed to be a lot more stable and sometimes, where I'd see students encounter issues, the CentOS one would run happily whilst the Ubuntu ones exited.

For reference, here's the relevant lines of code with the fix in the parent container -

https://github.com/spurin/container-systemd/blob/ubuntu_20.04/Dockerfile#L1-L30

[logan-connolly]

@spurin Thank you for getting back to me and with such detail. Looking forward to digging into the details. Thanks for the legacy mode tip for systemd. I am sure that will save a lot of headaches!

https://github.com/spurin/container-systemd/blob/ubuntu_20.04/Dockerfile#L25-L28 is a really nifty trick. I like the little source code patch before installing! Will definitely keep this for reference.

[carlosdoliveira]

tagging @logan-connolly @claudiospizzi @carlosdoliveira @bsahane @rkujawa @s2005 for reference

I've never seen such hardworking and passion for something! Congratulations for the excellent content and support. Definitely not my studies will get to another level as my productivity as well!!

[spurin]

Hi All,

I've fully released this version of the lab now using the source code provided. Therefore, I'm going to close the issue. It's been an awesome ride doing this and I hope you all enjoy making your own super awesome labs using it!

If this comes in useful, please consider a nomination for me here - https://stars.github.com/nominate/ 🙏