self-hosted build process for images

jfgibbins commented 7 years ago

I don't know how many people would want to be able to do this, and I'm sure a few have already done it on their own, but I personally would love to be able to run my own image server/jenkins/etc process like you run for images.linuxcontainers.org. Sadly, my ubuntu-fu is a bit weak on that subject as I'm just a poor dumb network guy. My goal would be to customize the process to tweak the images for my own environment. While I know that it's possible to pull an image, launch, modify, stop, and publish, it really lacks the elegance and automation that devops practice pushes for. Would it be possible, in the future, for you to publish an article or two, similar to when you did your lxd posts last year, on the basic process and flow. It appears all the code to do it, is available through the links on your website to GitHub, it's just a matter of understanding how it goes together. Anybody who's familiar, any help in learning would be greatly appreciated.

jfgibbins commented 7 years ago

Tried again reading through some of the repositories. Particularly in terms of using jenkins and figuring out how it goes, though noticed that then .jenkins file is in the .gitignore so couldn't read those. The effort cost me a few Excedrins.

stgraber commented 7 years ago

The way our images are built is roughly:

Daily run of templates/lxc-ubuntu (from github.com/lxc/lxc) using bin/build-image (from github.com/lxc/lxc-ci). That results in a number of artifacts that you can see on Jenkins: https://jenkins.linuxcontainers.org/job/lxc-template-ubuntu/
A separate server downloads those artifacts, generates a tiny metadata tarball for LXD with a metadata.yaml file based on the Jenkins metadata and includes some override files too. This server then sets up a simplestreams-compatible filesystem tree and makes it available over rsync.
Frontend servers then sync over rsync on a fixed interval which then becomes what you see at https://images.linuxcontainers.org

Not exactly something easy to replicate or that I'd even recommend to attempt to replicate. Our build pipeline is much more complex than what you'd want locally since we need to support all Linux distributions, all architectures, both lxc-download and LXD and both the native LXD protocol and simplestreams. So for every image we generate at least 4 set of metadata, additional build artifacts, gpg signatures, ... a lot of which you wouldn't need for a local image build system.

jfgibbins commented 7 years ago

No wonder I get a headache when I try to decipher it.

stgraber commented 7 years ago

Yeah, I have a vague plan to completely rework the template system in LXC for 3.0, moving it to something out of the LXC code tree entirely and moving from the current complex shell scripts to something more declarative.

A lot of those templates also include a lot of legacy stuff. Nowadays most Linux distributions build official tarballs for use in the cloud or Docker containers, those could be ingested pretty much as they are, just stripping some bits and adding some more bits, without having to bootstrap everything from scratch.

So hopefully we can move to a place where the vast majority of distribution images are effectively just repack of an official image with some bits added/removed and then the needed metadata for LXC/LXD added on top of that. Then the few distros that remain will have something close to what they have today.

jfgibbins commented 7 years ago

That sounds like it'd be a lot less work for you guys to maintain. And if it's something we can setup as well, then tweak to build our own images, that would be even better. For example here at least, images, with puppet-agent pre-installed, lxd uninstalled(no offense, I hate when I generate certificates because I do lxc list, when I'm shelled in, doh!), stuff like that. Speed up container spin-up. Sadly, puppet-agent alone eats up 4-5 mins, if not more when spinning up several simultaneously.

vStone commented 7 years ago

Maybe somebody can add support to packer to also allow creating images from a rootfs for lxd. I'm current contemplating writing some scripts around the old lxc tools to add lxd metadata...

kaneg commented 6 years ago

Some ideas to share: I use Ansible playbook to build containers and images from base image. Use reusable playbook roles to control different features for different purposes.

jochumdev commented 6 years ago

I use LXD as image server and manualy prepare these images with Saltstack and partialy with other stuff, then I use lxc publish to create an image from the container.

stgraber commented 6 years ago

Going to close this issue in favor of https://github.com/lxc/lxc/issues/2072 which tracks the development of a new unified tool to generate LXC and LXD images, this will allow you to easily build custom images for any distro that we publish images for.

canonical / lxd

self-hosted build process for images #3724