search

edannenberg / kubler

A generic, extendable build orchestrator.
BSD 2-Clause "Simplified" License
158 stars 40 forks source link

gentoobb Unnecessary Dependencies / Ability to Compose #73

Open berney opened 7 years ago

berney commented 7 years ago

I know the repos in gentoobb namespace's purpose are to server as examples, but I've noticed unnecessary dependencies, which are triggering my OCD. I'm learning S6 (especially in Docker) and came across these whilst looking for examples of how it was used: -

image unnecessary dependency
docker-registry openssl -> s6
grafana openssl -> s6
nodejs openssl -> s6
opensmtpd openssl -> s6

At least some are probably using OpenSSL but not S6, which gentoobb/openssl depends on.

If I understand correctly the current design is Single Inheritance and doesn't support Multiple Inheritance nor Mixins, or similar concepts. One of the strengths I like about gentoo-bb is the ability to easily build minimal images. It would be cool IMO if some sort of composing could be done. I think composing would allow inheriting from say both gentoo/nginx using both gentoo/s6 and gentoobb/openssl (and gentoobb/openssl depends on gentoobb/glibc rather than gentoobb/s6).

jbergstroem commented 7 years ago

I think using s6 is a personal preference. In general, docker images are usually single process entrypoints (and exits), meaning docker itself should manage processes. This also makes more sense as you scale up the amount of deployed containers and how they interconnect with orchestration like k8s.

jbergstroem commented 7 years ago

I think with current documentation and learning curve (what is gentoo, portage, builder containers, etc); people will build upon examples instead of creating something from scratch. I'm not sure how to best "break free" of this other than adding more examples or somehow abstracting containers further.

edannenberg commented 7 years ago

At least some are probably using OpenSSL but not S6, which gentoobb/openssl depends on.

S6 adds about 1.4MB to the image size so I went with flexibility. For example you get proper cron support in every image that extends from S6 (the actual crond is provided by busybox). If your OCD doesn't allow that you will have to create your own image stack. :stuck_out_tongue:

If I understand correctly the current design is Single Inheritance and doesn't support Multiple Inheritance nor Mixins, or similar concepts.

That's a Docker limitation, you can only have one parent image.

It would be cool IMO if some sort of composing could be done. I think composing would allow inheriting from say both gentoo/nginx using both gentoo/s6 and gentoobb/openssl (and gentoobb/openssl depends on gentoobb/glibc rather than gentoobb/s6)

See above, but once acbuild support is in you will have more options: https://github.com/containers/build/blob/master/Documentation/subcommands/dependency.md. Come to think of it, this will also have implications for image dependency/build order resolution.

I think with current documentation and learning curve (what is gentoo, portage, builder containers, etc); people will build upon examples instead of creating something from scratch. I'm not sure how to best "break free" of this other than adding more examples or somehow abstracting containers further.

All we can do is provide better documentation and good best practice examples in the reference namespace. People that care will build there own stack from scratch and the rest should be fine building upon the provided images.

@berney As you mentioned learning S6, you might want to take a look at s6-overlay. They are doing things a little bit differently with direct involvement from the S6 creator, didn't have the time yet to port this to gentoo-bb. For example they have the concept of a main service for each image, if that service fails the container will also fail.

edannenberg commented 7 years ago

I guess we could massage the example images a bit to be symlink friendly. This would allow image reuse in other namespaces. Example:

mycustomns/
  images/
    openssl/ <- custom openssl that does not extend from s6
    grafana/ <- symlinked from gentoobb/ but as there is no hardcoded namespace,
                this produces mycustomns/grafana image extending from mycustomns/openssl

This example would obv. not work for images that use s6, but could be useful in other cases I suppose.

berney commented 7 years ago

I don't know much about ACI or acbuild at the moment, but I see it will allow many dependencies, so this seems like it will provide what I was thinking of in the 1st post.

@edannenberg your reasoning for adding S6, with the cron and entr support makes sense. I am being pedantic. I have started to create my own stack. The symlink idea isn't bad. I've also been looking at https://github.com/just-containers/s6-overlay and the S6 documentation and experimenting inside containers, I think I get the concepts now and I've nearly worked out my image.

@jbergstroem thanks for your input as well. It's true S6 is a personal preference. The the proponents have convinced me it's a good idea, I'm not using k8s or similar yet, but I'll keep it in mind.

I like gentoo but a lot of people don't like it. I haven't tested it but I believe gentoo-bb will work on other distros with Docker support, so it's only used in the containers. Those not familiar with gentoo may find they need to learn more gentoo to get things done. I like the "stack" as it gives a great deal of flexibility, performance and minimalism. For instance for learning purposes I got a figlet-musl image without baselayout. I couldn't get a single static binary figlet, there's no static USE flag so I tried CFLAGs etc but still got dynamically linked binaries, I think due to the Makefile, it was just a learning exercise so I didn't keep pursuing it.

I think it will be good to try to ensure the examples are implementing best practices. Such as running unprivileged when possible. IIUC (I haven't tested yet) many images, such as the nginx based ones, could be run as an unprivileged user if they are configured to not listen on privileged ports (e.g. 8080 rather than 80). It's a personal preference but I like to use a load-balancer in front of containers. IIUC in most cases only the docker host needs privileges when setting up the port forwarding.

edit: Docker has deprecated MAINTAINER <maintainer> for LABEL maintainer <maintainer>.

I think getting over time improving the ease of developing, readability, and maintainability these repos in gentoo-bb and getting a larger library of high quality example images that follow best practices and have unique or "best-of-breed" attributes, weather it's support for something that else where doesn't publicly exist, more minimal, more performant, more secure, more functionality, more easy-of-deployment or ease-of-use etc. will help make gentoo-bb as good as can be. I think this is already the goal, and I'm just stating the obvious.

Along these lines of improving the ease of developing, readability, and maintainability I prefer YAML, over say JSON or Shell Scripting. I'm not show how feasible or desirable to you guys it is to move to configuring via YAML and then either inlined Shell or shell scripts when the YAML struct doesn't support something. I was thinking either build.conf, build.sh, Dockerfile.template replaced with YAML versions, or one big YAML file to do all the typical tasks.

edannenberg commented 7 years ago

I don't know much about ACI or acbuild at the moment, but I see it will allow many dependencies, so this seems like it will provide what I was thinking of in the 1st post.

Yep, IMAGE_PARENT="gentoobb/s6 gentoobb/openssl gentoobb/gcc" will then add those images as parents in listed order, each of these will have just gentoobb/glibc as parent. Well at least that's the plan. :)

I am being pedantic.

No worries, I remember having the same discussions with @jbergstroem back then. :stuck_out_tongue:

I think only the php images actually have a good reason for s6, everything else uses s6 just to be prepared for possible cron/fs watcher usage. Which in practice I only ever used twice I think. Maybe this should be revisited.

I've also been looking at https://github.com/just-containers/s6-overlay and the S6 documentation and experimenting inside containers, I think I get the concepts now and I've nearly worked out my image.

I haven't looked too deep into s6-overlay yet, but a lot of things made sense and given skarnet's involvement it's most likely the best setup. Should probably be enough to copy the s6 config from there with some modifications. If you do adapt it I'll happily accept a PR. ;)

I like gentoo but a lot of people don't like it.

One of the reasons I'd like to change the project name, Gentoo is really just an implementation detail.

I haven't tested it but I believe gentoo-bb will work on other distros with Docker support, so it's only used in the containers.

Yep, Docker, Bash and a couple of basics like sed are the only requirements on the build host. Has been recently tested on CoreOS and Mac. I might test building on Windows for shits and giggles some day.

Those not familiar with gentoo may find they need to learn more gentoo to get things done.

Yea, the documentation needs a Portage 101, everything else doesn't really matter that much. Maybe explain some terms like stage3, etc. It's on the todo list.

I think it will be good to try to ensure the examples are implementing best practices. Such as running unprivileged when possible. IIUC (I haven't tested yet) many images, such as the nginx based ones, could be run as an unprivileged user if they are configured to not listen on privileged ports (e.g. 8080 rather than 80).

Absolutely, I usually look up the corresponding reference image from official Dockerhub library before I add a new image, though that has been 2 years ago for most images. Always open for improvements. In the case of Nginx it's probably fine though as Nginx itself drops privileges for its worker threads.

Along these lines of improving the ease of developing, readability, and maintainability I prefer YAML, over say JSON or Shell Scripting. I'm not show how feasible or desirable to you guys it is to move to configuring via YAML and then either inlined Shell or shell scripts when the YAML struct doesn't support something. I was thinking either build.conf, build.sh, Dockerfile.template replaced with YAML versions, or one big YAML file to do all the typical tasks.

It's not really worth it just for build.conf IMO and I'm not sure how I would define the hooks of build.sh via YAML. Not to mention the parsing-YAML-with-bash can of worms. Let's keep it simple and flexible.

berney commented 7 years ago

I got the figlet-musl-static image to work. I went full ricer . It's now very minimal, at 666kB vs 11.38MB (5.7% the size 🚗 🚗 🚗):

edit: I updated the links to my new gentoo-bb-images repo.

I added an option to bob-core/build-root.sh to skip making /var, /run and /var/run. I've sent you a PR #76.

Has been recently tested on CoreOS and Mac

Cool. I thought it should just work.

I might test building on Windows for shits and giggles some day.

That would be interesting. Actually something like this but for Windows-based containers on Windows would be interesting. The Linux containers on Windows should be very much the same, especially if you can use the Linux subsystem for Bash.

I'll get back to the S6 stuff, once I know what I'm doing I'd be happy to send any PRs I think are warranted or at least worth of consideration.

It's not really worth it just for build.conf IMO and I'm not sure how I would define the hooks of build.sh via YAML. Not to mention the parsing-YAML-with-bash can of worms. Let's keep it simple and flexible.

Good points, especially parsing YAML since you are using Bash.

jbergstroem commented 7 years ago

@edannenberg said: I think only the php images actually have a good reason for s6, everything else uses s6 just to be prepared for possible cron/fs watcher usage. Which in practice I only ever used twice I think. Maybe this should be revisited.

I personally have a FaaS-approach to cron-job on top of k8s. There's a few projects for that but I just spawn an instance periodically.

@berney said: I got the figlet-musl-static image to work. I went full ricer . It's now very minimal, at 666kB vs 11.38MB (5.7% the size 🚗 🚗 🚗)

Nice. One of my goals with this project is to provide an alternative to alpine linux for docker images. Using musl and even going static is pretty much required.

I personally think that a few good examples of gentoo-bb (pending name change) would be images that already are popular built on top of alpine; for instance nginx (adding a third party module that is popular but lacking in alpine) or mariadb (switching memory allocator or some other gentoo-specific feature) and being as-competetive in resulting container size.

I also think that the users of the resulting containers will outweigh the users that create them; meaning we should try and maintain these examples; set up jobs over at travis and/or circleci to periodically rebuild as new versions are released.

Edit: at the end of the day this will likely be a strong showcase for the strong points of Gentoo, but most users won't care. If we can provide quick and easy ways to create your own, custom, super-small-and-easy-to-use containers and perhaps even maintain a few as a project (ref above); we're onto something.

berney commented 7 years ago

Nice. One of my goals with this project is to provide an alternative to alpine linux for docker images. Using musl and even going static is pretty much required.

Same. I think Alpine is a good base, but sometimes you want more minimal, or a feature they don't have etc. Having the power/flexibility of gentoo and gentoo-bb split of build-time and run-time dependencies is so awesome.

I personally think that a few good examples of gentoo-bb (pending name change) would be images that already are popular built on top of alpine; for instance nginx (adding a third party module that is popular but lacking in alpine) or mariadb (switching memory allocator or some other gentoo-specific feature) and being as-competetive in resulting container size.

I plan to publish most of the images I make, and I plan to try to make them as minimal, secure, and performant as I can.

I also think that the users of the resulting containers will outweigh the users that create them; meaning we should try and maintain these examples; set up jobs over at travis and/or circleci to periodically rebuild as new versions are released.

This is a good point, the users of the images could be very high. I think gentoo-bb has the potential to be the tool that's creating the best images available.

As you pointed out, creating some high quality images will need good QA to achieve it. This is in addition to QA of gentoo-bb. We should develop not just the build.conf, build.sh, Dockerfile.template, needed to build the image but also README.md documentation, testcases and the like to ensure the builds are working correctly and there's no regressions. This will help catch bugs in gentoo-bb as well as bugs in the image.

I for one would like to have good QA of my images. I'd like to know they are working properly especially after I've refactored things in them. Also testing a stack of images would also be good, when they use services of each other etc. I'm curious what tools would be suited for testing minimal containers, some tools may expect full package managers or certain tools to be available in the container, such designed tools probably aren't a good fit.

Also I've been bitten in the past by Gentoo where an upgrade of a package (atom) compiled (emerged) fine, but the resultant binaries didn't work properly. For infra, it's good to add automated testing that tests theses things.

azimut commented 6 years ago

The idea I got for compose was an/other abstraction file where we reference other images and have hooks on those alien stages if needed to add/undo something.

For example, If I want an image for znc.

- from: glibc
- from: openssl
- from: python2
  after_configure_bob_stage: |
    update_use dev-python -sqlite
- package: net-irc/znc

So some kind of abstraction that will generate the new build process. Might be a new big build.sh with all the steps stacked.

Advantages for me are:

Of course I think might be I am reinventing package sets, and yaml is might be not the best approach of a poor's man dsl. That said, I don't need something like this atm, so I haven't think about how to do it, but someday might be.