Docker benefits

[efrecon]

Simply wanted to point at the following which makes use of your great tool. This technique is able to squeeze down Docker images to a minimum, while still benefiting from the giant software and package catalog in Debian (or derivatives).

Thanks for this. I didn't think it was worth a PR as it is merely using exodus, rather than extending it. But maybe migrating in the main Dockerfile is something that you would be interested in?

[sangaline]

@efrecon Sorry for the slow reply here, but thanks for sharing this. It's definitely very cool to see! I'm potentially open to adding a docker output format, but I would be curious to hear more about your use case for a container here over using the bundles directly. I also think that it would be more inline with the project philosophy to produce a docker image using the local machine's binaries rather than producing one from an external base. I definitely see the utility of an external package source, I just think that it might be beyond the current scope of the project. Were you imagining this more as something that the exodus executable outputs, or adding it as part of the documentation?

On a related note, I've added a couple examples to the README which involve adding tarball bundles to images. I like the simplicity of adding something like

ENV PATH="/opt/exodus/bin:${PATH}"
ADD exodus-jq-bundle.tgz /opt/

to an existing Dockerfile for the purpose of adding a binary to an existing docker image.

[efrecon]

My use case is to produce minimal images to arrange for quick startup while still having access to any apt-get install necessary for a given image. These can be handy in the serverless world.

The whole advantage of doing this from inside a Dockerfile is that you do not need to have access to the binaries on the local machine before squeezing them into the target image using exodus. You can benefit from the images that are already present at the hub instead and can be handy from a CI standpoint as you always start from a well-known state in the different phases necessary to build your final image.

I understand that this might be outside of the scope of the project.

[sangaline]

Yeah, that definitely makes a lot of sense. I just view the role of exodus to be taking locally functioning binaries and bundling them in such a way that they can be easily transported. It's really cool to make that "local" context be a reproducible docker image, but I don't think it would make sense for a Dockerfile that makes a choice about a base system be an output format of the tool itself at this stage.

[efrecon]

Actually, for theses kind of purposes, a standalone deb or apk release package of exodus would probably be enough for most purposes. It would allow anyone to easily add exodus, without all the python cruft, then pick binaries and libraries as I exemplified and later restart from scratch, all this from the dockerfile.

Don't read me wrong, I have nothing against python. But having a standalone (but os-dependant) package would ease when picking packages, as it would allow to start from any Debian (or alpine) based image, rather than from the python ones.

[sangaline]

I played around with this a bit out of curiosity and I was able to manually bundle exodus with the necessary Python dependencies down to 7.6 MB (exodus.zip). Converting this to a tgz, you can do something like

FROM debian:buster-slim

# Install the exodus bundle.
ADD exodus.tgz /

# Install `jq` and bundle it.
RUN apt-get update && \
    apt-get install -y jq && \
RUN /exodus/bin/exodus jq | /bin/bash

and it can sort of produce bundles, but you'll need bash in the image to use the shell script launchers that are created. You can alternatively install gcc and musl, but the gcc dependency is itself quite large. A Dockerfile like this will work without using the python base, but I'm not sure how much leaner it is (probably not much because pip pulls in the kitchen sink).

FROM debian:buster-slim

# Install exodus and its dependencies.
RUN apt-get update && \
    apt-get install -y python python-pip musl musl-tools gcc && \
    pip install exodus_bundler

# Install `jq` and bundle it.
RUN apt-get install -y jq && \
    exodus jq | /bin/bash

# Copy over the exodus bundles.
FROM scratch
COPY --from=0 /opt/exodus /opt/exodus

# Define the entrypoint.
ENTRYPOINT ["/opt/exodus/bin/jq"]

I could image eventually having a fully self-contained bundle that's produced as part of the build process, but that's definitely a ways off at this point. It would be really cool if exodus was "self-hosting" in that it could bundle itself, but there's a lot to do before even thinking about dealing with interpreted scripts.

[efrecon]

That's great! Depending on how much work this is, maybe is this just a matter of generating the exodus.zip (or tgz) at each release automatically?

"self-hosting" would be cool indeed!

[sangaline]

I had to do a fair bit of that bundling manually because there's not any built-in support for interpreted scripts or adding non-library files right now. I have some plans in place to improve the situation with those in the near future, and I'll revisit self-hosting after those are put in place. After that, it should be pretty easy to add the bundling to the build process and make a release tarball available at https://circleci.intoli.com/intoli/exodus/exodus.tgz or something like that.

[sangaline]

@efrecon I've added a build step to the beta version-2 branch for creating self-hosted bundles with the new format. Everything packs down to 5.3 MB in the end. I had to cheat a little bit by compiling an entry point that invokes python with the script arguments because you can't have relative path shebangs in scripts (without using awk, or some other external dependency).

You can do something like this now to install it in a container with python or pip.

FROM alpine

# Install the exodus bundle.
ADD https://292-119092220-gh.circle-artifacts.com/0/exodus-x64.tgz /
RUN tar --strip-components 1 -p -zxf exodus-x64.tgz

The bundle doesn't include musl/gcc though, so you'll only be able to use bundled programs if bash is available to interpret the launchers.

[sangaline]

Alright, version 2.0 is now officially released. The URL

https://circleci.intoli.com/artifacts/intoli/exodus/exodus-x64.tgz

will resolve to the latest successful build.

I'll leave this open for now, I think that the final piece here would be to have generic launchers compiled during packaging and then have them modified during bundling instead of depending on gcc. If that were the case, then it would be possible to use Alpine is the initial base and then copy Exodus bundles over to the next stage with no other dependencies.