search

hercules-ci / arion

Run docker-compose with help from Nix/NixOS
Apache License 2.0
622 stars 48 forks source link

Declarative configuration of supporting operations (using OCI hooks) #218

Open deliciouslytyped opened 10 months ago

deliciouslytyped commented 10 months ago

I'm interested in executing code on container bringup and takedown (for example to snapshot volumes). I've taken some notes below on what I found on OCI hooks, and the question is whether there is a good way this could be handled by Arion.

I happen to be using podman, which seems to support OCI hooks, which appears to be the correct way to do something like this.

It may actually be more appropriate to ask whether podman-compose should be handling this with possibly custom compose attributes, and until it does, how it could be done with Arion.

My initial idea is that as long as --hooks-dir can be passed through to podman, maybe this should work.

@roberth do you have any thoughts?

Notes: OCI hooks seem to be specified in https://github.com/opencontainers/runtime-spec/blob/main/config.md#user-content-posix-platform-hooks , and the podman option seems to be explained in https://docs.podman.io/en/latest/markdown/podman.1.html#hooks-dir-path :

--hooks-dir=path

Each *.json file in the path configures a hook for Podman containers.
For more details on the syntax of the JSON files and the semantics of
hook injection, see oci-hooks(5). Podman and libpod currently support
both the 1.0.0 and 0.1.0 hook schemas, although the 0.1.0 schema is deprecated.

This option may be set multiple times; paths from later options have
higher precedence (oci-hooks(5) discusses directory precedence).

For the annotation conditions, libpod uses any annotations set in
the generated OCI configuration.

For the bind-mount conditions, only mounts explicitly requested by the
caller via --volume are considered. Bind mounts that libpod inserts by
default (e.g. /dev/shm) are not considered.

If --hooks-dir is unset for root callers, Podman and libpod currently default
to /usr/share/containers/oci/hooks.d and /etc/containers/oci/hooks.d in
order of increasing precedence. Using these defaults is deprecated. Migrate
to explicitly setting --hooks-dir.

Podman and libpod currently support an additional precreate state which is 
called before the runtime’s create operation. Unlike the other stages, which 
receive the container state on their standard input, precreate hooks receive
the proposed runtime configuration on their standard input. They may alter
that configuration as they see fit, and write the altered form to their standard
output.

WARNING: the precreate hook allows powerful changes to occur, such as
adding additional mounts to the runtime configuration. That power also
makes it easy to break things. Before reporting libpod errors, try running
a container with precreate hooks disabled to see if the problem is due to
one of the hooks.

oci-hooks(5) appears to be a documentation file in the podman common repo: https://github.com/containers/common/blob/main/pkg/hooks/README.md https://github.com/containers/common/blob/main/pkg/hooks/docs/oci-hooks.5.md

EDIT: Note: the oci-hooks(5) config seems to differ from the hook format given in runtime-spec.

The library implementing the hooks for podman appears to be documented at https://pkg.go.dev/github.com/containers/podman/v3/pkg/hooks#section-readme

Not all runtimes appear to support hooks, for example crio does and containerd will (but does not yet):

Given these, I'm not actually sure if it's the runtime behind podman supporting the hooks in podmans case, or podman itself. Going by the podman documentation, I get the impression that it's podman. But then does that mean hooks are possibly handled twice, both by podman and the runtime?

Here are some example hooks from a completely unrelated system: https://sarus.readthedocs.io/en/stable/config/configure_hooks.html

There are possibly other solutions such as dealing with systemd or filtering podman events, ( https://www.reddit.com/r/podman/comments/v6adqi/is_there_any_way_to_subscribe_to_podman_events/ ) but hooks appear to be the most correct solution. (or for example The alternative for runc hook users is to proxy or replace runc with their own version of runc, which is not a good solution. mentioned in the containerd issue.)

This may or may not be related to https://github.com/hercules-ci/arion/issues/48 . I'm not sure if the discussion there implies that this won't work with docker without something like https://github.com/awslabs/oci-add-hooks .

Some other misc links:

deliciouslytyped commented 10 months ago

podman-compose appears to implement some flag passing in https://github.com/containers/podman-compose/issues/278 but on a very superficial look it sounds a bit janky.

There is some consideration for extending the compose specification:

Docker appears to have something against the idea of lifecycle hooks because it's bad for portability (and security? - I'm somewhat skeptical this isn't a moot point.) or something. compose-spec seems to be a repo managed by docker, which is not a good sign for having lifecycle management in the spec.

There is also another idea of init containers (which may be extendable to lifecycle managing containers? - I haven't read the page yet):

deliciouslytyped commented 10 months ago

It may or may not be a problem that https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks requires paths to be absolute. (Why?)