Support multiple specs

[cpuguy83]

Right now we can only build one spec at a time, which works well enough for simple cases but is not ideal for multi-component images as it requires multiple build requests.

Options

The yaml parser supports multiple documents in one file (since this is part of the yaml spec). Example:

name: my-package1
# other fields

---

name: my-package2
# other fields

Alternatively we could change the spec such that the spec file a list of specs instead of a single flat spec. Example:

specs:
  - name: my-package1
    # other fields
  - name: my-package2
    # other fields

The advantage of this approach is specs can share a single set of build arguments and potentially other things (via yaml anchors) whereas in the yaml multi-doc version these are completely separate documents that can't share anything (per yaml spec).

I think overall I prefer the simplicity of option 1, meanwhile if needed we could add support for option 2 later without breaking anything:

type Project struct {
  *Spec
  Specs []*Spec
}

Build UX

When multiple docs are included this would add an additional namespace to all build targets. As an example, today we have build targets like mariner2/rpm, mariner2/toolkitroot, mariner2/container. With multi-doc this would add the package name as a prefix to each target, e.g. my-package1/mariner2/rpm. All existing semantics regarding default targets would remain... e.g. mariner2/container is the default target when the user does not specify one, with multi-doc it a target of docker build --target=my-package1 would default to my-package1/mariner2/container.

When a user does not specify any target (docker build .) this would need to select the last package specified. This is similar to how the dockerfile frontend works (last build stage in the dockerfile is used as the default).

Critical to making this useful: the spec for a package should be able to list one of the other packages as a build or runtime dependency, which would trigger a build of that package.

Example:

name: my-package1
# other fields

---

name: my-package2
dependencies:
  runtime:
    my-package1:

When executed with docker build --target=my-package2/mariner2/container (or no --target since it is last and as such the default), this should produce a container with both my-package2 and my-package1 installed and my-package1 would be built as part of the build invocation rather than installed from some repository.

This allows us to have inter-package dependencies without needing to make them available in a repository first.

[adamperlin]

So, just to be clear, this would be happening in a single container? In this case, our build graph would be: my-package1 => my-package2, so we would create our base container where we first build (and install) a package for my-package1, then run the build and install for my-package2?

[cpuguy83]

name: foo
# all the other things

---

name: bar
dependencies:
  runtime:
    foo:

So in this case the bar package does not need foo in order to be built, but it does need it when its installed. When creating the container image we will see we need to build foo in order to install bar so that will go and run (this would probably happen in parallel with building bar).

Likewise we can do something similar for build dependencies.

name: libfoo-dev
# all the things

---

name: foo
dependencies:
  build:
    libfoo-dev:

In this case we'd see that in order to build foo we need libfoo-dev which is in the list of packages provided in the spec file, so we go and build that.