Design: Think about how to "lock in" external artifacts at build time

[jeremyrickard]

👵 Picture it! Colorado, 2019. A bundle author wants to use the Helm mixin with a stable chart or the Kubernetes mixin with some manifests on the internet. As a user of Porter, they'd like to have those files sucked into the invocation image so all the goodness of signed bundle and digested invocation images apply to the manifests as well.

Currently, at build time we don't pass the YAML of the step down to the mixin. We should think about how we can pass all the relevant step YAML down so that the mixins could be smart enough to go and fetch the external resources. There are other related topics..should the mixin re-write the porter.yaml to reflect those inside the bundle? Should fetching the stuff be the default behavior?

[carolynvs]

The action commands pass in the yaml step data. So one idea would be to do something similar for build. When a mixin build is called, porter looks for all the steps for that mixin and passes them in on stdin.

For example, for this manifest:

mixins:
- helm

install:
- helm:
  ...
- exec:
  ...

upgrade:
- helm:
  ...

The mixin would get the following on stdin for helm build:

install:
- helm:
  ...

upgrade:
- helm:
  ...

[carolynvs]

For the "lock in" part, I can see a dev wanting to not lock in by default when they are iterating on the bundle (because who gets it right the first time?). Then when they are ready to publish, that's when locking it in would be helpful.

So maybe a workflow like this would be nice:

$ porter build
<do what we do today, for example if a helm command needs a chart, it won't be built into the bundle>

$ porter build --embed
<pass this flag onto the mixin during the build stage, so that the mixin can look for resources that should be embedded into the bundle at build time>

Alternatively, once we get to the point that build doesn't push to a registry anymore because we are using buildkit, that can open up a new workflow for us. Use the act of pushing as intent to share a final "good" bundle. Then at that point, porter does a final build that does extra best practices (at some point we do have opinions right?).

$ porter build
<same as today, but without pushing because buildkit or other reasons>

$ porter publish deislabs/offical-bundle-name
<rebuild package following best practices for distributing a bundle, then  publish it to an official location>

I think we could do the --embed flag first, and then introduce a publish flag later, making the manual build --embed step unnecessary.

[carolynvs]

We spoke about this offline and want to make --embed a flag to the mixins, but not exposed in porter itself right now. Maybe we will expose it later, but we don't want to do that immediately.