RFC: Execution Environments

[hone]

Readable

[buildpack-bot]

Maintainers,

As you review this RFC please queue up issues to be created using the following commands:

/queue-issue <repo> "<title>" [labels]...
/unqueue-issue <uid>

Issues

(none)

[samj1912]

I'm having a difficulty trying to grasp how environment is being used by different toml files. @hone could you please provide examples of places where you imagine it being used and how that information will be leveraged by buildpacks/platform?

[loewenstein]

Frankly, I don't get this proposal aligned with my mental model of what buildpacks do and how they help.

For me, container images are just another software artefact nowadays produced to ship software. Like a while back one was to produce JAR and/or WAR files that got delivered and finally deployed into some execution environment. Buildpacks are a great tool to create those, but I don't see how or why I would create a test container that was separate from the production ones. I would want my tests to validate exactly the artefact that I am about to deliver. Otherwise, how do I know I checked the right thing.

From my POV buildpacks should focus on that task, creating the container image, and leave the other CI tasks to specialized tools.

Please help me to adjust my mental model, why's this added complexity in the buildpack spec worth it?

[jabrown85]

Please help me to adjust my mental model, why's this added complexity in the buildpack spec worth it?

I'm not the author but I can speak for my own mental model.

One use case is from the app developer side. Today they can pack build <img> to get a production artifact.

If we imagine the resulting image is ruby + nodejs, the developer has to setup both of those environments locally to develop and AGAIN on say GitHub Actions. The way those installations happen can differ. The GitHub action may install a different minor version of node or ruby to run the tests for instance.

Setting up CI with buildpacks could make this experience easier. pack test would use the same buildpacks and therefore test a more prod-like experience as the buildpack would install the same versions. This is especially important if your buildpack has options like DO_THIS_GARBAGE_COLLECTION_SETTING that you would have to know and replicate in your CI environment.

Another future-proofing thing here may be around future ARM support. Building and running an ARM test image via pack test seems tractable.

[kanaksinghal]

I have a use-case for this. In our company we use a variety of languages/stack and I wanted to write a generic CI pipeline where buildpacks can help us build the container image by detecting the stack but then to run unit test, I need to detect the stack again myself and run specific commands for nodejs/go/java etc. In a typical CI pipeline we would do unit-test, run sonar quality checks, and build container image.

[jama22]

Really excited for this proposal, and I think there's a lot of interesting ways to make use of it.

In the context of a self-hosted PaaS I've seen a common dynamic where the "operator" needs to establish fine grained controls over what the final containers look like. These configurations are also prone to change as the artifact itself moves into different environments (service connectors, certificates, environment variables, access to secrets managers, etc.)

On the developer side, most folks don't ever see production-like environments, so being able to debug their applications in production-like environments becomes incredibly difficult. In the past, I've seen some of these per-environment configurations being managed through GitOps and config files. This gives the user some control over managing the movement of the workload through the system, but even still they may not be able to build that container and interact with it directly

Moving the environment configuration opens up a lot of possibilities in this use case. I could image an operator encoding their per-environment configurations into the builder, and giving the developer the ability to simulate how their application will change in behavior across configurations.

For CI/CD I think there have been some interesting ideas already discussed above and in the proposal. But I think a common use case that i can come up with is if you're building a library with buildpacks, you'll want to test on multiple OS-distros and architecture combinations (e.g. x86 vs win vs arm, ubuntu vs debian vs rhel). Those env configurations are commonly configured in CI itself or possibly externalized to some other config. Moving them back into the buildpack may provide for an improved UX

Changing what "prod-like" means I think there's a pretty reasonable use case here where operators may want to give developers the ability to change what "prod-like" means. For example, the GCP buildpacks has GOOGLE_DEVMODE that can be enabled for faster changes. I can see it being used as a developer-oriented configuration while maintaining guardrails in prod-like envs

[cz4rny]

Two questions:

1. How would one extract artifacts from the test env? Running unit tests, or any other static-analysis tools produces outputs separate from the end OCI image. Currently pack performs the build in a tmp directory, which is cleaned out at the end. How would one access the results of all the checks done in the dev env?

2. How would I ensure, that the code produced in another pack built, to the production environment is tested? Having the CNB_EXEC_ENV set could introduce a completely different set of layers and hence a different output. So I would test the source code and the binary produced from the test environment, but that doesn't mean I've tested the exact same binary, source code, and artifacts produced in the production env.

[jabrown85]

Two questions:

1. How would one extract artifacts from the test env? Running unit tests, or any other static-analysis tools produces outputs separate from the end OCI image. Currently pack performs the build in a tmp directory, which is cleaned out at the end. How would one access the results of all the checks done in the dev env?

This RFC aims to only produce a test image artifact. A test platform would then execute the image and collect the results (TAP/json/etc).

2. How would I ensure, that the code produced in another pack built, to the production environment is tested? Having the CNB_EXEC_ENV set could introduce a completely different set of layers and hence a different output. So I would test the source code and the binary produced from the test environment, but that doesn't mean I've tested the exact same binary, source code, and artifacts produced in the production env.

There would be no such guarantee. Production images vary quite a bit from stack to stack and there is no one size fits all solution for testing. For a practical example, a ruby or node app will often have test environment specific dependencies that should not make their way into the final production images. Another example is a simple go app on a scratch/tiny stack. The go tooling itself, go test, should not be packaged into the final production image.

Thinking out loud, a test pipeline could grab the SBOM/digests/shasum of the things it cares about and compare them to the production image that was built.

[cz4rny]

Right, so right now the image produced is a self-executing app built, whether it's npm, or go or whatever.

The test image would not execute anything but would contain the app with all of the, keeping to the npm example, dev dependencies. Maybe it's even built. And the pipeline would use that image, run its tools on top of that (static analysis, test, etc.), grab the produced outputs, and publish them.

[jabrown85]

The way I understand this is the produced image would have test processes contributed by buildpacks.

A golang buildpack would create a layer that contains go testing tools and maybe even run a go build ./... during the build phase so you can fail early. The go buildpack could contribute test process to launch.toml that runs go test ./.... That would be the default process launched when the image is executed in a testing pipeline. As you said, the pipeline could also choose to run static analysis as well as any other process on the image (maybe lint in this example) and capture the results.

[keskad]

Hi,

I wanted to comment the concept with a use-case. So, I don't see the point in creating a production images in environment X, while testing in environment Y. That's not consistent way of doing CD pipelines (current behavior) :slightly_smiling_face:

I give buildpacks to teams, they set e.g. BP_NODE_VERSION 16.1 and it will execute with Node 16.1, while in tests it will still execute with hardcoded Node 16.0 because I have to maintain additional images. Using buildpacks in this scenario is loosing sense for me :slightly_smiling_face: More consistent way is to just use the Dockerfile with base image used in tests, because it has consistent Maven and Java version inside.

Due to nature of buildpacks - preparing the environment (by installing required tooling, setting up those tools) - I think the test phase is even a neccesary scenario to implement a valid CD pipeline in order to gain reproducible environment everytime. The tests needs to be running on something that is CLOSE TO PRODUCTION :slightly_smiling_face:

So I see that build phase even as neccessary to implement a valid CD pipeline, so I like this proposal.

The case is similar to the principle, where we should not build artifact twice if promoting to next environment e.g. dev -> prod, but just reuse the artifact. There we should use the same build parameters, same tooling.

Its difficult to maitain e.g. Node 14, Node 15, Java 8, Java 11, and 15 other base images just for tests, while giving teams a possibility to set a Node or Java version in buildpacks.

A testing phase in my opinion would benefit in:

• not maintaing the base images internally just for running tests (in bigger organizations its a huge effort)
• have a consistent way of setting up both test & prod tooling (versions and parameters - e.g. team sets Node v16.1, then it is working in both test & prod)
• forget about testing command for each project (buildpack test phase would know a testing command for each technology just as it knows how to run the build)