Functional Testing Pipeline

[eeyun]

Until bldr has a more thorough featureset around integration testing, the core-plans maintainers should adopt pre-commit tooling that can tell use if version bumps could have a negative effect on the runtime behavior of a reverse dependency.

Example: Builder today can tell us if a package build breaks the compilation of another package. But, it cannot tell us if another compiled package will actually function.

E.G. core/web_app_srv sets pkg_deps = ( core/web_app_lib )

Let's pretend a change comes in that updates core/web_app_lib. What we need to know isn't just whether core/web_app_srv can build, but if it can actually still start up like a service should after such a change.

There are several challenges here. Reasonable testing of both services and libraries is not a ubiquitous operation. If a library changes we need to know both if a consuming package can compile and then whether a consuming compiled package will still start and run and perform it's typical operations. If a service changes, we simply need to know whether or not the service itself still functions.

My theory is that in defining runtime-level integration tests for services, we should be able to chain testing pipelines via dependencies to determine whether a theoretical low-level library change is going to break these runtime operations. On a long enough timeline we will hopefully support many of these features natively in builder. For now though, our users can't wait for that to happen.

[eeyun]

I'm reconfiguring this issue into a tracking Issue as having this card in analysis revealed more facets to the issue and this should be considered an Epic.

[predominant]

I've started some POC work on this, https://github.com/habitat-sh/habitat/pull/5605 - Please, comments and criticism. Lets get it done!