Open eeyun opened 6 years ago
I'm reconfiguring this issue into a tracking Issue as having this card in analysis revealed more facets to the issue and this should be considered an Epic.
I've started some POC work on this, https://github.com/habitat-sh/habitat/pull/5605 - Please, comments and criticism. Lets get it done!
Until bldr has a more thorough featureset around integration testing, the core-plans maintainers should adopt pre-commit tooling that can tell use if version bumps could have a negative effect on the runtime behavior of a reverse dependency.
Example: Builder today can tell us if a package build breaks the compilation of another package. But, it cannot tell us if another compiled package will actually function.
E.G.
core/web_app_srv
setspkg_deps = ( core/web_app_lib )
Let's pretend a change comes in that updates
core/web_app_lib
. What we need to know isn't just whethercore/web_app_srv
can build, but if it can actually still start up like a service should after such a change.There are several challenges here. Reasonable testing of both services and libraries is not a ubiquitous operation. If a library changes we need to know both if a consuming package can compile and then whether a consuming compiled package will still start and run and perform it's typical operations. If a service changes, we simply need to know whether or not the service itself still functions.
My theory is that in defining runtime-level integration tests for services, we should be able to chain testing pipelines via dependencies to determine whether a theoretical low-level library change is going to break these runtime operations. On a long enough timeline we will hopefully support many of these features natively in builder. For now though, our users can't wait for that to happen.