Idea: Configurable sidecars for things like test fixtures

[technosophos]

It would be nice to be able to attach an arbitrary sidecar to a job so that a user can do things like set up a fixture database for running tests against. They could share a mount point (like an emptydir) ant the networking would be really easy, too.

[krancour]

Huge plus one on this. I'd very much like to move OSBA to Brigade, but cannot until this is possible.

[technosophos]

Just to be clear... you can usually do this synchronously by first creating the mock in one job, and then testing against it in another. But it would definitely be cool to do both in one step

[krancour]

@techosophos, do you have an example of that somewhere? Isn't a step a job? i.e. "short-running?"

[technosophos]

One option is to do a helm install and install your service, later doing a helm delete.

[krancour]

@technosophos you know my affinity for simplicity. Compared to a sidecar, using Helm to satisfy a test's dependencies feels like overkill and seems like it could add some complications. Worse, it undermines something I love about Brigade, which is that even if it's built on top of k8s, k8s isn't in my face all the time. I think a brigade.js file is totally comprehensible even to someone who doesn't know k8s, and imho, that makes it a very good general-purpose CI platform. If I had to Helm-install test dependencies, it puts k8s front-and-center in my pipeline when it doesn't otherwise need to be.

[technosophos]

Well, we will get to this feature at some point. I was just trying to give you a way to get started right away.

I'm not totally sure I understand the objection. I mean, it's not exactly opaque to do:

events.on("whatever", () => {
  setup = new Job("setup", "helm:latest", ["helm install -n mysql stable/mysql"])
  test = new Job('test", "mytests:latest")
  teardown = new Job("teardwon", "helm:latest", ["helm delete --purge mysql"])

  Groups.runEach([setup, test, teardown])
})

But I can definitely see the draw of using a sidecar, especially if that makes it possible to run both containers in the same pod. That's the kind of thing we do want to be able to do... it just requires that we right the code to do it. ;-)

[radu-matei]

I've been going back and forth on this issue for a couple of weeks now. The main question I have is about pod lifetime, and how containers with different lifecycles might affect the Brigade build.

Consider the following Brigade job:

• container 1: some integration tests for my web application - lifecycle of this will be the duration of the tests
• container 2: a database - lifecycle for this is indefinite

Since we want to run these in the same pod, after container 1 exits (regardless of the exit code), container 2 will continue to run, meaning the job (thus the build) never finishes. Is there a way to alter this behaviour in Kubernetes? (I don't think so).

If not, when using sidecar jobs, we could add another sidecar that monitors the main container, and kills the sidecars when it exits - however, this would impact the final status of the pod (succeeded vs. failed).

Any thoughts on the approach here? Thanks!

[technosophos]

I think that for an initial go, we could require the developer to figure out how to stop their sidecar, and basically document very clearly that sidecars will continue running until they shut down.

We could make this controllable via Brigadier, and allow programmatic access to sidecars:

brigadier.getSidecar(sidecarName). stop()

It'll be a little lopsided, since it will provide ways of killing, but not of creating, sidecars. And it also imposes knowledge of the sidecar on the script... and I don't know if I like that.

Otherwise, I think @radu-matei 's suggestion of a monitor sidecar is our next best bet.

[krancour]

It's been a while since I last commented on this, so my recollection of the circumstances behind those comments is murky, but I'm going to try to offer some new insight here...

Imagine being on a sailboat that capsizes in the middle of the ocean and debris is floating around everywhere and you're going to drown unless you get creative. If you can lash together enough floating debris with ropes from the boat's rigging, you can construct a makeshift raft-- and this is how I conceptualize a test environment. It's a bunch of containers "lashed together" in a makeshift fashion to support your tests. This is how I get this done in Circle and in Jenkins.

Circle and Jenkins each allow me to approach the problem a little differently. In Jenkins, I lash together one big raft at the beginning of the pipeline and ride it all the way through. I don't prefer this approach because it's a lot harder to reason about. It's harder to imagine / ensure individual steps in the pipeline as being isolated from one another.

In Circle, I can lash together smaller rafts that live for the duration of a single step.

I don't know Brigade well enough to say how well this raft analogy translates, but I thought it might be useful to conceptualize the test environment in this way.

[krancour]

This is actually really hard to do in Kubernetes right now, but there's a KEP open to straighten that out.

Here are related issues:

• https://github.com/kubernetes/kubernetes/issues/25908
• https://github.com/kubernetes/enhancements/issues/753

[vdice]

Removing the v1.1 label from this issue.

[krancour]

This is well covered by the 2.0 proposal, which has been ratified and is now guiding the 2.0 development effort. It probably doesn't make sense to track this as a discrete issue anymore.