Sidecar container for Beam SDK

[tweise]

The proposal to support Beam Python applications with the Flink k8s operator requires the addition of a sidecar container for the Beam SDK worker pool to the Flink task manager pod.

The sidecar container needs to be specified with image, resources and entry point, but generally it is just another container that has access to external resources similar to the task manager container.

Since there isn't anything Beam specific per se, should the CRD offer something like a taskManagerSideCar section that could also be used for other containers the user may wish to run alongside a task manager in the same pod?

[mwylde]

My preference is that we do make it beam-specific rather than generic. There are already mechanisms available for users to inject generic side cars with whatever logic they want (i.e., mutating admission webhooks, which at lyft we use to inject various ops sidecars). In general, I think it's hard to make the CRD flexible enough to satisfy all use cases for something like this, and it's better to allow users to implement this themselves.

Adding explicit beam support to the CRD (or, as I think I prefer, creating a separate BeamApplication CRD) should improve the user experience by automatically creating the sidecar with the appropriate configuration.

[tweise]

Just a note that I'm going to follow-up on this. But in the meantime, a generalized sidecar container concept in the CRD would probably also benefit other Flink applications.

[chadrik]

@tweise is the side-car approach still relevant after your latest discoveries/realizations that the best option to run portable Beam pipelines on k8s is to create container images that have both Flink and the SDK dependencies?

[tweise]

@chadrik the main advantage is that sidecar would allow to use the stock SDK container images independent of Flink. This might become more important as we move to cross-language pipelines. At the moment though, we are not going to rely on this for our internal deployment. I will add an example for the single container approach in the next few weeks. Are you waiting for that?

[chadrik]

I will add an example for the single container approach in the next few weeks. Are you waiting for that?

It's not holding us up. Having it in the next few weeks will be fine.

[tweise]

@chadrik the example is pretty much ready to go, waiting for the 2.17 Beam release: https://github.com/lyft/flinkk8soperator/pull/134

[chadrik]

Thanks for the update. Perfect timing as we are getting started on this right now.

On Tue, Dec 3, 2019 at 7:02 PM Thomas Weise notifications@github.com wrote:

@chadrik https://github.com/chadrik the example is pretty much ready to go, waiting for the 2.17 Beam release: #134 https://github.com/lyft/flinkk8soperator/pull/134

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lyft/flinkk8soperator/issues/102?email_source=notifications&email_token=AAAPOE65JA3CBSKBEDG3HFLQW4MTHA5CNFSM4IVA2PN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEF3R3HQ#issuecomment-561454494, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAPOE6HQDNNVLEGU7OEGBTQW4MTHANCNFSM4IVA2PNQ .

[dmvk]

Hi, is there any progress on this? We'd love to use sidecar containers for loadbalacing minio traffic using https://blog.min.io/introducing-sidekick-a-high-performance-load-balancer/.

Thanks, D.