Kubernetes task scheduling planning/design

[claymcleod]

Hey Mike,

Just wanted to say that for our team, this would be really interesting. It's really just a matter of technology selection and not for any real technical reason.

Another reason I could put out there for prioritizing this would be that all of the major cloud providers are putting quite a bit of effort into maturing their Kubernetes-as-a-service offerings. I haven't paid a lot of attention to what they offer for Docker Swarm, it seems like it is mostly phasing out. Obviously we are heavy users of Azure, and a quick Googling suggests that Azure is retiring their Docker Swarm support in favor of focusing on AKS (also see this deprecation notice of ACS which gives more info).

cc @acchiao

[mlin]

Yea, Docker Swarm continues to make sense as the default for local use because it's so easy to start up -- but we want to support Kubernetes scheduling too for all the reasons you mention.

How are you trading off between object storage & shared (POSIX) file systems in your Kubernetes deployments? This is one of the biggest open questions about this feature right now -- how to move files between task containers. Our early experiments with Swarm clusters in AWS have been using FSx, which supplies a magic Lustre mount that can also read-through from and write-back to S3 when needed. That helps a lot to keep miniwdl "mini", but not sure if we can count on something like that for other platforms.

[aaronluoq]

General cloud providers can mount object storage, others can use NFS, Lustre, etc.

[lukasheinrich]

Hi all,

this is something that I'm interested as well. We've been so far using a homegrown workflow engine with existing kubernetes support (https://github.com/yadage/yadage) but long-term are considering taking up WDL. So we have some experience implementing k8s-based workflow engines

cc @clelange

[mlin]

Looks promising for test/dev: https://k0sproject.io

[lukasheinrich]

k3d and kind are also very nice to develop against. I saw this was moved form Backlog to Next. What would be a best way to contribute?

[lynnlangit]

note that K8s now supports containerd and NOT docker (unless you add a shim). To clarify K8 can still RUN a Docker image, but going forward to BUILD a Docker image, a shim would be required. - more here

[claymcleod]

@mlin any updates on the priority of getting this into miniwdl? Just checking in.

[marpiech]

@mlin I could help with that if you are considering external contributors