feature proposal: resource management on argo/kubeflow

[edublancas]

We want to add support for pipelines exported to argo and kubeflow to request specific resources for a given task (memory, CPU, GPU).

I'm thinking we could have a mapping in the config that matches task names to environments with certain characteristics:

my-config:
  resources_mapping:
    # train task should use this resources
    train:
      memory: some-value
      cpu: some-value
    # support for wildcards - all tasks with the fit- prefix should execute with this resources
    # say: fit-some-model, fit-some-other-model
    fit-*:
      memory: some-value
      cpu: some-value

I found this on argo's documentation, I think this is what we need.

Kubeflow uses argo under the hood, so I'm guessing it uses the same mechanism but if anyone has a link to the kubeflow docs that explain this part, please share it.

what it isn't clear to me is how the GPU comes into play here. Please comment if you have any extra info.