search

kubeflow / metadata

Repository for assets related to Metadata.
Apache License 2.0
121 stars 69 forks source link

K8s Metadata watcher: handle multiple inputs & outputs #249

Closed jlewi closed 2 years ago

jlewi commented 3 years ago

/kind feature

Follow on to #241 and #246. In that PR the watcher looks at annotations on Kubernetes resources and creates artifacts from them.

apiVersion: batch/v1
kind: Job
metadata:
  generateName: watcher-job-
  annotations:
    "metadata.kubeflow.org/input": |
      {"apiVersion":"metadata.kubeflow.org/v1alpha1","kind":"Dataset","metadata":{"name":"fakeInput"},"spec":{"prop1":"fakevalue"}}
    "metadata.kubeflow.org/output": |
      {"apiVersion":"metadata.kubeflow.org/v1alpha1","kind":"Dataset","metadata":{"name":"fakeOutput"},"spec":{"prop1":"fakeoutput"}}
spec:
   ...

Right now the watcher can only handle the case where input and output contains a single artifact. We should extend that to handle the case where the the input and output can be lists of items. In which case we should create one artifact for each item in the list.

/cc @Swikar @karlschriek