K8s watcher creates simple lineage graphs for K8s objects

[jlewi]

• K8s objects will now be logged as executions instead of artifacts

• The watcher looks for annotations defining inputs and outputs

  • If the annotations are present they will be logged as artifacts
  • The watcher will then create events linking the inputs & outputs to the execution

• The net result is that we can generate lineage graphs just by attaching annotations to K8s object.

• The current implementation is pretty primitive; there are a number of issues to fix in follow ups

  • Handling multiple inputs and outputs

  • Handling the case where there already artifacts in the DB corresponding to the annotations

  • Right now a new artifact is created for each resource.

• Add an example job that we can use to test things.

Miscellaneous fixes

• For the watcher add Makefile rules to build with GCB

  • For the manifests create a kustomize file so its easy to overwrite the docker image with the latest image

• Check in the generated go client libraries for the metadata service.

• Update watcher deployment to v1.

• Change the resource_list.json to watch jobs not pods/services by default.

Related to: #241

/assign @karlschriek /assign @Swikar

[kubeflow-bot]

This change is 

[Jeffwan]

I can help review the PR.

[Jeffwan]

/assign @Jeffwan

[jlewi]

Test flakes should be fixed by #251

[k8s-ci-robot]

@jlewi: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
kubeflow-metadata-presubmit	62c836879ee430b7ac1fc4268171ec6fe20f24ae	link	/test kubeflow-metadata-presubmit

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Jeffwan, jlewi To complete the pull request process, please assign neuromage after the PR has been reviewed. You can assign the PR to them by writing /assign @neuromage in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/kubeflow/metadata/blob/master/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[jlewi]

I'm closing this because I no longer have merge rights and I'm not sure what's going on with metadata anymore.

[Swikar]

Sure

On May 3, 2021, at 6:28 AM, Jeremy Lewi @.***> wrote:

 I'm closing this because I no longer have merge rights and I'm not sure what's going on with metadata anymore.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub, or unsubscribe.