Integration Ideas

[msteffen]

This issue should be rewritten as an RFC, but just recording some of our ideas from this morning:

• When integrating with other systems, Pachyderm should ideally be the triggering mechanism for external systems: running a KF pipeline, serving a new model, etc.
  • Part of what makes Pachyderm valuable is the conceptual model for preserving data lineage, and pushing people to our technical model will also push them to our conceptual model
  • This allows Pachyderm to maximize the amount of data-lineage-derived value we can provide.
  • If users are e.g. triggering their own pipelines, then there's nothing to prevent them from processing stale data or processing mismatched data (today's logs with yesterday's billing), which is one of the main problems Pachyderm is designed to prevent. Pachyderm will record what happened in its commit lineage, so the problem will at least be easier to debug, but that's only marginally better than the status quo of shared-S3-bucket free-for-all.
  • That said, we recognize that not everybody is willing to work this way or can migrate all at once. So, one integration feature is a simple way for external systems to trigger their own Pachyderm jobs. I, at least, imagine using deferred processing to trigger the start of a new job, running a dummy API inside of the job that waits for a kill signal, and using a kill signal (also sent by the external system) to signal the end of a pachyderm job
• We hope to make progress on pipeline templates, which will make dummy pipelines (like the kill API mentioned above, or the KubeFlow trigger pipeline) easy to run without having to learn the Pachyderm pipeline spec
• Finally, we hope to allow PPS to create arbitrary kubernetes resources on incoming commits, rather than just Pachyderm pipelines (i.e. RCs).
  • This could include kubernetes secrets (e.g. to signal to Seldon that an updated model version is ready for serving), TFJobs, or XYZDeploy CRDs for other AI systems that we've looked at
  • If we define a CRD for Pachyderm pipelines, then the creation of arbitrary Kubernetes resources might not even be a special case for PPS. Pachyderm pipelines might be one resource among many that PPS is able to create

[jmandel1027]

This would be a major win for my organization, if we had a Pachd operator to manage the state of repos + pipelines, it would greatly simplify and enhance our deployment cruft. We embrace gitops workflows on our end, utilizing Flux, Kustomize and Helm, and our pachyderm pipelines are a notable exception to this rule.

These guys did something similar to what we'd ultimately like to see, but it's incompatible w/ recent versions of pachyderm, doesn't support the full pipeline spec and doesn't utilize docker. https://github.com/adarga-ai/pachyderm-pipeline-controller

major bump to this proposal.