search

argoproj / argo-workflows

Workflow Engine for Kubernetes
https://argo-workflows.readthedocs.io/
Apache License 2.0
15.07k stars 3.2k forks source link

Custom artifact repository plugins #5862

Open jalberti opened 3 years ago

jalberti commented 3 years ago

Summary

There are several Github issues that discuss volumes and how they can and do relate to artifact management, e.g. https://github.com/argoproj/argo-workflows/issues/1024, https://github.com/argoproj/argo-workflows/issues/1349. Today the supported artifact repository types are fixed, and extensions are only feasible in-tree, if I understand this correctly? Iā€™m wondering if there were any specific thoughts on how CSI could be used, or a CSI like concept, to allow artifact repositories to be provided as plugins?

The problem at the moment is, unless I overlooked something, that while I can get volumes into the pod, into my workflow, and I can also write arbitrary code, and can even have side cars as demonstrated here https://github.com/argoproj/argo-workflows/issues/4988, ... if my input or output does not fit into the standard supported artifact repository set, I'm not able to use the concept of artifact declaration. Essential the "declaration" of "input/output" is lost. If one would be able to bring their own repository, then we can use the concept of artifacts together with custom code, i.e. we would have extensibility and declarative artifacts.

E.g. we could define a stable JSON-RPC interface, and allow community provided artifact repositories in form of a side car container? The artifact repository provider would create the container image with the JSON-RPC server, Argo would act as the JSON-RPC client. The Argo workflow would define an artifact repository of type jsonrpc, the artifact repository driver container and the workflow step container would share a common volumeMount such that artifact content does not need to pass the json-rpc pipe.

Use Cases

This would be useful to add support for custom community supported artifact repositories.

Message from the maintainers:

Impacted by this bug? Give it a šŸ‘. We prioritise the issues with the most šŸ‘.

alexec commented 2 years ago

This fits in well with our plugin architecture. I'm not sure if the best way to do this is sidecar, or if it should be done using binaries embedded on the image.

alexec commented 2 years ago

This would fix the #1540 support.