Configure artifacts using URIs, e.g `s3://my-bucket/my-key`

[alexec]

Summary

It might be convenient and more compact to represent artifacts by URIs. A bit like go getter, but with the option to put as well as get.

Use Cases

Add more repositories.

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[alexec]

• Local files
• Git
• Mercurial
• HTTP
• Amazon S3
• Google GCP

We'd also need:

• Raw
• Artifactory (maybe just use HTTP)
• OSS

How do you do secrets?

[Ark-kun]

go getter,

Does go-getter support canonical storage URIs? E.g. gs:// s3:// http:// etc.

How do you do secrets?

Maybe we can have a configmap that can specify authentication for all URI schemes. The confimap can be per-cluster or specified per-workflow (like ArtifactRepositoryRef)

In the future we could also make it possible to specify auth/secrets per-bucket.

P.S. I do not have a big stake in this. The scenarios that could be useful to me is:

• Upstream produces a URI (e.g. "gs://my-bucket/file.txt") via an Argo parameter, then downstream consumes it as a file (Argo artifact).
• Upstream produces an artifact, downstream wants to receive a URI for that artifact as an input parameter. There are workarounds for these scenarios now, but they're pretty cumbersome or even impossible. For example, given a URI, some intermediate template could split it into scheme, bucket an key and output those as separate parameters that can be plugged into the downstream's s3 structure. It would be easier if we could do:

name: downstream
inputs:
  parameters:
  - name: input1-uri
  artifacts:
  - name: input1
    path: ...
    uri: {{inputs.parameters.input1-uri}}

name: dag-task-1
template: downstream
arguments:
  parameters:
  - name: input1-uri
     valueFrom: {{tasks.upstream.outputs.parameters.some-uri}}

[steve-marmalade]

I have an argo step that generates a list of GCS URIs of the form gs://my-bucket/path/to/file that I'd like to map over. It would be convenient to pass these URIs directly to the gcs artifact handler, rather than having to post-process them into bucket and key. I found this issue while searching to see whether this is supported, and figured I'd post my use case in the hope that it's helpful. Thanks!

[alexec]

I'm planning on doing some work in the next few that would be made much easier with URNs for artifacts. To be able to load/save artifacts we'll need to be able to:

• Have an URL or URN for any artifact
• Have a context for accessing the artifact, e.g. a namespace, workflow, or artifact repository, than allows us to ask - is the principal/subject allowed to read/write/delete the artifact.
• Combine the URN and auth to perform the operation.
• In the current implementation, location and auth are combined into a single struct.
• Some artifacts point to files, some to directories.

Lets look at some URNs:

Repo	Example	Notes
Artifactory	URL
Git	https://github.com/argoproj/argo-workflows.git or git@github.com:argoproj/argo-workflows.git	SSH and HTTPS URLs are different
HDFS	hdfs://hadoopNS/data/users.csv
HTTP	URL
GCS	//www.googleapis.com/storage/v1/bucket/foo.zip
OSS	https://BucketName.Endpoint/Object?SignatureParameters
Raw	base64(value)	Not sure how to represent this for large values.
S3	https://s3-eu-west-1.amazonaws.com/bucket/foo	There are several variants

[alexec]

Currently, you need the workflow to find the artifact. We could restrict this workflows using artifactRepositioryRef (i.e. a named configmap).

Auth URN:

artifactRepositoryRef:v1:{namespace}:{name}:{key}
workflow:v1:{namespace}:{name}:{nodeID}:input:{artifactName}
workflow:v1:{namespace}:{name}:{nodeID}:output:{artifactName}

Artifact URN:

artifact:v1:s3:{endpoint}:{bucket}:{key}
artifact:v1:git:{repo}:{branch}

Dataflow gets it wrong, secrets should not go in the URN:

https://github.com/argoproj-labs/argo-dataflow/blob/11251f5806ed8bfbd9e6b5017259c19b006118ca/docs/META.md