argoproj / argo-workflows

Workflow Engine for Kubernetes
https://argo-workflows.readthedocs.io/
Apache License 2.0
14.61k stars 3.13k forks source link

Validation error when using string handling for defining S3 artifact key #10874

Open sakvaka opened 1 year ago

sakvaka commented 1 year ago

Pre-requisites

What happened/what you expected to happen?

Since S3 artifacts cannot be parsed directly from URI (see https://github.com/argoproj/argo-workflows/issues/4349), I need to extract the bucket and key parts from the URI. I tried using sprig.splitList to split the URI at / symbols, and slice the list to get the bucket and key components. Then, I use the third component as the bucket name and with sprig.join, I join the other parts but the first three to get the s3 key part.

However, this string handling seems to work when defining parameters but not when defining an s3 artifact key. Submitting the workflow below gives a validation error Bad Request: templates.starthere.steps[0].main templates.main.steps[2].consume.arguments.file.from, artifact location, or key is required. The error may be related to the list slicing part ([3:]), since removing the slicing makes the validation error message disappear. Surprisingly, the same slicing is handled correctly with the bucket: specification, which is however without the sprig.join.

The whalesay templates are in the following workflow just to demonstrate there is no syntax error in the variable expressions.

Version

V3.4.6

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: key-only-artifacts-
spec:
  entrypoint: starthere
  templates:
    - name: starthere
      inputs:
        parameters:
          - name: tilelist
            value: >
              [ { "tile": "S2B_MSIL1C_20230410T090559_N0509_R050_T36VUM_20230410T094735.SAFE", "status": "FETCH_OK", "s3_path": "s3://MYBUCKET/eodata/Sentinel-2/S2B_MSIL1C_20230410T090559_N0509_R050_T36VUM_20230410T094735.SAFE.tgz" }]
      steps:
        - - name: main
            template: main
            arguments:
              parameters:
                - name: s3key
                  value: "{{item.s3_path}}"
            withParam: "{{inputs.parameters.tilelist}}"
    - name: main
      inputs:
        parameters:
          - name: s3key
      steps:
        - - name: sayhello
            template: whalesay
            arguments:
              parameters:
              - name: message
                value: "{{=sprig.splitList('/', inputs.parameters.s3key)[2]}}"
        - - name: sayhelloagain
            template: whalesay
            arguments:
              parameters:
              - name: message
                value: "{{=sprig.join('/', sprig.splitList('/', inputs.parameters.s3key)[3:])}}" # this is ok
        - - name: consume
            template: consume
            arguments:
              artifacts:
                - name: file
                  s3:
                    endpoint: "s3.amazonaws.com"
                    bucket: "{{=sprig.splitList('/', inputs.parameters.s3key)[2]}}" # this is ok
                    region: "eu-central-1"
                    useSDKCreds: true
                    key: "{{=sprig.join('/', sprig.splitList('/', inputs.parameters.s3key)[3:])}}" # error! removing [3:] the validation error disappears
    - name: consume
      container:
        image: argoproj/argosay:v2
        args: [cat, /tmp/file, less]
      inputs:
        artifacts:
          - name: file
            path: /tmp/file
    - name: whalesay
      inputs:
        parameters:
        - name: message
      container:
        image: docker/whalesay
        command: [cowsay]
        args: ["{{inputs.parameters.message}}"]

Logs from the workflow controller

(No logs from the controller since the workflow cannot be submitted, but this is the log from the server.)

time="2023-04-11T08:01:25.174Z" level=info msg="finished unary call with code InvalidArgument" error="rpc error: code = InvalidArgument desc = templates.starthere.steps[0].main templates.main.steps[2].consume.arguments.file.from, artifact location, or key is required" grpc.code=InvalidArgument grpc.method=CreateWorkflow grpc.service=workflow.WorkflowService grpc.start_time="2023-04-11T08:01:25Z" grpc.time_ms=5.562 span.kind=server system=gr

Logs from in your workflow's wait container

(No logs since the workflow cannot be submitted.)
stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.