Artifacts using Azure Shared Access Signatures fails with AuthenticationFailed

Pre-requisites

[X] I have double-checked my configuration
[ ] I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
[X] I have searched existing issues and could not find a match for this bug
[x] I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

When my workflow try to use artifacts using Azure in the logs of the main container we see the fatal error in the wait container as attached below. I followed this doc to generate the secret, but after some trail and error we found that the secret must be a base64 of the base64 of the SAS token, but then we face this issue.

Version(s)

v3.4.5

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: test-artifact
spec:
  workflowSpec:
    templates:
      - name: test-artifact-template
        outputs:
          artifacts:
            - name: hello-word
              path: /tmp/hello_world.txt
              azure:
                endpoint: https://[ACCOUNTNAME].blob.core.windows.net
                container: [CONTAINER]
                blob: path
                accountKeySecret:
                  name: argowf-artifacts
                  key: shared-access-key
        container:
          name: test-artifact
          image: busybox
          command: [sh, -c]
          args: ["echo hello world | tee /tmp/hello_world.txt"]

        tolerations:
          - key: topology.cluster/reserved-for
            operator: Equal
            value: high-workloads
            effect: NoSchedule
          - key: topology.cluster/reserved-for
            operator: Equal
            value: croupier-worker
            effect: NoSchedule
          - key: topology.cluster/multiple-availability-zones
            operator: Exists
            effect: NoSchedule
        nodeSelector: {}

    entrypoint: test-artifact-template
    ttlStrategy:
      secondsAfterCompletion: 60
    activeDeadlineSeconds: 1800
    podGC:
      strategy: OnPodSuccess
  schedule: 0 1 * * *

Logs from the workflow controller

time="2024-11-04T16:02:12.943Z" level=info msg="Processing tap-beta/sync-zuora-db" cronWorkflow=tap-beta/sync-zuora-db
time="2024-11-04T16:02:12.954Z" level=info msg="CronWorkflow tap-beta/sync-zuora-db added" cronWorkflow=tap-beta/sync-zuora-db
time="2024-11-04T16:02:17.181Z" level=info msg="Enforcing history limit for 'sync-zuora-db'" namespace=tap-beta workflow=sync-zuora-db
time="2024-11-04T16:02:27.184Z" level=info msg="Enforcing history limit for 'sync-zuora-db'" namespace=tap-beta workflow=sync-zuora-db
time="2024-11-04T16:02:37.186Z" level=info msg="Enforcing history limit for 'sync-zuora-db'" namespace=tap-beta workflow=sync-zuora-db
time="2024-11-04T16:02:47.189Z" level=info msg="Enforcing history limit for 'sync-zuora-db'" namespace=tap-beta workflow=sync-zuora-db
time="2024-11-04T16:02:57.191Z" level=info msg="Enforcing history limit for 'sync-zuora-db'" namespace=tap-beta workflow=sync-zuora-db
time="2024-11-04T16:03:07.194Z" level=info msg="Enforcing history limit for 'sync-zuora-db'" namespace=tap-beta workflow=sync-zuora-db
time="2024-11-04T16:03:17.196Z" level=info msg="Enforcing history limit for 'sync-zuora-db'" namespace=tap-beta workflow=sync-zuora-db
time="2024-11-04T16:03:27.198Z" level=info msg="Enforcing history limit for 'sync-zuora-db'" namespace=tap-beta workflow=sync-zuora-db
time="2024-11-04T16:03:37.201Z" level=info msg="Enforcing history limit for 'sync-zuora-db'" namespace=tap-beta workflow=sync-zuora-db
time="2024-11-04T16:03:47.204Z" level=info msg="Enforcing history limit for 'sync-zuora-db'" namespace=tap-beta workflow=sync-zuora-db

Logs from in your workflow's wait container

time="2024-11-04T16:15:35.624Z" level=info msg="Starting Workflow Executor" version=v3.4.5
time="2024-11-04T16:15:35.628Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2024-11-04T16:15:35.628Z" level=info msg="Executor initialized" deadline="2024-11-04 16:45:32 +0000 UTC" includeScriptOutput=false namespace=tap-beta podName=sync-zuora-db-t96gg-start-zuora-process-template-2309314069 template="{\"name\":\"start-zuora-process-template\",\"inputs\":{},\"outputs\":{\"artifacts\":[{\"name\":\"zuora-jobs\",\"path\":\"/tmp/result\",\"azure\":{\"endpoint\":\"https://tapdevstoragecommon.blob.core.windows.net\",\"container\":\"argowf-artifacts\",\"accountKeySecret\":{\"name\":\"argowf-artifacts\",\"key\":\"shared-access-key\"},\"blob\":\"sync-zuora-db\"}}]},\"metadata\":{},\"container\":{\"name\":\"sync-zuora-db\",\"image\":\"topconagriculture.azurecr.io/market-jobs-sync-zuora-db:develop\",\"command\":[\"sh\",\"-c\"],\"args\":[\"rushx start:run-zuora-process --output-path /tmp/result\"],\"envFrom\":[{\"configMapRef\":{\"name\":\"sync-zuora-db-configmap\"}},{\"secretRef\":{\"name\":\"sync-zuora-db-secret\"}}],\"resources\":{\"limits\":{\"cpu\":\"800m\",\"memory\":\"800Mi\"},\"requests\":{\"cpu\":\"400m\",\"memory\":\"400Mi\"}},\"volumeMounts\":[{\"name\":\"sync-zuora-db-logger\",\"mountPath\":\"/app/config\"}]},\"tolerations\":[{\"key\":\"topology.cluster/reserved-for\",\"operator\":\"Equal\",\"value\":\"high-workloads\",\"effect\":\"NoSchedule\"},{\"key\":\"topology.cluster/reserved-for\",\"operator\":\"Equal\",\"value\":\"croupier-worker\",\"effect\":\"NoSchedule\"},{\"key\":\"topology.cluster/multiple-availability-zones\",\"operator\":\"Exists\",\"effect\":\"NoSchedule\"}]}" version="&Version{Version:v3.4.5,BuildDate:2023-02-07T12:34:55Z,GitCommit:1253f443baa8ad1610d2e62ec26ecdc85fe1b837,GitTag:v3.4.5,GitTreeState:clean,GoVersion:go1.18.10,Compiler:gc,Platform:linux/amd64,}"
time="2024-11-04T16:15:35.628Z" level=info msg="Starting deadline monitor"
time="2024-11-04T16:15:39.629Z" level=info msg="Main container completed" error="<nil>"
time="2024-11-04T16:15:39.629Z" level=info msg="Deadline monitor stopped"
time="2024-11-04T16:15:39.629Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2024-11-04T16:15:39.629Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2024-11-04T16:15:39.629Z" level=info msg="No output parameters"
time="2024-11-04T16:15:39.629Z" level=info msg="Saving output artifacts"
time="2024-11-04T16:15:39.630Z" level=info msg="Staging artifact: zuora-jobs"
time="2024-11-04T16:15:39.630Z" level=info msg="Copying /tmp/result from container base image layer to /tmp/argo/outputs/artifacts/zuora-jobs.tgz"
time="2024-11-04T16:15:39.630Z" level=info msg="/var/run/argo/outputs/artifacts/tmp/result.tgz -> /tmp/argo/outputs/artifacts/zuora-jobs.tgz"
time="2024-11-04T16:15:39.630Z" level=info msg="Saving to Azure Blob Storage" blob=sync-zuora-db container=argowf-artifacts endpoint="https://tapdevstoragecommon.blob.core.windows.net"
time="2024-11-04T16:15:39.676Z" level=info msg="Save artifact" artifactName=zuora-jobs duration=45.754737ms error="unable to upload file /tmp/argo/outputs/artifacts/zuora-jobs.tgz to Azure: ===== RESPONSE ERROR (ErrorCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:31438d92-c01e-0085-5bd4-2ebe69000000\nTime:2024-11-04T16:15:39.6749060Z, Details: \n   AuthenticationErrorDetail: The MAC signature found in the HTTP request 'vGr1BDcFmvp4+VUWeD09daasNeZWwBiTl9k82fUmmaA=' is not the same as any computed signature. Server used following string to sign: 'PUT\n\n\n389\n\napplication/octet-stream\n\n\n\n\n\n\nx-ms-blob-type:BlockBlob\nx-ms-date:Mon, 04 Nov 2024 16:15:39 GMT\nx-ms-version:2020-10-02\n/tapdevstoragecommon/argowf-artifacts/sync-zuora-db'.\n" key=sync-zuora-db
time="2024-11-04T16:15:39.676Z" level=error msg="executor error: unable to upload file /tmp/argo/outputs/artifacts/zuora-jobs.tgz to Azure: ===== RESPONSE ERROR (ErrorCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:31438d92-c01e-0085-5bd4-2ebe69000000\nTime:2024-11-04T16:15:39.6749060Z, Details: \n   AuthenticationErrorDetail: The MAC signature found in the HTTP request 'vGr1BDcFmvp4+VUWeD09daasNeZWwBiTl9k82fUmmaA=' is not the same as any computed signature. Server used following string to sign: 'PUT\n\n\n389\n\napplication/octet-stream\n\n\n\n\n\n\nx-ms-blob-type:BlockBlob\nx-ms-date:Mon, 04 Nov 2024 16:15:39 GMT\nx-ms-version:2020-10-02\n/tapdevstoragecommon/argowf-artifacts/sync-zuora-db'.\n"
time="2024-11-04T16:15:39.703Z" level=info msg="Create workflowtaskresults 403"
time="2024-11-04T16:15:39.703Z" level=warning msg="failed to patch task set, falling back to legacy/insecure pod patch, see https://argoproj.github.io/argo-workflows/workflow-rbac/" error="workflowtaskresults.argoproj.io is forbidden: User \"system:serviceaccount:tap-beta:default\" cannot create resource \"workflowtaskresults\" in API group \"argoproj.io\" in the namespace \"tap-beta\""
time="2024-11-04T16:15:39.723Z" level=info msg="Patch pods 200"
time="2024-11-04T16:15:39.728Z" level=info msg="Alloc=8750 TotalAlloc=15799 Sys=19154 NumGC=5 Goroutines=7"
time="2024-11-04T16:15:39.728Z" level=fatal msg="unable to upload file /tmp/argo/outputs/artifacts/zuora-jobs.tgz to Azure: ===== RESPONSE ERROR (ErrorCode=AuthenticationFailed) =====\nDescription=Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.\nRequestId:31438d92-c01e-0085-5bd4-2ebe69000000\nTime:2024-11-04T16:15:39.6749060Z, Details: \n   AuthenticationErrorDetail: The MAC signature found in the HTTP request 'vGr1BDcFmvp4+VUWeD09daasNeZWwBiTl9k82fUmmaA=' is not the same as any computed signature. Server used following string to sign: 'PUT\n\n\n389\n\napplication/octet-stream\n\n\n\n\n\n\nx-ms-blob-type:BlockBlob\nx-ms-date:Mon, 04 Nov 2024 16:15:39 GMT\nx-ms-version:2020-10-02\n/tapdevstoragecommon/argowf-artifacts/sync-zuora-db'.\n"

argoproj / argo-workflows

Artifacts using Azure Shared Access Signatures fails with AuthenticationFailed #13864