search

argoproj / argo-workflows

Workflow Engine for Kubernetes
https://argo-workflows.readthedocs.io/
Apache License 2.0
15k stars 3.2k forks source link

artifact passing: "x509: certificate signed by unknown authority" #6635

Closed mbu93 closed 2 years ago

mbu93 commented 3 years ago

Hey! I am using argo workflows together with minio as a base for various machine learning automation setups. I love it, thanks for your efforts! I did however encounter I problem when switching to the new minio-operator version, that I can't seem to figure out (in a non-hacky way). Hope you can help me out here.

Summary

What happened/what you expected to happen?

Minio-operator has been around for a while now, in favour of its (now deprecated) former version. In order to use the new buckets for artifact passing, one needs to make use of https. In our case, the certificates used to do so, are self-signed wich causes issues when artifacts need to be uploaded. After the .tar.gz has been created, the upload fails with error message ... x509: certificate signed by unknown authority. I'm curious how to configure things right here. Do I really need to mount the related .ca file to ANY workflow using artifacts itself, in order to use minio with https (and a self-signed) cert?

Have you double-checked your configuration? Maybe 30% of issues are wrong configuration.

I've double-checked my config. To my best knowledge, there is no option that allows me to ignore the insecure authority. Please verify if you like:

containerRuntimeExecutor: docker
artifactRepository:
  archiveLogs: false
  s3:
    endpoint: minio.minio-ssd.svc.cluster.local:443
    bucket: test
    insecure: false
    useSDKCreds: false
    accessKeySecret:
      name: minio-key
      key: access_key_id
    secretKeySecret:
      name: minio-secret
      key: secret_access_key

I've also tried setting the kubeletInsecure Option, but I assume, that would require me to use kubelet as workflows executor?

Diagnostics

What Kubernetes provider are you using?

On-premise, 10-node cluster, each node is a Centos-7 virtual machine, Kubernetes Version 1.20.8

What version of Argo Workflows are you running?

The version used for argo-cli and argo-workflow-controller was v2.12.5.

What executor are you running?Docker/K8SAPI/Kubelet/PNS/Emissary

Docker was used as an executor (I have no other possibility).

Did this work in a previous version? I.e. is it a regression?

I just switched to the minio operator version and thus can't confirm for later versions.

Workflow YAML:

 ```yaml apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: creationTimestamp: "2021-08-31T14:53:20Z" generateName: artifact-passing- generation: 4 labels: workflows.argoproj.io/completed: "true" workflows.argoproj.io/phase: Failed workflows.argoproj.io/resubmitted-from-workflow: artifact-passing-pj8v7 name: artifact-passing-5bs6q namespace: twin-pipelines resourceVersion: "660135225" uid: f28ff596-914e-4d97-b55f-3de083bd455d spec: arguments: {} entrypoint: artifact-example serviceAccountName: pipeline-runner templates: - arguments: {} inputs: {} metadata: {} name: artifact-example outputs: {} steps: - - arguments: {} name: generate-artifact template: whalesay - - arguments: artifacts: - from: '{{steps.generate-artifact.outputs.artifacts.hello-art}}' name: message name: consume-artifact template: print-message - arguments: {} container: args: - sleep 1; cowsay hello world | tee /tmp/hello_world.txt command: - sh - -c image: docker/whalesay:latest name: "" resources: {} inputs: {} metadata: {} name: whalesay outputs: artifacts: - name: hello-art path: /tmp/hello_world.txt - arguments: {} container: args: - cat /tmp/message command: - sh - -c image: alpine:latest name: "" resources: {} inputs: artifacts: - name: message path: /tmp/message metadata: {} name: print-message outputs: {} status: conditions: - status: "True" type: Completed finishedAt: "2021-08-31T14:54:07Z" message: child 'artifact-passing-5bs6q-1898326195' failed nodes: artifact-passing-5bs6q: children: - artifact-passing-5bs6q-2310228769 displayName: artifact-passing-5bs6q finishedAt: "2021-08-31T14:54:07Z" id: artifact-passing-5bs6q message: child 'artifact-passing-5bs6q-1898326195' failed name: artifact-passing-5bs6q outboundNodes: - artifact-passing-5bs6q-1898326195 phase: Failed progress: 1/1 resourcesDuration: cpu: 36 memory: 36 startedAt: "2021-08-31T14:53:20Z" templateName: artifact-example templateScope: local/artifact-passing-5bs6q type: Steps artifact-passing-5bs6q-1898326195: boundaryID: artifact-passing-5bs6q displayName: generate-artifact finishedAt: "2021-08-31T14:53:57Z" hostNodeName: cuda01-p100 id: artifact-passing-5bs6q-1898326195 message: 'failed to save outputs: timed out waiting for the condition' name: artifact-passing-5bs6q[0].generate-artifact phase: Error progress: 1/1 resourcesDuration: cpu: 36 memory: 36 startedAt: "2021-08-31T14:53:20Z" templateName: whalesay templateScope: local/artifact-passing-5bs6q type: Pod artifact-passing-5bs6q-2310228769: boundaryID: artifact-passing-5bs6q children: - artifact-passing-5bs6q-1898326195 displayName: '[0]' finishedAt: "2021-08-31T14:54:07Z" id: artifact-passing-5bs6q-2310228769 message: child 'artifact-passing-5bs6q-1898326195' failed name: artifact-passing-5bs6q[0] phase: Failed progress: 1/1 resourcesDuration: cpu: 36 memory: 36 startedAt: "2021-08-31T14:53:20Z" templateName: artifact-example templateScope: local/artifact-passing-5bs6q type: StepGroup phase: Failed progress: 1/1 resourcesDuration: cpu: 36 memory: 36 startedAt: "2021-08-31T14:53:20Z" ```

Logs from the workflow controller:

 ``` time="2021-08-31T12:55:45.257Z" level=info msg="config map" name=argo-workflow-controller-configmap time="2021-08-31T12:55:45.283Z" level=info msg="Workers: workflow: 32, pod: 32" time="2021-08-31T12:55:45.286Z" level=info msg="Persistence disabled - so archived workflow GC disabled - you must restart the controller if you enable this" time="2021-08-31T12:55:45.290Z" level=info msg="Starting workflow TTL controller (workflowTTLWorkers 4)" time="2021-08-31T12:55:45.490Z" level=info msg="Started workflow TTL worker" time="2021-08-31T12:55:54.474Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:55:54.476Z" level=info msg="Pod node artifact-passing-fnjrp-7432736 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:55:54.499Z" level=info msg="Created pod: artifact-passing-fnjrp[0].generate-artifact (artifact-passing-fnjrp-7432736)" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:55:54.500Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:55:54.537Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660077457 workflow=artifact-passing-fnjrp time="2021-08-31T12:56:04.503Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:04.505Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:04.520Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660077545 workflow=artifact-passing-fnjrp time="2021-08-31T12:56:14.525Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:14.527Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.430Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.430Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-fnjrp-7432736 templateName=whalesay workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="Step group node artifact-passing-fnjrp-4055596050 deemed failed: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="node artifact-passing-fnjrp-4055596050 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="node artifact-passing-fnjrp-4055596050 message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="node artifact-passing-fnjrp-4055596050 finished: 2021-08-31 12:56:40.432283488 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="step group artifact-passing-fnjrp-4055596050 was unsuccessful: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="Outbound nodes of artifact-passing-fnjrp-7432736 is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="Outbound nodes of artifact-passing-fnjrp is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="node artifact-passing-fnjrp phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="node artifact-passing-fnjrp message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="node artifact-passing-fnjrp finished: 2021-08-31 12:56:40.432517288 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="Checking daemoned children of artifact-passing-fnjrp" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="Updated message -> child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.432Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T12:56:40.451Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660077841 workflow=artifact-passing-fnjrp time="2021-08-31T13:00:06.330Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:06.332Z" level=info msg="Pod node artifact-passing-fnjrp-7432736 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:06.345Z" level=info msg="Created pod: artifact-passing-fnjrp[0].generate-artifact (artifact-passing-fnjrp-7432736)" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:06.345Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:06.359Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660079525 workflow=artifact-passing-fnjrp time="2021-08-31T13:00:16.348Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:16.349Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:16.366Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660079624 workflow=artifact-passing-fnjrp time="2021-08-31T13:00:26.371Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:26.373Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.415Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.415Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-fnjrp-7432736 templateName=whalesay workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="Step group node artifact-passing-fnjrp-4055596050 deemed failed: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="node artifact-passing-fnjrp-4055596050 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="node artifact-passing-fnjrp-4055596050 message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="node artifact-passing-fnjrp-4055596050 finished: 2021-08-31 13:00:52.417418597 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="step group artifact-passing-fnjrp-4055596050 was unsuccessful: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="Outbound nodes of artifact-passing-fnjrp-7432736 is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="Outbound nodes of artifact-passing-fnjrp is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="node artifact-passing-fnjrp phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="node artifact-passing-fnjrp message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="node artifact-passing-fnjrp finished: 2021-08-31 13:00:52.417652801 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="Checking daemoned children of artifact-passing-fnjrp" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="Updated message -> child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.417Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:00:52.432Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660079916 workflow=artifact-passing-fnjrp time="2021-08-31T13:59:56.715Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:59:56.716Z" level=info msg="Pod node artifact-passing-fnjrp-7432736 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:59:56.728Z" level=info msg="Created pod: artifact-passing-fnjrp[0].generate-artifact (artifact-passing-fnjrp-7432736)" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:59:56.728Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T13:59:56.747Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660108615 workflow=artifact-passing-fnjrp time="2021-08-31T14:00:06.733Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:06.735Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:06.754Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660108704 workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.721Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.721Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-fnjrp-7432736 templateName=whalesay workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.722Z" level=info msg="Step group node artifact-passing-fnjrp-4055596050 deemed failed: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.722Z" level=info msg="node artifact-passing-fnjrp-4055596050 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.722Z" level=info msg="node artifact-passing-fnjrp-4055596050 message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.722Z" level=info msg="node artifact-passing-fnjrp-4055596050 finished: 2021-08-31 14:00:42.722724904 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.722Z" level=info msg="step group artifact-passing-fnjrp-4055596050 was unsuccessful: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.722Z" level=info msg="Outbound nodes of artifact-passing-fnjrp-7432736 is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.722Z" level=info msg="Outbound nodes of artifact-passing-fnjrp is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.722Z" level=info msg="node artifact-passing-fnjrp phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.722Z" level=info msg="node artifact-passing-fnjrp message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.722Z" level=info msg="node artifact-passing-fnjrp finished: 2021-08-31 14:00:42.722943298 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.722Z" level=info msg="Checking daemoned children of artifact-passing-fnjrp" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.723Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.723Z" level=info msg="Updated message -> child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.723Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.723Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:00:42.741Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660109020 workflow=artifact-passing-fnjrp time="2021-08-31T14:14:03.243Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:03.244Z" level=info msg="Pod node artifact-passing-fnjrp-7432736 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:03.254Z" level=info msg="Created pod: artifact-passing-fnjrp[0].generate-artifact (artifact-passing-fnjrp-7432736)" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:03.254Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:03.271Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660115546 workflow=artifact-passing-fnjrp time="2021-08-31T14:14:13.260Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:13.261Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:13.279Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660115636 workflow=artifact-passing-fnjrp time="2021-08-31T14:14:23.286Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:23.288Z" level=info msg="Workflow step group node artifact-passing-fnjrp-4055596050 not yet completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.409Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.410Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-fnjrp-7432736 templateName=whalesay workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="Step group node artifact-passing-fnjrp-4055596050 deemed failed: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="node artifact-passing-fnjrp-4055596050 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="node artifact-passing-fnjrp-4055596050 message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="node artifact-passing-fnjrp-4055596050 finished: 2021-08-31 14:14:49.411222907 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="step group artifact-passing-fnjrp-4055596050 was unsuccessful: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="Outbound nodes of artifact-passing-fnjrp-7432736 is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="Outbound nodes of artifact-passing-fnjrp is [artifact-passing-fnjrp-7432736]" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="node artifact-passing-fnjrp phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="node artifact-passing-fnjrp message: child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="node artifact-passing-fnjrp finished: 2021-08-31 14:14:49.411365931 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="Checking daemoned children of artifact-passing-fnjrp" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="Updated message -> child 'artifact-passing-fnjrp-7432736' failed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.411Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-fnjrp time="2021-08-31T14:14:49.429Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660115925 workflow=artifact-passing-fnjrp time="2021-08-31T14:16:55.295Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:16:55.295Z" level=info msg="Updated phase -> Running" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:16:55.298Z" level=info msg="Steps node artifact-passing-djjm8 initialized Running" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:16:55.298Z" level=info msg="StepGroup node artifact-passing-djjm8-2975015209 initialized Running" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:16:55.299Z" level=info msg="Pod node artifact-passing-djjm8-187514843 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:16:55.310Z" level=info msg="Created pod: artifact-passing-djjm8[0].generate-artifact (artifact-passing-djjm8-187514843)" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:16:55.310Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:16:55.339Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660116982 workflow=artifact-passing-djjm8 time="2021-08-31T14:17:05.314Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:05.316Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:05.331Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660117073 workflow=artifact-passing-djjm8 time="2021-08-31T14:17:15.335Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:15.337Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.406Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.406Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-djjm8-187514843 templateName=whalesay workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.407Z" level=info msg="Step group node artifact-passing-djjm8-2975015209 deemed failed: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.408Z" level=info msg="node artifact-passing-djjm8-2975015209 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.408Z" level=info msg="node artifact-passing-djjm8-2975015209 message: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.408Z" level=info msg="node artifact-passing-djjm8-2975015209 finished: 2021-08-31 14:17:41.40810134 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.408Z" level=info msg="step group artifact-passing-djjm8-2975015209 was unsuccessful: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.408Z" level=info msg="Outbound nodes of artifact-passing-djjm8-187514843 is [artifact-passing-djjm8-187514843]" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.408Z" level=info msg="Outbound nodes of artifact-passing-djjm8 is [artifact-passing-djjm8-187514843]" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.408Z" level=info msg="node artifact-passing-djjm8 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.408Z" level=info msg="node artifact-passing-djjm8 message: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.408Z" level=info msg="node artifact-passing-djjm8 finished: 2021-08-31 14:17:41.408331268 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.408Z" level=info msg="Checking daemoned children of artifact-passing-djjm8" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.408Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.408Z" level=info msg="Updated message -> child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.408Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.408Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:17:41.422Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660117382 workflow=artifact-passing-djjm8 time="2021-08-31T14:33:13.468Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:13.469Z" level=info msg="Pod node artifact-passing-djjm8-187514843 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:13.479Z" level=info msg="Created pod: artifact-passing-djjm8[0].generate-artifact (artifact-passing-djjm8-187514843)" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:13.480Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:13.498Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660124874 workflow=artifact-passing-djjm8 time="2021-08-31T14:33:23.485Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:23.487Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:23.505Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660124966 workflow=artifact-passing-djjm8 time="2021-08-31T14:33:33.511Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:33.513Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.838Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.838Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-djjm8-187514843 templateName=whalesay workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="Step group node artifact-passing-djjm8-2975015209 deemed failed: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="node artifact-passing-djjm8-2975015209 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="node artifact-passing-djjm8-2975015209 message: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="node artifact-passing-djjm8-2975015209 finished: 2021-08-31 14:33:59.84042281 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="step group artifact-passing-djjm8-2975015209 was unsuccessful: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="Outbound nodes of artifact-passing-djjm8-187514843 is [artifact-passing-djjm8-187514843]" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="Outbound nodes of artifact-passing-djjm8 is [artifact-passing-djjm8-187514843]" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="node artifact-passing-djjm8 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="node artifact-passing-djjm8 message: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="node artifact-passing-djjm8 finished: 2021-08-31 14:33:59.840687014 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="Checking daemoned children of artifact-passing-djjm8" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="Updated message -> child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.840Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:33:59.864Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660125279 workflow=artifact-passing-djjm8 time="2021-08-31T14:45:00.006Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:00.007Z" level=info msg="Pod node artifact-passing-djjm8-187514843 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:00.017Z" level=info msg="Created pod: artifact-passing-djjm8[0].generate-artifact (artifact-passing-djjm8-187514843)" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:00.017Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:00.035Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660130668 workflow=artifact-passing-djjm8 time="2021-08-31T14:45:10.023Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:10.025Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:10.047Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660130760 workflow=artifact-passing-djjm8 time="2021-08-31T14:45:20.052Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:20.054Z" level=info msg="Workflow step group node artifact-passing-djjm8-2975015209 not yet completed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.161Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.162Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-djjm8-187514843 templateName=whalesay workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="Step group node artifact-passing-djjm8-2975015209 deemed failed: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="node artifact-passing-djjm8-2975015209 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="node artifact-passing-djjm8-2975015209 message: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="node artifact-passing-djjm8-2975015209 finished: 2021-08-31 14:45:46.163355789 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="step group artifact-passing-djjm8-2975015209 was unsuccessful: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="Outbound nodes of artifact-passing-djjm8-187514843 is [artifact-passing-djjm8-187514843]" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="Outbound nodes of artifact-passing-djjm8 is [artifact-passing-djjm8-187514843]" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="node artifact-passing-djjm8 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="node artifact-passing-djjm8 message: child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="node artifact-passing-djjm8 finished: 2021-08-31 14:45:46.163518793 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="Checking daemoned children of artifact-passing-djjm8" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="Updated message -> child 'artifact-passing-djjm8-187514843' failed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.163Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-djjm8 time="2021-08-31T14:45:46.201Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660131066 workflow=artifact-passing-djjm8 time="2021-08-31T14:48:31.364Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:48:31.364Z" level=info msg="Updated phase -> Running" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:48:31.366Z" level=info msg="Steps node artifact-passing-pj8v7 initialized Running" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:48:31.366Z" level=info msg="StepGroup node artifact-passing-pj8v7-1355320965 initialized Running" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:48:31.367Z" level=info msg="Pod node artifact-passing-pj8v7-854535231 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:48:31.375Z" level=info msg="Created pod: artifact-passing-pj8v7[0].generate-artifact (artifact-passing-pj8v7-854535231)" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:48:31.375Z" level=info msg="Workflow step group node artifact-passing-pj8v7-1355320965 not yet completed" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:48:31.396Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660132426 workflow=artifact-passing-pj8v7 time="2021-08-31T14:48:41.380Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:48:41.382Z" level=info msg="Workflow step group node artifact-passing-pj8v7-1355320965 not yet completed" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:48:41.395Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660132519 workflow=artifact-passing-pj8v7 time="2021-08-31T14:48:51.400Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:48:51.402Z" level=info msg="Workflow step group node artifact-passing-pj8v7-1355320965 not yet completed" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.530Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.530Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-pj8v7-854535231 templateName=whalesay workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="Step group node artifact-passing-pj8v7-1355320965 deemed failed: child 'artifact-passing-pj8v7-854535231' failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="node artifact-passing-pj8v7-1355320965 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="node artifact-passing-pj8v7-1355320965 message: child 'artifact-passing-pj8v7-854535231' failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="node artifact-passing-pj8v7-1355320965 finished: 2021-08-31 14:49:17.532468701 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="step group artifact-passing-pj8v7-1355320965 was unsuccessful: child 'artifact-passing-pj8v7-854535231' failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="Outbound nodes of artifact-passing-pj8v7-854535231 is [artifact-passing-pj8v7-854535231]" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="Outbound nodes of artifact-passing-pj8v7 is [artifact-passing-pj8v7-854535231]" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="node artifact-passing-pj8v7 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="node artifact-passing-pj8v7 message: child 'artifact-passing-pj8v7-854535231' failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="node artifact-passing-pj8v7 finished: 2021-08-31 14:49:17.532738572 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="Checking daemoned children of artifact-passing-pj8v7" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="Updated message -> child 'artifact-passing-pj8v7-854535231' failed" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.532Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-pj8v7 time="2021-08-31T14:49:17.548Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660132831 workflow=artifact-passing-pj8v7 time="2021-08-31T14:53:20.830Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:53:20.830Z" level=info msg="Updated phase -> Running" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:53:20.833Z" level=info msg="Steps node artifact-passing-5bs6q initialized Running" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:53:20.833Z" level=info msg="StepGroup node artifact-passing-5bs6q-2310228769 initialized Running" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:53:20.833Z" level=info msg="Pod node artifact-passing-5bs6q-1898326195 initialized Pending" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:53:20.892Z" level=info msg="Created pod: artifact-passing-5bs6q[0].generate-artifact (artifact-passing-5bs6q-1898326195)" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:53:20.892Z" level=info msg="Workflow step group node artifact-passing-5bs6q-2310228769 not yet completed" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:53:20.911Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660134823 workflow=artifact-passing-5bs6q time="2021-08-31T14:53:30.899Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:53:30.901Z" level=info msg="Workflow step group node artifact-passing-5bs6q-2310228769 not yet completed" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:53:30.915Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Running resourceVersion=660134912 workflow=artifact-passing-5bs6q time="2021-08-31T14:53:40.920Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:53:40.921Z" level=info msg="Workflow step group node artifact-passing-5bs6q-2310228769 not yet completed" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.060Z" level=info msg="Processing workflow" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.060Z" level=info msg="Pod failed" displayName=generate-artifact namespace=twin-pipelines pod=artifact-passing-5bs6q-1898326195 templateName=whalesay workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.061Z" level=info msg="Step group node artifact-passing-5bs6q-2310228769 deemed failed: child 'artifact-passing-5bs6q-1898326195' failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.062Z" level=info msg="node artifact-passing-5bs6q-2310228769 phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.062Z" level=info msg="node artifact-passing-5bs6q-2310228769 message: child 'artifact-passing-5bs6q-1898326195' failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.062Z" level=info msg="node artifact-passing-5bs6q-2310228769 finished: 2021-08-31 14:54:07.062100084 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.062Z" level=info msg="step group artifact-passing-5bs6q-2310228769 was unsuccessful: child 'artifact-passing-5bs6q-1898326195' failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.062Z" level=info msg="Outbound nodes of artifact-passing-5bs6q-1898326195 is [artifact-passing-5bs6q-1898326195]" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.062Z" level=info msg="Outbound nodes of artifact-passing-5bs6q is [artifact-passing-5bs6q-1898326195]" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.062Z" level=info msg="node artifact-passing-5bs6q phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.062Z" level=info msg="node artifact-passing-5bs6q message: child 'artifact-passing-5bs6q-1898326195' failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.062Z" level=info msg="node artifact-passing-5bs6q finished: 2021-08-31 14:54:07.06231562 +0000 UTC" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.062Z" level=info msg="Checking daemoned children of artifact-passing-5bs6q" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.062Z" level=info msg="Updated phase Running -> Failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.062Z" level=info msg="Updated message -> child 'artifact-passing-5bs6q-1898326195' failed" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.062Z" level=info msg="Marking workflow completed" namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.062Z" level=info msg="Checking daemoned children of " namespace=twin-pipelines workflow=artifact-passing-5bs6q time="2021-08-31T14:54:07.077Z" level=info msg="Workflow update successful" namespace=twin-pipelines phase=Failed resourceVersion=660135225 workflow=artifact-passing-5bs6q ```

The workflow's pods that are problematic:

 ```yaml apiVersion: v1 items: - apiVersion: v1 kind: Pod metadata: annotations: cni.projectcalico.org/podIP: "" cni.projectcalico.org/podIPs: "" workflows.argoproj.io/node-message: timed out waiting for the condition workflows.argoproj.io/node-name: artifact-passing-5bs6q[0].generate-artifact workflows.argoproj.io/template: '{"name":"whalesay","arguments":{},"inputs":{},"outputs":{"artifacts":[{"name":"hello-art","path":"/tmp/hello_world.txt"}]},"metadata":{},"container":{"name":"","image":"docker/whalesay:latest","command":["sh","-c"],"args":["sleep 1; cowsay hello world | tee /tmp/hello_world.txt"],"resources":{}},"archiveLocation":{"archiveLogs":false,"s3":{"endpoint":"minio.minio-ssd.svc.cluster.local:443","bucket":"test","insecure":false,"accessKeySecret":{"name":"minio-key","key":"access_key_id"},"secretKeySecret":{"name":"minio-secret","key":"secret_access_key"},"key":"artifact-passing-5bs6q/artifact-passing-5bs6q-1898326195"}}}' creationTimestamp: "2021-08-31T14:53:20Z" labels: workflows.argoproj.io/completed: "true" workflows.argoproj.io/workflow: artifact-passing-5bs6q name: artifact-passing-5bs6q-1898326195 namespace: twin-pipelines ownerReferences: - apiVersion: argoproj.io/v1alpha1 blockOwnerDeletion: true controller: true kind: Workflow name: artifact-passing-5bs6q uid: f28ff596-914e-4d97-b55f-3de083bd455d resourceVersion: "660135229" uid: 166a07d6-6c0f-4a52-965f-66d5e956eba4 spec: containers: - command: - argoexec - wait env: - name: ARGO_POD_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: metadata.name image: argoproj/argoexec:v2.12.5 imagePullPolicy: IfNotPresent name: wait resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /argo/podmetadata name: podmetadata - mountPath: /var/run/docker.sock name: docker-sock readOnly: true - mountPath: /argo/secret/minio-secret name: minio-secret readOnly: true - mountPath: /argo/secret/minio-key name: minio-key readOnly: true - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: pipeline-runner-token-xmwtk readOnly: true - args: - sleep 1; cowsay hello world | tee /tmp/hello_world.txt command: - sh - -c image: docker/whalesay:latest imagePullPolicy: Always name: main resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: pipeline-runner-token-xmwtk readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: cuda01-p100 preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Never schedulerName: default-scheduler securityContext: {} serviceAccount: pipeline-runner serviceAccountName: pipeline-runner terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - downwardAPI: defaultMode: 420 items: - fieldRef: apiVersion: v1 fieldPath: metadata.annotations path: annotations name: podmetadata - hostPath: path: /var/run/docker.sock type: Socket name: docker-sock - name: minio-secret secret: defaultMode: 420 items: - key: secret_access_key path: secret_access_key secretName: minio-secret - name: minio-key secret: defaultMode: 420 items: - key: access_key_id path: access_key_id secretName: minio-key - name: pipeline-runner-token-xmwtk secret: defaultMode: 420 secretName: pipeline-runner-token-xmwtk status: conditions: - lastProbeTime: null lastTransitionTime: "2021-08-31T14:53:20Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2021-08-31T14:53:20Z" message: 'containers with unready status: [wait main]' reason: ContainersNotReady status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2021-08-31T14:53:20Z" message: 'containers with unready status: [wait main]' reason: ContainersNotReady status: "False" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2021-08-31T14:53:20Z" status: "True" type: PodScheduled containerStatuses: - containerID: docker://b09c5c6b844c40a348effe10b123d4ccdd5b2fba2f703701404d94d3aaa4d713 image: docker/whalesay:latest imageID: docker-pullable://docker/whalesay@sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b lastState: {} name: main ready: false restartCount: 0 started: false state: terminated: containerID: docker://b09c5c6b844c40a348effe10b123d4ccdd5b2fba2f703701404d94d3aaa4d713 exitCode: 0 finishedAt: "2021-08-31T14:53:25Z" reason: Completed startedAt: "2021-08-31T14:53:24Z" - containerID: docker://8005ed619702ef75e17413e419ce0504c97d7f0ee43d96552213d7a22a90483d image: argoproj/argoexec:v2.12.5 imageID: docker-pullable://argoproj/argoexec@sha256:1841ef4c44e582f66af82091fd15eeb493faddb359a54fb75087c993b5404e0b lastState: {} name: wait ready: false restartCount: 0 started: false state: terminated: containerID: docker://8005ed619702ef75e17413e419ce0504c97d7f0ee43d96552213d7a22a90483d exitCode: 1 finishedAt: "2021-08-31T14:53:57Z" message: timed out waiting for the condition reason: Error startedAt: "2021-08-31T14:53:22Z" hostIP: 172.18.60.241 phase: Failed podIP: 10.42.6.16 podIPs: - ip: 10.42.6.16 qosClass: BestEffort startTime: "2021-08-31T14:53:20Z" kind: List metadata: resourceVersion: "" selfLink: "" ```

Logs from in your workflow's wait container:

 ``` time="2021-08-31T14:53:40.563Z" level=info msg="Saving from /tmp/argo/outputs/artifacts/hello-art.tgz to s3 (endpoint: minio.minio-ssd.svc.cluster.local:443, bucket: test, key: artifact-passing-5bs6q/artifact-passing-5bs6q-1898326195/hello-art.tgz)" time="2021-08-31T14:53:40.576Z" level=warning msg="Failed to put file: Get https://minio.minio-ssd.svc.cluster.local/test/?location=: x509: certificate signed by unknown authority" time="2021-08-31T14:53:57.022Z" level=info msg="S3 Save path: /tmp/argo/outputs/artifacts/hello-art.tgz, key: artifact-passing-5bs6q/artifact-passing-5bs6q-1898326195/hello-art.tgz" time="2021-08-31T14:53:57.022Z" level=info msg="Creating minio client minio.minio-ssd.svc.cluster.local:443 using static credentials" time="2021-08-31T14:53:57.022Z" level=info msg="Saving from /tmp/argo/outputs/artifacts/hello-art.tgz to s3 (endpoint: minio.minio-ssd.svc.cluster.local:443, bucket: test, key: artifact-passing-5bs6q/artifact-passing-5bs6q-1898326195/hello-art.tgz)" time="2021-08-31T14:53:57.036Z" level=warning msg="Failed to put file: Get https://minio.minio-ssd.svc.cluster.local/test/?location=: x509: certificate signed by unknown authority" time="2021-08-31T14:53:57.036Z" level=error msg="executor error: timed out waiting for the condition" time="2021-08-31T14:53:57.036Z" level=info msg="Killing sidecars" time="2021-08-31T14:53:57.041Z" level=info msg="Alloc=8616 TotalAlloc=24264 Sys=70848 NumGC=7 Goroutines=7" time="2021-08-31T14:53:57.052Z" level=fatal msg="timed out waiting for the condition" ```

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

alexec commented 3 years ago

The certificate you're MinIO instance is running with in not recognised because it is not in the certificate chain. For non-production, try insecure: true.

mbu93 commented 3 years ago

I actually used this setting, but if I got this right, setting insecure: true will enforce http communication, which is sadly no longer an option when using minio operator. What I'd need would be an option to allow insecure certificates. Does using insecure: true provide such behaviour in argo's latest version?

alexec commented 3 years ago

Huh? Maybe prefix the endpoint with “https”?

mbu93 commented 3 years ago

Nope. This sadly results in

level=warning msg="Failed to create new S3 client: address https://minio.minio-ssd.svc.cluster.local:443: too many colons in address"
alexec commented 3 years ago

Sorry. I misunderstood your previous comment. Insecure - I thought it ignores invalid certificates. If you’ve not tested it, give it a go. If it does not work, then I don’t think there is a work-around. Either you’ll need to run MinIO using HTTP (data transits unencrypted.), or build your own executor image and import your certificates on to it (hard).

hofalk commented 3 years ago

We are also struggling with the exact same issue and running Minio with HTTP is unfortunately not an option in our corporate setup.

Maybe this can be easily fixed by allowing for some sort of executorExtraMounts configuration setting inside the workflow-controller-configmap, where one could add a reference to an external secret/configmap and place it under the main certificates path inside the executor so it gets picked up by the environment when talking to the minio backend? Not sure if it is that easy, as different Executors probably rely on different mechanisms/locations for their trusted CAs...

jaredmcqueen commented 3 years ago

Experiencing the same problem.

Would init containers be able mount the CA-trusted certificate prior to workflow inputs/outputs/artifacts?

Another idea would be patching / mounting the correct CA-signed certs to the workflow-server container prior to workflows running? Not that familiar with argo workflows yet...

One more thing - a CA-signed cert is actually already present on the workflow pods that get spun up - the path is /var/run/secrets/kubernetes.io/serviceaccount/ca.crt. If only we could have the S3 handler use that to verify.

jhart99 commented 2 years ago

One work around for this issue is to add an environment variable to the argoexec to point to kubernetes certificate authority which is injected into the container. This works because go looks for this environment variable to find SSL certs.

I did this by building a custom container for argoexec with this Dockerfile.

FROM quay.io/argoproj/argoexec:latest
ENV SSL_CERT_FILE=/run/secrets/kubernetes.io/serviceaccount/ca.crt

Then tagging this container and pushing it to our local repository. To use this, you can edit your worflow-controller deployment replacing the executor with the newly built executor.

# ...
spec:
  template:
    spec:
      containers:
      - args:
        - --configmap
        - workflow-controller-configmap
        - --executor-image
        - myspecial/argoexec
        - --namespaced
# ...

This will let Argo Workflows successfully pass artifacts back and forth with Minio (using the newer version of Minio Operator which disallows insecure connections) over TLS.

cotjoey commented 2 years ago

Experiencing the same issue with attempting secure HTTPS communications between Argo Workflow v3.1.13 & MinIO v4.2.7. The newest MinIO now runs on port 443. I was hoping for a quick resolution such as a flag in the ConfigMap "Skip TLS Verify". I will attempt @jhart99 's solution - both application are running on Kubernetes, so hopefully that'll do the trick. Doing a quick cURL on the MinIO endpoint reveals the following certificate:


* Server certificate:
*  subject: O=system:nodes; CN=system:node:*.minio-tenant-1-hl.minio-tenant.svc.cluster.local
*  start date: Dec  8 21:08:55 2021 GMT
*  expire date: Dec  8 21:08:55 2022 GMT
*  issuer: CN=UCP Client Root CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.```
cotjoey commented 2 years ago

Update: @jhart99 's solution worked for me (adding ENV SSL_CERT_FILE=/run/secrets/kubernetes.io/serviceaccount/ca.crt to the Docker image).

alexec commented 2 years ago

@cotjoey do you think this would work as a permanent solution for all users? One we should bake into our image? If so, would you like to raise a PR:

https://github.com/argoproj/argo-workflows/blob/master/Dockerfile

jhart99 commented 2 years ago

@cotjoey do you think this would work as a permanent solution for all users? One we should bake into our image? If so, would you like to raise a PR:

https://github.com/argoproj/argo-workflows/blob/master/Dockerfile

This works and I suggested it as a temporary workaround. I think the better permanent solution would be to add the environment variable, if required, to the workflow-controller-configmap:

data:
  executor:
    env:
      - name: SSL_CERT_FILE  # edited by agilgur5: should be FILE, not DIR
        value: /run/secrets/kubernetes.io/serviceaccount/ca.crt

I haven't tested this method yet, but I believe this would work according to the documentation for the configmap. This means that no changes to the argoexec container would be necessary and there wouldn't be the chance for someone to inject a trusted cert when not running in kubernetes.

I could work on (in a few days) making a PR for the getting started example so that it works out of the box and maybe this could be highlighted in some documentation.

cotjoey commented 2 years ago

I just tested this environment variable inside the workflow-controller-configmap (Note that the variable name is SSL_CERT_FILE, not SSL_CERT_DIR - it failed with SSL_CERT_DIR) and it works!:

data:
  executor:
    env:
      - name: SSL_CERT_FILE
        value: /run/secrets/kubernetes.io/serviceaccount/ca.crt

Now I am getting the same error, but from the argo-server deployment logs, when I click on "...logs from the artifacts.":

time="2021-12-14T20:25:15.079Z" level=info msg="Creating minio client minio.minio-tenant.svc.cluster.local:443 using static credentials"
time="2021-12-14T20:25:15.079Z" level=info msg="Getting from s3 (endpoint: minio.minio-tenant.svc.cluster.local:443, bucket: argo-workflow, key: pod-hppzw/pod-4165558653/main.log) to /tmp/artifact884264937"

In the browser it shows:

failed to get file: Get "https://minio.minio-tenant.svc.cluster.local/argo-workflow/?location=": x509: certificate signed by unknown authority

I'll attempt to use the same environment variable in the argo-server container to see what happens. Will keep you posted.

alexec commented 2 years ago

Reviewing the usage of ca.crt, it seems it is meant for accessing the Kubernetes API. I'm surprised it worked.

jhart99 commented 2 years ago

If you use the minio operator, the certificate for the minio endpoint is signed by the kubernetes cluster CA. From the minio documentation:

MinIO Tenants deploy with TLS enabled by default, where the MinIO Operator uses the Kubernetes certificates.k8s.io API to generate the required x.509 certificates. Each certificate is signed using the Kubernetes Certificate Authority (CA) configured during cluster deployment. While Kubernetes mounts this CA on Pods in the cluster, Pods do not trust that CA by default. You must copy the CA to a directory such that the update-ca-certificates utility can find and add it to the system trust store to enable validation of MinIO TLS certificates: cp /var/run/secrets/kubernetes.io/serviceaccount/ca.crt /usr/local/share/ca-certificates/ update-ca-certificates

Using this environment variable lets the executor trust the kubernetes CA (and by extension the minio endpoint) without having to run update-ca-certificates.

cotjoey commented 2 years ago

The other error I am trying to fix, didn't resolve itself by using the environment variable. I tried many different things in the argo-server deployment, but nothing yielded positive results:

As a reminder, the error I get in the Argo Workflow UI:

failed to get file: Get "https://minio.minio-tenant.svc.cluster.local/argo-workflow/?location=": x509: certificate signed by unknown authority

Things I've tried in the argo-server deployment YAML:

spec.template.spec.containers.env:
- name: ARGO_INSECURE_SKIP_VERIFY
  value: "true"
- name: SSL_CERT_FILE
  value: /run/secrets/kubernetes.io/serviceaccount/ca.crt
spec.template.spec.containers.args:
- server
- --secure
- --insecure-skip-verify

--insecure-skip-tls-verify is an invalid option, the container crashes with an error, even if it is listed as a valid command-line parameter in the --help output:

spec.template.spec.containers.args:
- server
- --secure
- --insecure-skip-verify
- --insecure-skip-tls-verify

Any thoughts?

cotjoey commented 2 years ago

@jhart99 - I just realized you add a comment about how to add ca.crt to the host system CA certificates store.

I'll attempt to do that on my host systems and see what happens. I am not familiar with those certificates, so if anyone can chime in, it'd be appreciated.

I don't know where that original ca.crt lives on the host system (the one that is mounted in the pod), so I'll probably have to spin up a temporary pod and cp /run/secrets/kubernetes.io/serviceaccount/ca.crt from inside the pod to /usr/local/share/ca-certificates/ on my host system and run update-ca-certificates

cotjoey commented 2 years ago

This is what I did to fix the other issue (I am on Red Hat 8):

I inspected the Dockerfile of the argocli (https://hub.docker.com/r/argoproj/argocli/dockerfile) and found that it was picking up the certificates from /etc/ssl/certs in the container.

So I mounted a hostPath volume from the host (/etc/pki/tls/certs/ca-bundle.crt) to /etc/ssl/certs in the container.

Before that I had to add the Kubernetes CA to /etc/pki/tls/certs/ca-bundle.crt, the ones in /run/secrets/kubernetes.io/serviceaccount/ca.crt

The configuration looks like this in my argo-server deployment YAML:

     volumeMounts:
        - mountPath: /etc/ssl/certs/ca-bundle.crt
          name: host-file
          readOnly: true
    volumes:
      - name: host-file
        hostPath:
          path: /etc/pki/tls/certs/ca-bundle.crt
          type: File
alexec commented 2 years ago

@cotjoey seem like the right solution to me?

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

mxm-tr commented 2 years ago

Hi, it seems like this solution would fit in case the minio certificate has been signed by the kube CA, is that correct?

However if you're luck (like me) and your S3 SSL cert has been signed by some other root ca, I don't see any alternative to rebuilding your own version of the executor image as suggested in the previous comment: https://github.com/argoproj/argo-workflows/issues/6635#issuecomment-974487984

One possible feature that could avoid a rebuild would be to add more customization options to the executor container, such as additional volumeMounts to add custom certificates. Currently I don't think this is supported.

EDIT: It looks like specifying custom resources on the executor is now supported: https://github.com/argoproj/argo-workflows/blob/92b3ef27af7a7e6b930045e95072a47c8745b1d3/docs/workflow-controller-configmap.yaml#L214-L237

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

edevash commented 2 years ago

Any roadmap to bring in support for TLS based MINIO Endpoint? All we need is to provide certificate for validation... We have an external generated certs, in which MINIO is been hosted. Now that rootca, needs to be available, not at build time, but at execution time. Also we dont have provision to override the executor command line, which can at least enable us manually install the rootca, during startup in the container. If Any roadmap to bring in this support?

mxm-tr commented 2 years ago

@edevash have you tried mounting the SSL certificates under /etc/pki/tls/certs/ca-bundle.crt using a Kubernetes secret and a volumeMount? You may be able to do that in the Workflow Contoller configmap, under executor:

edevash commented 2 years ago

@edevash have you tried mounting the SSL certificates under /etc/pki/tls/certs/ca-bundle.crt using a Kubernetes secret and a volumeMount? You may be able to do that in the Workflow Contoller configmap, under executor:

Did you reference some thing like below?

executor:
  imagePullPolicy: Always
  volumes:
  - name: sip-ca
    secret:
      secretName: eric-sec-sip-tls-trusted-root-cert
      items:
      - key: ca.crt
        path: ca-bundle.crt
  volumeMounts:
  - name: sip-ca
    mountPath: /etc/pki/tls/certs

If so, it didn't work. I am on argo 3.2.9. was any thing introduced later. On inspecting the pod, this volumes did not appear...

mxm-tr commented 2 years ago

If so, it didn't work. I am on argo 3.2.9. was any thing introduced later. On inspecting the pod, this volumes did not appear...

Ok so it looks like it is not implemented then, this would probably need something similar to https://github.com/argoproj/argo-workflows/issues/7390, to mount custom volumes on the executor.

jinnjwu commented 2 years ago

we hit same issue, issue is happening in argoexec image, our solution:

FROM argoproj/argoexec:v3.3.4
COPY CA.cer /user/local/share/ca-certificates/ca.crt
RUN update-ca-certificates
stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

rmathagiarun commented 2 years ago

Similar to the issue mentioned by @edevash , we have external generated Certs and rootca that needs to be available at execution time and not at build time.

We are planning to use the following workaround done on top of workflow/controller/workflowpod.go,

  1. Add an additional mount volume(/usr/local/share/ca-certificates). This volume will be pointed to a secret file which contains external generated CA cert.
  2. Update both init and wait containers to run update-ca-certificates before executing argoexec [init or wait].

We have validated this in our setup and it seems to work. Though, not straightforward, this helps mount CA certs at execution time.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If this is a mentoring request, please provide an update here. Thank you for your contributions.

stale[bot] commented 2 years ago

This issue has been closed due to inactivity. Feel free to re-open if you still encounter this issue.

jinnjwu commented 1 year ago

we have same issue in 3.4.2

cmontemuino commented 1 year ago

Solution from https://github.com/argoproj/argo-workflows/issues/6635#issuecomment-1127604348 does not with newer versions. Starting on v3.4.0, the argoproj/argoexec image is based on gcr.io/distroless/static. Hence, running update-ca-certificates is not possible anymore.

Just in case it's helpful for others, this is how we had to do it very recently:

ARG BASE_IMAGE=<acr-endpoint>/argoproj/argoexec:v3.4.9

FROM ${BASE_IMAGE}
COPY CA.cer /etc/ssl/certs/ca.crt
ENV SSL_CERT_FILE=/etc/ssl/certs/ca.crt

Our k8s clusters do not have internet access, thus using our own cert makes sense to us.

gbrochard-pro commented 7 months ago

Hi,

I am facing the same issue with my argoexec container. I tried building my own image with my ca certificate file but I can't find where I need to setup argo workflow controiller to use my image for argoexec ? How did you do guys ? @cmontemuino and @jinnjwu

agilgur5 commented 7 months ago

I can't find where I need to setup argo workflow controiller to use my image for argoexec ?

Pass the --executor-image flag to the Controller per https://github.com/argoproj/argo-workflows/issues/6635#issuecomment-974487984

gbrochard-pro commented 7 months ago

Hello, when i edit my deployment it says that: error-workflow-controller I am editing my deployment with these lines: config-deploy-workflow-controller

gbrochard-pro commented 7 months ago

I don't know what key it is talking about.

gbrochard-pro commented 7 months ago

It is working now ! I had to patch my deployment instead of editing it.