Argo workflow Pod status phase coming "Succeeded" when "no space left on device"

[RATANAJANGIR]

Pre-requisites

• [X] I have double-checked my configuration
• [X] I can confirm the issues exists when I tested with :latest
• [ ] I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

Hi I need help here, I am running "argo submit workflow.yaml ........" command. Here when pod is running phase issue is came with "no space left on device" after that pod phase is succeeded and here few logs of command get displayed from "workflow.yaml" argo workflow template.

here is full logs of pod

time="2023-07-18T13:10:20.175Z" level=error msg="unable to decode or infer timestamp from log line: parsing time \"failed\" as \"2006-01-02T15:04:05Z07:00\": cannot parse \"failed\" as \"2006\"" namespace=dpsworkflow podName=edp-airflow-app-feature-upgrade-packages-workflow-180723130647-dps-buildpack-3787918803 workflow=edp-airflow-app-feature-upgrade-packages-workflow-180723130647 edp-airflow-app-feature-upgrade-packages-workflow-180723130647-dps-buildpack-3787918803: Warning: Platform requested deprecated API '0.3' edp-airflow-app-feature-upgrade-packages-workflow-180723130647-dps-buildpack-3787918803: Warning: CNB_PLATFORM_API is unset; using Platform API version '0.3' edp-airflow-app-feature-upgrade-packages-workflow-180723130647-dps-buildpack-3787918803: CNB_PLATFORM_API should be set to avoid breaking changes when upgrading the lifecycle edp-airflow-app-feature-upgrade-packages-workflow-180723130647-dps-buildpack-3787918803: failed to watch file "/var/log/pods/dpsworkflow_edp-airflow-app-feature-upgrade-packages-workflow-180723130647-dps-buildpack-3787918803_1f6aa596-715d-4af7-8356-1935bf145a42/main/0.log": no space left on device Finishing: Branches

Version

v3.4.8

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: dps-buildpack-template
spec:
  templates:
    - name: dps-buildpack
      inputs:
        parameters:
          - name: repo_name
          - name: build_subpath
          - name: buildpack-version
      script:
        imagePullPolicy: "IfNotPresent"
        image: "{{`xxxxxxxxxxx:{{inputs.parameters.buildpack-version}}`}}"
        resources:
          limits:
            cpu: 500m
            memory: 2Gi
          requests:
            cpu: 1m
            memory: 300Mi
        securityContext:
          privileged: true
        envFrom:
          - secretRef:
              name: workflow-secrets
        volumeMounts:
          - name: workdir
            mountPath: /home/argo       
        command: [ "/bin/bash" ]
        source: |
          cd /home/argo/
          /cnb/lifecycle/detector -app {{`{{inputs.parameters.repo_name}}`}}/{{`{{inputs.parameters.build_subpath}}`}}
          /cnb/lifecycle/builder -app {{`{{inputs.parameters.repo_name}}`}}/{{`{{inputs.parameters.build_subpath}}`}}

Logs from the workflow controller

status=`echo $(argo get $(jobName) -n dpsworkflow -o json | jq '.status.phase' -r)`

here status coming Succeeded!

but it should be failed.

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[terrytangyuan]

Could you paste your controller log and live workflow manifest?

[sarabala1979]

@RATANAJANGIR are you using a resource template? Can you try to request more disk space for the pod using VPC?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[tooptoop4]

yo @RATANAJANGIR where da controller log