main container output log sequence is incorrect ,and artifact log output sequence is incorrect - Githubissues

argoproj / argo-workflows

Workflow Engine for Kubernetes

https://argo-workflows.readthedocs.io/

Apache License 2.0

15.07k stars 3.2k forks source link

main container output log sequence is incorrect ,and artifact log output sequence is incorrect #12155

Open wisonzhu opened 1 year ago

wisonzhu commented 1 year ago

Pre-requisites

[X] I have double-checked my configuration
[X] I can confirm the issues exists when I tested with :latest
[ ] I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

view artifact log :

view main container output:

view vm log:

Version

all version

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

Execute a git clone of a project  or any shell script in shell terminal ,view standard output sequence

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

mozarik commented 1 year ago

encountering this issue also

tooptoop4 commented 1 year ago

can u mention the min and max versions u tried on?

wisonzhu commented 1 year ago

can u mention the min and max versions u tried on?

etc v3.4.11, v3.4.12(max), v3.2.3(min)

tooptoop4 commented 1 week ago

this is caused by stdout, stderr being separate and not interleaved by timestamp https://github.com/argoproj/argo-workflows/blob/4742e9dd6ab2b797d32cb0953849fdcfe82ea325/cmd/argoexec/commands/emissary.go#L246-L256 some posts like https://www.reddit.com/r/golang/comments/ep02yz/concurrency_problem_when_writing_to_stdout_and/ suggest to use https://pkg.go.dev/os/exec#Cmd.CombinedOutput but that would not allow real-time log streaming