Workflow never terminates

Is this a BUG REPORT or FEATURE REQUEST?: BUG

What happened: My workflow never terminate -> https://github.com/argoproj/argo/blob/master/examples/sidecar-dind.yaml

What you expected to happen: It should terminate on success or failure.

How to reproduce it (as minimally and precisely as possible): argo submit https://raw.githubusercontent.com/argoproj/argo/master/examples/sidecar-dind.yaml

Anything else we need to know?:

Environment:

Argo version:

$ argo version
argo: v2.3.0
BuildDate: 2019-05-20T22:11:23Z
GitCommit: 88fcc70dcf6e60697e6716edc7464a403c49b27e
GitTreeState: clean
GitTag: v2.3.0
GoVersion: go1.11.5
Compiler: gc
Platform: darwin/amd64

Kubernetes version :

$ kubectl version -o yaml
clientVersion:
buildDate: "2019-06-06T01:44:30Z"
compiler: gc
gitCommit: 5e53fd6bc17c0dec8434817e69b04a25d8ae0ff0
gitTreeState: clean
gitVersion: v1.14.3
goVersion: go1.12.5
major: "1"
minor: "14"
platform: darwin/amd64
serverVersion:
buildDate: "2019-06-19T20:48:40Z"
compiler: gc
gitCommit: f53039cc1e5295eed20969a4f10fb6ad99461e37
gitTreeState: clean
gitVersion: v1.12.8-gke.10
goVersion: go1.10.8b4
major: "1"
minor: 12+
platform: linux/amd64

Other debugging information (if applicable):

workflow result:


$ argo get <workflowname>
Name:                sidecar-dind-6jm5v
Namespace:           kintohub
ServiceAccount:      default
Status:              Running
Created:             Mon Sep 02 18:42:52 +0800 (3 minutes ago)
Started:             Mon Sep 02 18:42:52 +0800 (3 minutes ago)
Duration:            3 minutes 9 seconds

STEP PODNAME DURATION MESSAGE ● sidecar-dind-6jm5v sidecar-dind-6jm5v 3m

- executor logs:

$ kubectl logs -c init $ kubectl logs -c wait time="2019-09-02T10:42:53Z" level=info msg="Creating a docker executor" time="2019-09-02T10:42:53Z" level=info msg="Executor (version: v2.3.0, build_date: 2019-05-20T22:10:54Z) initialized (pod: kintohub/sidecar-dind-6jm5v) with template:\n{\"name\":\"dind-sidecar-example\",\"inputs\":{},\"outputs\":{},\"metadata\":{},\"container\":{\"name\":\"\",\"image\":\"docker:17.10\",\"command\":[\"sh\",\"-c\"],\"args\":[\"until docker ps; do sleep 3; done; docker run --rm debian:latest cat /etc/os-release\"],\"env\":[{\"name\":\"DOCKER_HOST\",\"value\":\"127.0.0.1\"}],\"resources\":{}},\"sidecars\":[{\"name\":\"dind\",\"image\":\"docker:18.09.4-dind\",\"resources\":{},\"securityContext\":{\"privileged\":true},\"mirrorVolumeMounts\":true}]}" time="2019-09-02T10:42:53Z" level=info msg="Waiting on main container" time="2019-09-02T10:42:53Z" level=error msg="executor error: Failed to establish pod watch: unknown (get pods)\ngithub.com/argoproj/argo/errors.Wrap\n\t/go/src/github.com/argoproj/argo/errors/errors.go:88\ngithub.com/argoproj/argo/errors.InternalWrapErrorf\n\t/go/src/github.com/argoproj/argo/errors/errors.go:78\ngithub.com/argoproj/argo/workflow/executor.(WorkflowExecutor).waitMainContainerStart\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:885\ngithub.com/argoproj/argo/workflow/executor.(WorkflowExecutor).Wait\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:856\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:32\ngithub.com/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngithub.com/spf13/cobra.(Command).execute\n\t/go/src/github.com/spf13/cobra/command.go:766\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/src/github.com/spf13/cobra/command.go:852\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/src/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1333" time="2019-09-02T10:42:53Z" level=info msg="Killing sidecars" time="2019-09-02T10:42:53Z" level=warning msg="Failed to get pod 'sidecar-dind-6jm5v': pods \"sidecar-dind-6jm5v\" is forbidden: User \"system:serviceaccount:kintohub:default\" cannot get resource \"pods\" in API group \"\" in the namespace \"kintohub\"" time="2019-09-02T10:42:53Z" level=error msg="executor error: pods \"sidecar-dind-6jm5v\" is forbidden: User \"system:serviceaccount:kintohub:default\" cannot get resource \"pods\" in API group \"\" in the namespace \"kintohub\"\ngithub.com/argoproj/argo/errors.Wrap\n\t/go/src/github.com/argoproj/argo/errors/errors.go:88\ngithub.com/argoproj/argo/errors.InternalWrapError\n\t/go/src/github.com/argoproj/argo/errors/errors.go:71\ngithub.com/argoproj/argo/workflow/executor.(WorkflowExecutor).getPod\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:620\ngithub.com/argoproj/argo/workflow/executor.(WorkflowExecutor).KillSidecars\n\t/go/src/github.com/argoproj/argo/workflow/executor/executor.go:1061\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:37\ngithub.com/argoproj/argo/cmd/argoexec/commands.NewWaitCommand.func1\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:16\ngithub.com/spf13/cobra.(Command).execute\n\t/go/src/github.com/spf13/cobra/command.go:766\ngithub.com/spf13/cobra.(Command).ExecuteC\n\t/go/src/github.com/spf13/cobra/command.go:852\ngithub.com/spf13/cobra.(Command).Execute\n\t/go/src/github.com/spf13/cobra/command.go:800\nmain.main\n\t/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:17\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1333" time="2019-09-02T10:42:53Z" level=info msg="No output parameters" time="2019-09-02T10:42:53Z" level=info msg="No output artifacts" time="2019-09-02T10:42:53Z" level=info msg="Alloc=3754 TotalAlloc=9554 Sys=70590 NumGC=3 Goroutines=5"

- workflow-controller logs:

$ kubectl logs -n argo $(kubectl get pods -l app=workflow-controller -n argo -o name)

argoproj / argo-workflows

Workflow never terminates #1590