Closed jhart99 closed 6 years ago
@jhart99 thanks for reporting this. I'm going to have a look.
@jhart99 having hard time reproduce the problem. Do you have logs for this? For example, running the controller with --v=6
? Keep trying, anyway, Thanks again
I can get this to happen pretty consistently with the codefresh/workflow-controller docker container. I recompiled from the latest sources and cannot reproduce this, so I'll close it.
@jhart99 obviously we don't control codefresh/workflow-controller
instance. Thanks
The latest version in the docker repository (codefresh/workflow-controller:latest referenced by the k8s deployment) has a segfault if there are stale workflow jobs left in kubernetes. Workflow-controller will keep crashing and restarting after it gets into this state.
Steps
I was able to resolve this by purging all of the jobs that workflow-controller made before restarting it. All the jobs were in the same context so I didn't filter them from any other jobs.
Error from the logs: