search

wave-k8s / wave

Kubernetes configuration tracking controller
Apache License 2.0
661 stars 81 forks source link

wave increases the number of replica defined #76

Closed cscetbon closed 4 years ago

cscetbon commented 4 years ago

I'm testing wave and it works fine when there are no issues with a container. However when a container fails to start I see multiple containers being created. Is it expected ? why does wave say the config changed when it did not ? Is it because the pod died ?

Here are the events

0s          Normal    AddWatch            configmap/app-init                             Adding watch for ConfigMap app-init
0s          Normal    ScalingReplicaSet   deployment/mypod              Scaled up replica set mypod-55bcb6f4c8 to 1
0s          Normal    ConfigChanged       deployment/mypod              Configuration hash updated to 0a0dcef966ea2af06283c6a7fd21c89e5c0469dd9c42894cad2cc1bcae1cca64
0s          Normal    ScalingReplicaSet   deployment/mypod              Scaled up replica set mypod-699c7876f5 to 1
0s          Normal    SuccessfulCreate    replicaset/mypod-55bcb6f4c8   Created pod: mypod-55bcb6f4c8-pvcp8
0s          Normal    Scheduled           pod/mypod-55bcb6f4c8-pvcp8    Successfully assigned myns/mypod-55bcb6f4c8-pvcp8 to ip-xx-xx-xx-xx.compute.internal
0s          Normal    SuccessfulCreate    replicaset/mypod-699c7876f5   Created pod: mypod-699c7876f5-c5288
0s          Normal    Scheduled           pod/mypod-699c7876f5-c5288    Successfully assigned myns/mypod-699c7876f5-c5288 to ip-xx-xx-xx-xx.compute.internal
0s          Normal    Pulled              pod/mypod-699c7876f5-c5288    Container image "repo/app-init:0.1" already present on machine
0s          Normal    Created             pod/mypod-699c7876f5-c5288    Created container
0s          Normal    Pulled              pod/mypod-55bcb6f4c8-pvcp8    Container image "repo/app-init:0.1" already present on machine
0s          Normal    Created             pod/mypod-55bcb6f4c8-pvcp8    Created container
0s          Normal    Started             pod/mypod-699c7876f5-c5288    Started container
0s          Normal    Started             pod/mypod-55bcb6f4c8-pvcp8    Started container
0s          Normal    Pulling             pod/mypod-55bcb6f4c8-pvcp8    pulling image "repo/app:0.1"
0s          Normal    Pulling             pod/mypod-699c7876f5-c5288    pulling image "repo/app:0.1"
0s          Normal    Pulled              pod/mypod-699c7876f5-c5288    Successfully pulled image "repo/app:0.1"
0s          Normal    Pulled              pod/mypod-55bcb6f4c8-pvcp8    Successfully pulled image "repo/app:0.1"
0s          Normal    Created             pod/mypod-699c7876f5-c5288    Created container
0s          Normal    Created             pod/mypod-55bcb6f4c8-pvcp8    Created container
0s          Warning   Failed              pod/mypod-55bcb6f4c8-pvcp8    Error: failed to start container "app": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \".....\": no such file or directory": unknown
0s          Warning   Failed              pod/mypod-699c7876f5-c5288    Error: failed to start container "app": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \".....\": no such file or directory": unknown
0s          Normal    Pulling             pod/mypod-55bcb6f4c8-pvcp8    pulling image "repo/app:0.1"
0s          Normal    Pulling             pod/mypod-699c7876f5-c5288    pulling image "repo/app:0.1"
0s          Normal    Pulled              pod/mypod-699c7876f5-c5288    Successfully pulled image "repo/app:0.1"
0s          Normal    Pulled              pod/mypod-55bcb6f4c8-pvcp8    Successfully pulled image "repo/app:0.1"
0s          Normal    Created             pod/mypod-699c7876f5-c5288    Created container
0s          Normal    Created             pod/mypod-55bcb6f4c8-pvcp8    Created container
0s          Warning   Failed              pod/mypod-699c7876f5-c5288    Error: failed to start container "app": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \".....\": no such file or directory": unknown
0s          Warning   Failed              pod/mypod-55bcb6f4c8-pvcp8    Error: failed to start container "app": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \".....\": no such file or directory": unknown
JoelSpeed commented 4 years ago

Hey @cscetbon, from the events, I'm failing at the moment to see anything that's related to Wave, could you possibly reproduce this and post some logs from Wave while this is happening?

Could you also confirm that the replica count is increasing in the deployment as you have suggested? This seems like a very strange behaviour that I would not expect Wave to be able to do

cscetbon commented 4 years ago

@JoelSpeed I think you're right, it must come from the deployment itself. As Wave only adds the annotation with the checksum which triggers a restart of the deployment I also can't see how it could be the culprit here. Thanks