Closed ghodss closed 5 years ago
cc @ironcladlou @bgrant0607
Whats the use case for restarting the pods without any changes to the spec?
Note that there wont be any way to rollback the change if pods started failing when they were restarted.
Whenever services get into some wedged or undesirable state (maxed out connections and are now stalled, bad internal state, etc.). It's usually one of the first troubleshooting steps if a service is seriously misbehaving.
If the first pod fails as it is restarted, I would expect it to cease continuing or continue retrying to start the pod.
Also, a rolling restart with no spec change reallocates pods across the cluster.
However, I would also like the ability to do this without rescheduling the pods. That could be a rolling label change, but may pick up new dynamic config or clear the local file state.
On Wed, Sep 2, 2015 at 12:01 AM, Sam Ghods notifications@github.com wrote:
Whenever services get into some wedged or undesirable state (maxed out connections and are now stalled, bad internal state, etc.). It's usually one of the first troubleshooting steps if the service is seriously misbehaving.
If the first pod fails as it is restarted, I would expect it to cease continuing or continue retrying to start the pod.
β Reply to this email directly or view it on GitHub https://github.com/kubernetes/kubernetes/issues/13488#issuecomment-136931790 .
Clayton Coleman | Lead Engineer, OpenShift
@smarterclayton Is that like my option 2 listed above? Though why would labels be changed?
Re. wedged: That's what liveness probes are for.
Re. rebalancing: see #12140
If we did support this, I'd lump it with #9043 -- the same mechanism is required.
I suppose this would more be for a situation where the pod is alive and responding to checks but still needs to be restarted. One example is a service with an in-memory cache or internal state that gets corrupted and needs to be cleared.
I feel like asking for an application to be restarted is a fairly common use case, but maybe I'm incorrect.
Corruption would just be one pod, which could just be killed and replaced by the RC.
The other case mentioned offline was to re-read configuration. That's dangerous to do implicitly, because restarts for any reason would cause containers to load the new configuration. It would be better to do a rolling update to push a new versioned config reference (e.g. in an env var) to the pods. This is similar to what motivated #1353.
@bgrant0607 have we decided that we don't want to do this?
@gmarek Nothing, for now. Too many things are underway already.
Can we have a post v1.1
milestone (or something) for the stuff that we deem important, but we lack people to fix them straight away?
I would be a fan of this feature as well, you don't want to be forced to switch tags for every minor update you want to roll out.
I'm a fan of this feature. Use case: Easily upgrade all the pods to use a newly-pushed docker image (with imagePullPolicy: Always
). I currently use a bit of a hacky solution: Rolling-update with or without the :latest
tag on the image name.
Another use case: Updating secrets.
I'd really like to see this feature. We run node apps on kubernetes and currently have certain use cases where we restart pods to clear in app pseudo caching.
Here's what I'm doing for now:
kubectl get pod | grep 'pod-name' | cut -d " " -f1 - | xargs -n1 -P 10 kubectl delete pod
This deletes pods 10 at a time and works well in a replication controller set up. It does not address any concerns like pod allocation or new pods failing to start. It's a quick solution when needed.
I would really like to be able to do a rolling restart. The main reason is we will feed ENV variables into pods using ConfigMap and then if we change config we need to restart the consumers of that ConfigMap.
Yes, there are a lot of cases when you really want to restart pod/container without changes inside... Configs, cache, reconnect to external services, etc. I really hope the feature will be developed.
Small work around (I use deployments and I want to change configs without having real changes in image/pod):
k8s will see that definition of the deployment has been changed and will start process of replacing pods PS: if someone has better solution, please share
Thank you @paunin
@paunin Thats exactly the case where we need it currently - We have to change ConfigMap values that are very important to the services and need to be rolled-out to the containers within minutes up to some hours. If no deployment happens in the meantime the containers will all fail at the same time and we will have partial downtime of at least some seconds
from (kinda-related #9043): a one-liner of @paunin's approach, where RESTART_
is the environment variable, and it's set to an ISO timestamp:
kubectl patch deployment mydeployment \
-p'{"spec":{"template":{"spec":{"containers":[{"name":"mycontainer","env":[{"name":"RESTART_","value":"$(date -uIseconds)"}]}]}}}}'
(note for some reason environment variables starting with _
just seem to disappear, and numeric env value
s cause errors, they need to be strings)
@paunin @rcoup We do something very similar nowadays, we have an env var literally called "DUMMY_VAR_FOR_NO_OP_DEPLOYMENT".
It feels like the right solution here would enable you to restart a deployment, and reuse most of the deployment parameters for rollouts like MinReadyCount, while allowing for command-line overrides like increasing the parallelism for emergency situations where you need everything to bounce immediately.
any progress on this?
I feel like this addition is an unnecessary bloating of the CLI API. This can be easily achieved by updating deployment environment variable values as suggested by @paunin.
We would also like to see this for deployments maybe like kubectl restart deployment some-api
Kubernetes is allowed to restart Pods for all sorts of reasons, but the cluster admin isn't allowed to. I understand the moral stand that 'turn it off and on again' may not be a desired way to operate... but I also think it should be ok to let those people who wish to, to restart a Deployment without resorting to the range of less appetizing tricks like:
'No, no, I'm not restarting anything, just correcting a typo in this label here' π
This feature will be useful in pair with kubectl apply
: apply
will update configs, including Replication Controllers, but pods won't be restarted.
So we need a method to restart these pods Blue-Green way.
@DmitryRomanenko how about switching from ReplicationControllers to Deployments? That will enable you to run updates of ReplicaSets (successor to ReplicationController).
@kargakis it's not possible: Deployments update only Replica Sets and Pods.
With kubectl apply
we also update ConfigMaps, Services, etc.
@DmitryRomanenko if the issue is "I want to restart Pods when ConfigMap/Secret is updated" then possible solution is to have versions for your ConfigMap and Secrets, and make that version a part of your Deployment spec. So with kubectl apply
ing the spec of Deployment is changed and Pods are recreated.
In other situations I don't see why Pods must be restarted (I mean Service/Ingress/etc update).
@tyranron, thanks! What is the best way to version ConfigMap
s? Should I create new config map with a different name for new deployment?
@DmitryRomanenko you actually can, why not? But in this case you should take care about removing old ones. On the other hand it may be useful for rollback. But in most cases specifying version via labels is enough.
I do believe that the best solution here could be some kind of watcher or hash-sum checker on configmap
object. Which should trigger related objects/pods restart(anything uses that configmap
,secret
). Not sure it's something accessible in k8s
architecture though...
I also think that it's better to have control from configmap|secret
object to trigger restart on changes or not.
@tyranron,
So with kubectl applying the spec of Deployment is changed and Pods are recreated.
Could you explain? Should I just use kubectl apply -f new_config.yml
with updated deployments, and these deployments will be rolling-restarted?
@DmitryRomanenko yup, exactly.
@DmitryRomanenko with applying new specs you are updating Deployment, and on Deployment update restart is triggered if its spec was changed.
By default the restart strategy is RollingUpdate
, but you can specify another one explicitly too.
Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale
.
Stale issues rot after an additional 30d of inactivity and eventually close.
Prevent issues from auto-closing with an /lifecycle frozen
comment.
If this issue is safe to close now please do so with /close
.
Send feedback to sig-testing, kubernetes/test-infra and/or @fejta
.
/lifecycle stale
A small change to @rcoup's solution: make sure the date
is evaluated within the shell:
kubectl patch deployment mydeployment -p '{"spec":{"template":{"spec":{"containers":[{"name":"mycontainer","env":[{"name":"RESTART_","value":"'$(date +%s)'"}]}]}}}}'
/remove-lifecycle stale /lifecycle frozen
Been using Swarm mode for some time which is considered less flexible than Kubernetes, I am able to restart service tasks (read: deployment pods), by simply doing a force update (with no change to the specs) as in docker service update --force <service-name>
. I would expect Kubernetes to do this as well.
As for configmaps and secrets, swarm does not allow you to edit those, you need to rotate them instead. You do this by creating new configmaps/secrets, updating the service spec to use the new ones, then delete the older ones. I see this is typically what is recommended above by versioning your configmaps/secerts and updating the deployments that use them. To be honest, this behavior of rotation is one major reason I left swarm for! It is very inconvenient to have a local copy, update then create new resources and finally update the dependent resources. Add to that, secrets in swarm cannot be read from the api, you have to mount them inside any container (or exec inside a container that uses them), then cat
their files.
On a related note, I used openshift for some time and I believe it auto-restarts pods on env/configmap/secret change? I stand corrected though.
should be app responsibility to watch filesystem for changes, as mentioned you can use checksums on the configmap/secret and force restarts that way
but if you don't want to change the config at all and just do a rolling restart with arbitrary pause, a simple pipeline does the job (this one sleeps 30seconds between terminated pod)
kubectl get po -l release=my-app -o name | cut -d"/" -f2 | while read p;do kubectl delete po $p;sleep 30;done
do note that if you press ctrl+c, it's not easy to restart where you left off
@so0k, alternative command:
kubectl get pods|grep somename|awk '{print $1}' | xargs -i sh -c 'kubectl delete pod -o name {} && sleep 4'
Two and half years on and people are still crafting new workarounds, with dummy env vars, dumy labels, ConfigMap and Secret watcher sidecars, scaling to zero, and straight out rolling-update shell scripts to simulate the ability the trigger a rolling update. Is this still something cluster admins should not be allowed to do honestly, without the tricks?
https://stackoverflow.com/questions/41735829/update-a-deployment-image-in-kubernetes
kubectl scale --replicas=0 deployment application
kubectl scale --replicas=1 deployment application
https://stackoverflow.com/questions/40366192/kubernetes-how-to-make-deployment-to-update-image
Another trick is to intially run:
kubectl set image deployment/my-deployment mycontainer=myimage:latest
and then:
kubectl set image deployment/my-deployment mycontainer=myimage
It will actually be triggering the rolling-update but be sure you have also imagePullPolicy: "Always" set.
another trick I found, where you don't have to change the image name, is to change the value of a field that will trigger a rolling update, like terminationGracePeriodSeconds. You can do this using kubectl edit deployment your_deployment or kubectl apply -f your_deployment.yaml or using a patch like this:
kubectl patch deployment your_deployment -p \ '{"spec":{"template":{"spec":{"terminationGracePeriodSeconds":31}}}}'
http://rancher.com/docs/rancher/v1.4/en/cattle/upgrading/
# Force an upgrade even though the docker-compose.yml for the services didn't change
$ rancher-compose up --force-upgrade
@so0k @KIVagant deleting pods means downtime, even when running multiple instances. When someone runs a single pod with strategy.rollingUpdate.maxUnavailable = 0
a regular deployment first creates a new pod before it terminates the existing one. The kubectl patch deployment
trick triggers this behavior, deleting pods doesn't. I'd really like a non-hacky way to trigger this behavior on demand.
For instance, when running images from hub.docker.com, the same tag can be patched for security updates. I'd really like to "pull the latest images", and perform a rolling update for any outdated image.
Rollout of ConfigMap/Secret updates is #22368 Easier rollout of new images is #1697 In-place rolling updates is #9043
Restart on image builds: https://github.com/GoogleCloudPlatform/freshpod Helm summit presentation of a trick using a templated annotation to trigger deployment rollouts: https://www.youtube.com/watch?v=dSw0w1x96ak
@bgrant0607 I think the important use-case that is not covered by other tickets is this one: https://github.com/kubernetes/kubernetes/issues/13488#issuecomment-136946912
@ghodss wrote: I suppose this would more be for a situation where the pod is alive and responding to checks but still needs to be restarted. One example is a service with an in-memory cache or internal state that gets corrupted and needs to be cleared.
I feel like asking for an application to be restarted is a fairly common use case, but maybe I'm incorrect.
I'd like to force a rolling restart to clear all application state without manual tricks.
I've got a similar one-liner based on the approach that @rcoup and @paunin described, but it should work for the more general case. It relies on JSON output and uses jq for parsing. I have this loaded in to my bashrc as a function so I can call it anywhere
kubectl-restart() {
kubectl get deploy $1 -o json | jq \
'del(
.spec.template.spec.containers[0].env[]
| select(.name == "RESTART_"))
| .spec.template.spec.containers[0].env += [{name: "RESTART_", value: now|tostring}]' | kubectl apply -f -
}
This allows me to just say: kubectl-restart my-deployment-name
and it will "update" the RESTART_
env var in the first container to the current timestamp. I'm far from a jq expert so there may be a better way to do it, but basically it deletes the old RESTART_
env var from the output (if it exists) and then adds it back in there with the current time.
It does feel very weird to me that there is no native way to do this though... Certainly a room full of engineers would agree that the ability to "turn it off and back on again" is something we'd like to have.
That is a fine hack and all, but it has big downside. The next time you deploy, using kubectl apply -f
, that component will get restarted if it has a RESTART_xxx env var even if nothing else changes. In other words, it causes spurious restarts on next deployment if it has ever been restarted between last deployment and this deployment. Not ideal...
That is why rolling-restart functionality should be baked into the Deployment controller, not built on top.
I wrote a bash function to accomplish the "patch deployment of terminationGracePeriodSeconds
" strategy @whereisaaron quoted in his comment above (source: https://stackoverflow.com/a/40368520/90442).
# $1 is a valid namespace
function refresh-all-pods() {
echo
DEPLOYMENT_LIST=$(kubectl -n $1 get deployment -o json|jq -r .items[].metadata.name)
echo "Refreshing pods in all Deployments"
for deployment_name in $DEPLOYMENT_LIST ; do
TERMINATION_GRACE_PERIOD_SECONDS=$(kubectl -n $1 get deployment "$deployment_name" -o json|jq .spec.template.spec.terminationGracePeriodSeconds)
if [ "$TERMINATION_GRACE_PERIOD_SECONDS" -eq 30 ]; then
TERMINATION_GRACE_PERIOD_SECONDS='31'
else
TERMINATION_GRACE_PERIOD_SECONDS='30'
fi
patch_string="{\"spec\":{\"template\":{\"spec\":{\"terminationGracePeriodSeconds\":$TERMINATION_GRACE_PERIOD_SECONDS}}}}"
kubectl -n $1 patch deployment $deployment_name -p $patch_string
done
echo
}
Comment / update it via the gist here. I'll echo others' comments that it would be better if this wasn't necessary.
Just by way of a more specifically kube related justification, restart also allows the re-running of an init-container, which can be used to roll keys, update config, or whatever you use init containers for.
@kubernetes/sig-apps-feature-requests @kow3ns @janetkuo
kubectl rolling-update
is useful for incrementally deploying a new replication controller. But if you have an existing replication controller and want to do a rolling restart of all the pods that it manages, you are forced to do a no-op update to an RC with a new name and the same spec. It would be useful to be able to do a rolling restart without needing to change the RC or to give the RC spec, so anyone with access to kubectl could easily initiate a restart without worrying about having the spec locally, making sure it's the same/up to date, etc. This could work in a few different ways:kubectl rolling-restart
that takes an RC name and incrementally deletes all the pods controlled by the RC and allows the RC to recreate them.kubectl rolling-update
with a flag that lets you specify an old RC only, and it follows the logic of either 1 or 2.kubectl rolling-update
with a flag that lets you specify an old RC only, and it auto-generates a new RC based on the old one and proceeds with normal rolling update logic.All of the above options would need the MaxSurge and MaxUnavailable options recently introduced (see #11942) along with readiness checks along the way to make sure that the restarting is done without taking down all the pods.
@nikhiljindal @kubernetes/kubectl