drivebyer
commented
1 month ago @Leo791, as the code shows:
// updateStatefulSet is a method to update statefulset in Kubernetes
func updateStatefulSet(cl kubernetes.Interface, logger logr.Logger, namespace string, stateful *appsv1.StatefulSet, recreateStateFulSet bool) error {
_, err := cl.AppsV1().StatefulSets(namespace).Update(context.TODO(), stateful, metav1.UpdateOptions{})
if recreateStateFulSet {
sErr, ok := err.(*apierrors.StatusError)
if ok && sErr.ErrStatus.Code == 422 && sErr.ErrStatus.Reason == metav1.StatusReasonInvalid {
failMsg := make([]string, len(sErr.ErrStatus.Details.Causes))
for messageCount, cause := range sErr.ErrStatus.Details.Causes {
failMsg[messageCount] = cause.Message
}
logger.V(1).Info("recreating StatefulSet because the update operation wasn't possible", "reason", strings.Join(failMsg, ", "))
propagationPolicy := metav1.DeletePropagationForeground
if err := cl.AppsV1().StatefulSets(namespace).Delete(context.TODO(), stateful.GetName(), metav1.DeleteOptions{PropagationPolicy: &propagationPolicy}); err != nil { //nolint
return errors.Wrap(err, "failed to delete StatefulSet to avoid forbidden action")
}
}
}
if err != nil {
logger.Error(err, "Redis statefulset update failed")
return err
}
logger.V(1).Info("Redis statefulset successfully updated ")
return nil
}StatefulSets are only deleted when you attempt to update forbidden fields, such as the persistentVolumeClaimTemplate field. Therefore, in my opinion, when a StatefulSet gets stuck in a pending state due to insufficient resources, we need to manually delete the StatefulSet (and its pods) under the current code design.
Leo791
What version of redis operator are you using?
redis-operator version: v0.18.0
Does this issue reproduce with the latest release? Yes
What operating system and processor architecture are you using (
kubectl version)?Client Version: v1.30.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.0
What did you do?
redis.opstreelabs.in/recreate-statefulset: "true"to the crdWhat did you expect to see?
What did you see instead?
The operator throws the following errors:
And if we exec into a cluster pod and ask for the cluster node info we get:
We believe the operator is promoting the follower to leader, but expects it to be named leader-3 instead of follower-x. This causes the updated of the stateful set to be blocked and we cannot rollback the cluster to a healthy state.
Is there a way to prevent failover from occurring and the promotion to occur?