brownleej
commented
3 years ago You can work around this by adding the instances that are in pending to the instancesToRemoveWithoutExclusion list, which will delete them without checking that data has been moved off of them. This is dangerous in general, which is why the operator doesn't do it by default, but if the pods have never run, or if the cluster is otherwise healthy, it should be safe.
There's also some discussion of this in a recent design doc on automating replacements: https://github.com/FoundationDB/fdb-kubernetes-operator/wiki/Design-for-Automating-Replacements-through-the-Operator. Though that's a different use case, it can face the same problem.
johscheuer
Scenario
End state One set of pods running with the old configuration (pre update) One set of pods pending with the bad configuration One set of pods running with the good configuration
The reconciliation is failing early with "Cannot check the exclusion state of instance log-2, which has no IP address". Thus neither the old pods, nor the bad pending pods will get removed.
https://github.com/FoundationDB/fdb-kubernetes-operator/blob/f3104bb4a3c3fccaf28f5aad2dbc0b997c3fff55/controllers/confirm_exclusion_completion.go#L45-L53
Operator logs
``` 2020-11-04T18:50:29.461Z INFO controller Reconciliation terminated early {"namespace": "timeseries", "name": "foundationdb-cluster", "lastAction": "controllers.ConfirmExclusionCompletion"} 2020-11-04T18:50:29.461Z ERROR controller Error in reconciliation {"subReconciler": "controllers.ConfirmExclusionCompletion", "namespace": "timeseries", "cluster": "foundationdb-cluster", "error": "Cannot check the exclusion state of instance log-2, which has no IP address"} github.com/go-logr/zapr.(*zapLogger).Error /go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128 github.com/FoundationDB/fdb-kubernetes-operator/controllers.(*FoundationDBClusterReconciler).Reconcile /workspace/controllers/cluster_controller.go:137 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211 k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1 /go/pkg/mod/k8s.io/apimachinery@v0.17.0/pkg/util/wait/wait.go:152 k8s.io/apimachinery/pkg/util/wait.JitterUntil /go/pkg/mod/k8s.io/apimachinery@v0.17.0/pkg/util/wait/wait.go:153 k8s.io/apimachinery/pkg/util/wait.Until /go/pkg/mod/k8s.io/apimachinery@v0.17.0/pkg/util/wait/wait.go:88 2020-11-04T18:50:29.461Z ERROR controller-runtime.controller Reconciler error {"controller": "foundationdbcluster", "request": "timeseries/foundationdb-cluster", "error": "Cannot check the exclusion state of instance log-2, which has no IP address"} github.com/go-logr/zapr.(*zapLogger).Error /go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:258 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211 k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1 /go/pkg/mod/k8s.io/apimachinery@v0.17.0/pkg/util/wait/wait.go:152 k8s.io/apimachinery/pkg/util/wait.JitterUntil /go/pkg/mod/k8s.io/apimachinery@v0.17.0/pkg/util/wait/wait.go:153 k8s.io/apimachinery/pkg/util/wait.Until /go/pkg/mod/k8s.io/apimachinery@v0.17.0/pkg/util/wait/wait.go:88 ```