The `helm.operator-sdk/upgrade-force` annotation shows not work as expected

[camilamacedo86]

@camilamacedo86 thanks for the update. your test updated the replica, which is not an immutable field. this is not a good test for the --force option, since it will pass even without

with further tests today, it appears the --force behavior in the 0.17.1 base image is backwards. I'd like to share the following steps to reproduce the issue:

1. create cr a managed by helm without helm.operator-sdk/upgrade-force annotation
2. make an updates to cr to trigger the following changes one at a time: a. updating replica b. changing memory limit c. changing resource labels
3. all 3 cases are updated properly by operator (pods are recreated as needed)
4. annotate pr to turn on --force: oc annotate elasticstack/logging "helm.operator-sdk/upgrade-force"="True"
5. repeat changes in step 2, helm operator start to fail

   {"level":"error","ts":1591299355.1403763,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"elasticstack-controller","request":"ibm-common-services/logging","error":"failed update (failed to replace object: Job.batch \"logging-elk-elasticsearch-pki-init\" is invalid: [spec.selector: Required value, spec.template.metadata.labels: Invalid value: map[string]string{\"app\":\"logging-elk-elasticsearch\", \"app.kubernetes.io/instance\":\"common-logging\", \"chart\":\"ibm-icplogging\", \"component\":\"secret-init\", \"heritage\":\"Helm\", \"release\":\"logging\", \"role\":\"initialization\"}: `selector` does not match template `labels`, spec.selector: Invalid value: \"null\": field is immutable, spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"\", GenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"app\":\"logging-elk-elasticsearch\", \"app.kubernetes.io/instance\":\"common-logging\", \"chart\":\"ibm-icplogging\", \"component\":\"secret-init\", \"heritage\":\"Helm\", \"release\":\"logging\", \"role\":\"initialization\"}, Annotations:map[string]string{\"productID\":\"068a62892a1e4db39641342e592daa25\", \"productMetric\":\"FREE\", \"productName\":\"IBM Cloud Platform Common Services\", \"productVersion\":\"3.4.0\"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:\"\", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\"kubectl\", Image:\"quay.io/opencloudio/logging-pki-init@sha256:d73cbc2895486c425bde39497b10e00fd45b550e4953eb8d3e04c1ac3936b4a8\", Command:[]string(nil), Args:[]string(nil), WorkingDir:\"\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:\"APP_NAME\", Value:\"logging-elk-elasticsearch\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CHART_NAME_VERSION\", Value:\"ibm-icplogging-3.1.3\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"COMPONENT_NAME\", Value:\"secret-init\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"HERITAGE\", Value:\"Helm\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"ROLE_NAME\", Value:\"initialization\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RELEASE_NAME_LABEL\", Value:\"logging\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RELEASE_NAME\", Value:\"logging-elk\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"REQUIRED_CERT_ENTRY_COUNT\", Value:\"21\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CA_KEYSTORE_PASSWORD\", Value:\"Y2hhbmdlbWU=\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CA_TRUSTSTORE_PASSWORD\", Value:\"Y2hhbmdlbWU=\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"APP_KEYSTORE_PASSWORD\", Value:\"Y2hhbmdlbWU=\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT1_NAME\", Value:\"sgadmin\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT1_KEYSTORE\", Value:\"jks\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT2_NAME\", Value:\"kibana\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT2_KEYSTORE\", Value:\"pem\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT2_SAN\", Value:\"dns:kibana,dns:kibana.ibm-common-services,dns:localhost,ip:127.0.0.1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT2_KEY_FORMAT\", Value:\"pkcs8\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT3_NAME\", Value:\"logstash\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT3_KEYSTORE\", Value:\"pem\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT3_SAN\", Value:\"dns:logstash,dns:logstash.ibm-common-services,dns:localhost,ip:127.0.0.1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT3_KEY_FORMAT\", Value:\"pkcs8\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT4_NAME\", Value:\"logstash-monitoring\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT4_KEYSTORE\", Value:\"jks\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT5_NAME\", Value:\"logstash-elasticsearch\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT5_KEYSTORE\", Value:\"jks\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT6_NAME\", Value:\"filebeat\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT6_KEYSTORE\", Value:\"pem\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT6_KEY_FORMAT\", Value:\"pkcs1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT7_NAME\", Value:\"curator\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT7_KEYSTORE\", Value:\"pem\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT7_SAN\", Value:\"dns:localhost,ip:127.0.0.1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT7_KEY_FORMAT\", Value:\"pkcs1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT7_KEY_ENCRYPTION\", Value:\"false\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT8_NAME\", Value:\"superuser\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT8_KEYSTORE\", Value:\"pem\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT8_KEY_FORMAT\", Value:\"pkcs1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT9_NAME\", Value:\"elasticsearch\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT9_KEYSTORE\", Value:\"jks\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT9_SAN\", Value:\"dns:elasticsearch,dns:elasticsearch.ibm-common-services,dns:localhost,ip:127.0.0.1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT10_NAME\", Value:\"elasticsearch-transport\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT10_KEYSTORE\", Value:\"jks\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT10_SAN\", Value:\"dns:localhost,ip:127.0.0.1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT11_NAME\", Value:\"kibanarouter\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT11_KEYSTORE\", Value:\"pem\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT11_KEY_FORMAT\", Value:\"pkcs1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT11_SAN\", Value:\"dns:kibana,dns:kibana.ibm-common-services\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT12_NAME\", Value:\"elasticsearch-router\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT12_SAN\", Value:\"dns:elasticsearch,dns:elasticsearch.ibm-common-services,dns:localhost,ip:127.0.0.1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT12_DN\", Value:\"CN=elasticsearch,OU=IBM Cloud Private\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT12_KEYSTORE\", Value:\"pem\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT12_KEY_FORMAT\", Value:\"pkcs1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT12_KEY_ENCRYPTION\", Value:\"false\", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList{\"memory\":resource.Quantity{i:resource.int64Amount{value:536870912, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"\", Format:\"BinarySI\"}}, Requests:core.ResourceList{\"memory\":resource.Quantity{i:resource.int64Amount{value:67108864, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"\", Format:\"BinarySI\"}}}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc00a2f25f0), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"Never\", TerminationGracePeriodSeconds:(*int64)(0xc02e767960), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"logging-elk-job\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc03f6df420), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration{core.Toleration{Key:\"dedicated\", Operator:\"Exists\", Value:\"\", Effect:\"NoSchedule\", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable]) and failed rollback: failed to replace object: Job.batch \"logging-elk-elasticsearch-pki-init\" is invalid: [spec.selector: Required value, spec.template.metadata.labels: Invalid value: map[string]string{\"app\":\"logging-elk-elasticsearch\", \"app.kubernetes.io/instance\":\"common-logging\", \"chart\":\"ibm-icplogging\", \"component\":\"secret-init\", \"heritage\":\"Helm\", \"release\":\"logging\", \"role\":\"initialization\"}: `selector` does not match template `labels`, spec.selector: Invalid value: \"null\": field is immutable, spec.template: Invalid value: core.PodTemplateSpec{ObjectMeta:v1.ObjectMeta{Name:\"\", GenerateName:\"\", Namespace:\"\", SelfLink:\"\", UID:\"\", ResourceVersion:\"\", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string{\"app\":\"logging-elk-elasticsearch\", \"app.kubernetes.io/instance\":\"common-logging\", \"chart\":\"ibm-icplogging\", \"component\":\"secret-init\", \"heritage\":\"Helm\", \"release\":\"logging\", \"role\":\"initialization\"}, Annotations:map[string]string{\"productID\":\"068a62892a1e4db39641342e592daa25\", \"productMetric\":\"FREE\", \"productName\":\"IBM Cloud Platform Common Services\", \"productVersion\":\"3.4.0\"}, OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:\"\", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, Spec:core.PodSpec{Volumes:[]core.Volume(nil), InitContainers:[]core.Container(nil), Containers:[]core.Container{core.Container{Name:\"kubectl\", Image:\"quay.io/opencloudio/logging-pki-init@sha256:d73cbc2895486c425bde39497b10e00fd45b550e4953eb8d3e04c1ac3936b4a8\", Command:[]string(nil), Args:[]string(nil), WorkingDir:\"\", Ports:[]core.ContainerPort(nil), EnvFrom:[]core.EnvFromSource(nil), Env:[]core.EnvVar{core.EnvVar{Name:\"APP_NAME\", Value:\"logging-elk-elasticsearch\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CHART_NAME_VERSION\", Value:\"ibm-icplogging-3.1.3\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"COMPONENT_NAME\", Value:\"secret-init\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"HERITAGE\", Value:\"Helm\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"ROLE_NAME\", Value:\"initialization\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RELEASE_NAME_LABEL\", Value:\"logging\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"RELEASE_NAME\", Value:\"logging-elk\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"REQUIRED_CERT_ENTRY_COUNT\", Value:\"21\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CA_KEYSTORE_PASSWORD\", Value:\"Y2hhbmdlbWU=\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CA_TRUSTSTORE_PASSWORD\", Value:\"Y2hhbmdlbWU=\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"APP_KEYSTORE_PASSWORD\", Value:\"Y2hhbmdlbWU=\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT1_NAME\", Value:\"sgadmin\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT1_KEYSTORE\", Value:\"jks\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT2_NAME\", Value:\"kibana\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT2_KEYSTORE\", Value:\"pem\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT2_SAN\", Value:\"dns:kibana,dns:kibana.ibm-common-services,dns:localhost,ip:127.0.0.1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT2_KEY_FORMAT\", Value:\"pkcs8\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT3_NAME\", Value:\"logstash\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT3_KEYSTORE\", Value:\"pem\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT3_SAN\", Value:\"dns:logstash,dns:logstash.ibm-common-services,dns:localhost,ip:127.0.0.1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT3_KEY_FORMAT\", Value:\"pkcs8\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT4_NAME\", Value:\"logstash-monitoring\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT4_KEYSTORE\", Value:\"jks\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT5_NAME\", Value:\"logstash-elasticsearch\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT5_KEYSTORE\", Value:\"jks\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT6_NAME\", Value:\"filebeat\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT6_KEYSTORE\", Value:\"pem\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT6_KEY_FORMAT\", Value:\"pkcs1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT7_NAME\", Value:\"curator\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT7_KEYSTORE\", Value:\"pem\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT7_SAN\", Value:\"dns:localhost,ip:127.0.0.1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT7_KEY_FORMAT\", Value:\"pkcs1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT7_KEY_ENCRYPTION\", Value:\"false\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT8_NAME\", Value:\"superuser\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT8_KEYSTORE\", Value:\"pem\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT8_KEY_FORMAT\", Value:\"pkcs1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT9_NAME\", Value:\"elasticsearch\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT9_KEYSTORE\", Value:\"jks\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT9_SAN\", Value:\"dns:elasticsearch,dns:elasticsearch.ibm-common-services,dns:localhost,ip:127.0.0.1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT10_NAME\", Value:\"elasticsearch-transport\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT10_KEYSTORE\", Value:\"jks\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT10_SAN\", Value:\"dns:localhost,ip:127.0.0.1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT11_NAME\", Value:\"kibanarouter\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT11_KEYSTORE\", Value:\"pem\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT11_KEY_FORMAT\", Value:\"pkcs1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT11_SAN\", Value:\"dns:kibana,dns:kibana.ibm-common-services\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT12_NAME\", Value:\"elasticsearch-router\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT12_SAN\", Value:\"dns:elasticsearch,dns:elasticsearch.ibm-common-services,dns:localhost,ip:127.0.0.1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT12_DN\", Value:\"CN=elasticsearch,OU=IBM Cloud Private\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT12_KEYSTORE\", Value:\"pem\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT12_KEY_FORMAT\", Value:\"pkcs1\", ValueFrom:(*core.EnvVarSource)(nil)}, core.EnvVar{Name:\"CERT12_KEY_ENCRYPTION\", Value:\"false\", ValueFrom:(*core.EnvVarSource)(nil)}}, Resources:core.ResourceRequirements{Limits:core.ResourceList{\"memory\":resource.Quantity{i:resource.int64Amount{value:536870912, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"\", Format:\"BinarySI\"}}, Requests:core.ResourceList{\"memory\":resource.Quantity{i:resource.int64Amount{value:67108864, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:\"\", Format:\"BinarySI\"}}}, VolumeMounts:[]core.VolumeMount(nil), VolumeDevices:[]core.VolumeDevice(nil), LivenessProbe:(*core.Probe)(nil), ReadinessProbe:(*core.Probe)(nil), StartupProbe:(*core.Probe)(nil), Lifecycle:(*core.Lifecycle)(nil), TerminationMessagePath:\"/dev/termination-log\", TerminationMessagePolicy:\"File\", ImagePullPolicy:\"IfNotPresent\", SecurityContext:(*core.SecurityContext)(0xc00f073630), Stdin:false, StdinOnce:false, TTY:false}}, EphemeralContainers:[]core.EphemeralContainer(nil), RestartPolicy:\"Never\", TerminationGracePeriodSeconds:(*int64)(0xc05be4ed60), ActiveDeadlineSeconds:(*int64)(nil), DNSPolicy:\"ClusterFirst\", NodeSelector:map[string]string(nil), ServiceAccountName:\"logging-elk-job\", AutomountServiceAccountToken:(*bool)(nil), NodeName:\"\", SecurityContext:(*core.PodSecurityContext)(0xc045808bd0), ImagePullSecrets:[]core.LocalObjectReference(nil), Hostname:\"\", Subdomain:\"\", Affinity:(*core.Affinity)(nil), SchedulerName:\"default-scheduler\", Tolerations:[]core.Toleration{core.Toleration{Key:\"dedicated\", Operator:\"Exists\", Value:\"\", Effect:\"NoSchedule\", TolerationSeconds:(*int64)(nil)}}, HostAliases:[]core.HostAlias(nil), PriorityClassName:\"\", Priority:(*int32)(nil), PreemptionPolicy:(*core.PreemptionPolicy)(nil), DNSConfig:(*core.PodDNSConfig)(nil), ReadinessGates:[]core.PodReadinessGate(nil), RuntimeClassName:(*string)(nil), Overhead:core.ResourceList(nil), EnableServiceLinks:(*bool)(nil), TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}: field is immutable]","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\tpkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\tpkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\tpkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\tpkg/mod/k8s.io/apimachinery@v0.17.4/pkg/util/wait/wait.go:88"}`

6. annotate pr to turn off --force: oc annotate elasticstack/logging "helm.operator-sdk/upgrade-force"="False"
7. helm operator back to normal, all resources updated {"level":"info","ts":1591302627.458174,"logger":"helm.controller","msg":"Reconciled release","namespace":"ibm-common-services","name":"logging","apiVersion":"elasticstack.ibm.com/v1alpha1","kind":"ElasticStack","release":"logging"

the observed behaviour appears to be:

• when the operator annotation set to "helm.operator-sdk/upgrade-force"="True", the behavior is consistent with helm --force option NOT applied
• when the operator annotation set to "helm.operator-sdk/upgrade-force"="False", the behavior is consistent with helm --force option applied

this is the opposite from in the sdk doc: https://sdk.operatorframework.io/docs/helm/reference/advanced_features/#use-helm-upgrade---force-for-deployment

from : https://github.com/operator-framework/operator-sdk/issues/3144#issuecomment-639134918

[joelanford]

Is this reproducible with the helm CLI? I'm fairly sure the helm-operator is correctly setting the upgrade.Force option during the upgrade, so I would expect the helm CLI to have this same issue.

If it does, we should close this issue and get clarification on this upstream in the Helm project.

[joelanford]

@camilamacedo86 @dian-xh, looks like this has already been discussed upstream. Check out https://github.com/helm/helm/issues/7173.

I think we should close this since the helm-operator is working as expected (i.e. applying the force option when the force annotation is set to true).

[camilamacedo86]

HI @dian-xh,

As described in the above comments I understand that all is working as expected. See that SDK lib will allow you to apply the flag, however, how the flags work is something that is part of the Helm implementation.

So, I am closing this one. However, please, feel free to ping us to re-open it or raise new issues if you see that it is required.