patchKubeStatusAnnotation not working when istio sidecar injected

[emrahcetiner]

What happened: if istio sidecar injection enabled, patch command for pod annotation couse error:

cmd/proxy.go:302    check function error    {"error": "failed to update proxyInfo: update failed: Pod \"postgresql-stolon-proxy-5487888cdb-2k9fb\" is invalid: spec: Forbidden: pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations` (only additions to existing tolerations)\n  core.PodSpec{\n  \tVolumes: []core.Volume{{Name: \"postgresql-stolon-token-6v52p\", VolumeSource: core.VolumeSource{Secret: &core.SecretVolumeSource{SecretName: \"postgresql-stolon-token-6v52p\", DefaultMode: &420}}}, {Name: \"istio-envoy\", VolumeSource: core.VolumeSource{EmptyDir: &core.EmptyDirVolumeSource{Medium: \"Memory\"}}}, {Name: \"istio-certs\", VolumeSource: core.VolumeSource{Secret: &core.SecretVolumeSource{SecretName: \"istio.postgresql-stolon\", DefaultMode: &420, Optional: &true}}}},\n  \tInitContainers: []core.Container{\n  \t\t{\n  \t\t\t... // 15 identical fields\n  \t\t\tTerminationMessagePolicy: \"File\",\n  \t\t\tImagePullPolicy:          \"IfNotPresent\",\n  \t\t\tSecurityContext: &core.SecurityContext{\n  \t\t\t\t... // 3 identical fields\n  \t\t\t\tWindowsOptions:         nil,\n  \t\t\t\tRunAsUser:              &0,\n- \t\t\t\tRunAsGroup:             nil,\n+ \t\t\t\tRunAsGroup:             &0,\n  \t\t\t\tRunAsNonRoot:           &false,\n  \t\t\t\tReadOnlyRootFilesystem: &false,\n  \t\t\t\t... // 2 identical fields\n  \t\t\t},\n  \t\t\tStdin:     false,\n  \t\t\tStdinOnce: false,\n  \t\t\tTTY:       false,\n  \t\t},\n  \t},\n  \tContainers: []core.Container{\n  \t\t{Name: \"stolon\", Image: \"sorintlab/stolon:v0.13.0-pg10\", Command: []string{\"/bin/bash\", \"-ec\", \"exec gosu stolon stolon-proxy\\n\"}, Ports: []core.ContainerPort{{Name: \"metrics\", ContainerPort: 8080, Protocol: \"TCP\"}, {Name: \"stolon\", ContainerPort: 5432, Protocol: \"TCP\"}}, Env: []core.EnvVar{{Name: \"POD_NAME\", ValueFrom: &core.EnvVarSource{FieldRef: &core.ObjectFieldSelector{APIVersion: \"v1\", FieldPath: \"metadata.name\"}}}, {Name: \"STPROXY_CLUSTER_NAME\", Value: \"postgresql-stolon\"}, {Name: \"STPROXY_STORE_BACKEND\", Value: \"kubernetes\"}, {Name: \"STPROXY_KUBE_RESOURCE_KIND\", Value: \"configmap\"}, {Name: \"STPROXY_LISTEN_ADDRESS\", Value: \"0.0.0.0\"}, {Name: \"STPROXY_METRICS_LISTEN_ADDRESS\", Value: \"0.0.0.0:8080\"}, {Name: \"STPROXY_DEBUG\", Value: \"true\"}, {Name: \"STPROXY_LOG_LEVEL\", Value: \"debug\"}, {Name: \"STPROXY_TCP_KEEPALIVE_COUNT\", Value: \"0\"}, {Name: \"STPROXY_TCP_KEEPALIVE_IDLE\", Value: \"0\"}, {Name: \"STPROXY_TCP_KEEPALIVE_INTERVAL\", Value: \"0\"}}, VolumeMounts: []core.VolumeMount{{Name: \"postgresql-stolon-token-6v52p\", ReadOnly: true, MountPath: \"/var/run/secrets/kubernetes.io/serviceaccount\"}}, ReadinessProbe: &core.Probe{Handler: core.Handler{TCPSocket: &core.TCPSocketAction{Port: intstr.IntOrString{IntVal: 5432}}}, InitialDelaySeconds: 10, TimeoutSeconds: 5, PeriodSeconds: 10, SuccessThreshold: 1, FailureThreshold: 3}, TerminationMessagePath: \"/dev/termination-log\", TerminationMessagePolicy: \"File\", ImagePullPolicy: \"IfNotPresent\"},\n  \t\t{\n  \t\t\t... // 15 identical fields\n  \t\t\tTerminationMessagePolicy: \"File\",\n  \t\t\tImagePullPolicy:          \"IfNotPresent\",\n  \t\t\tSecurityContext: &core.SecurityContext{\n  \t\t\t\t... // 3 identical fields\n  \t\t\t\tWindowsOptions:         nil,\n  \t\t\t\tRunAsUser:              &1337,\n- \t\t\t\tRunAsGroup:             nil,\n+ \t\t\t\tRunAsGroup:             &1337,\n  \t\t\t\tRunAsNonRoot:           &true,\n  \t\t\t\tReadOnlyRootFilesystem: &true,\n  \t\t\t\t... // 2 identical fields\n  \t\t\t},\n  \t\t\tStdin:     false,\n  \t\t\tStdinOnce: false,\n  \t\t\tTTY:       false,\n  \t\t},\n  \t},\n  \tRestartPolicy:                 \"Always\",\n  \tTerminationGracePeriodSeconds: &30,\n  \t... // 21 identical fields\n  }\n"}

What you expected to happen: information of proxy, keeper and sentinel patch to pod's annotation.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?: i have istio-sidecar. istio inject their sidecar automaticaly in same pod with stolon-proxy/keeper/sentinel.

Environment:

• Stolon version: v0.16.0
• Stolon running environment (if useful to understand the bug):
• Others: istio version: 1.4.6

[sgotti]

@emrahcetiner I'm not sure this is a stolon bug. We're just patching the pod annotations and it works with a standard pod so something strange happens when istio inject another container in the pod. You should try to dig inside this issue and report here if you find something useful.

[elkh510]

hi @sgotti could you tell pls which configuration you checked and it worked for you. if not, tell me plz how can this be diagnosed? the fact is that on configuration with postgres helm chart (without stolon) all work as expected and the problem only occurs in conjunction with stolon thanks in advance!

[sgotti]

@elkh510 I said that stolon updates pods metadata and without istio everything works correctly. So it's istio that affects something. Probably it has some sort of admission webhooks that are doing something strange. You should investigate what part of istio affects this and report it.

[mstinglin]

I think this issue on https://github.com/istio/istio/issues/17646 "HTTP PATCH call failing when istio is active on pod" might be related
"Seems we have overcome the issue Instead of a PATCH call, we now use a POST call with this URL parameter: "_HttpMethod", "PATCH" We will do some more tests, but it seems the issue is resolved."

I am not sure how we would be able to fix it with stolon.

[elkh510]

problem resolves by adding this parameter --pg-advertise-port string postgresql instance port from outside. Use it to expose port different than local port with a PAT networking config https://github.com/sorintlab/stolon/blob/master/doc/commands/stolon-keeper.md