search

grafana / tempo-operator

Grafana Tempo Kubernetes operator
https://grafana.com/docs/tempo/latest/setup/operator/
GNU Affero General Public License v3.0
61 stars 30 forks source link

Oauth-proxy on OpenShift causes infinite reconciliation loop #1018

Closed pavolloffay closed 2 months ago

pavolloffay commented 2 months ago 
kubectl apply -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
spec:
  storage:
    secret:
      name: minio-test
      type: s3
  storageSize: 1Gi
  template:
    queryFrontend:
      jaegerQuery:
        enabled: true
        ingress:
          type: route
EOF
kubectl apply -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoMonolithic
metadata:
  name: simplestmono
spec:
  storage:
    traces:
      backend: s3 
      size: 3Gi 
      s3: 
        secret: minio-test
  jaegerui:
    enabled: true 
    route:
      enabled: true
EOF

The secret tempo-simplest-cookie-proxy keeps updating, only the resource version

~ » k get secret tempo-simplest-cookie-proxy -o yaml                                                                                                                                                                                                                                                                                   ploffay@fedora
apiVersion: v1
data:
  session_secret: SUpCS1RJbURWN3FNTGdwQW1qUG5YQT09
kind: Secret
metadata:
  creationTimestamp: "2024-08-27T14:06:28Z"
  labels:
    app.kubernetes.io/component: query-frontend-proxy
    app.kubernetes.io/instance: simplest
    app.kubernetes.io/managed-by: tempo-operator
    app.kubernetes.io/name: tempo
  name: tempo-simplest-cookie-proxy
  namespace: ploffay
  ownerReferences:
  - apiVersion: tempo.grafana.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: TempoStack
    name: simplest
    uid: 139aeb6f-2397-4b1c-8d6a-48571d83fa22
  resourceVersion: "466146"
  uid: f1a51906-3158-47c9-a3e6-93529777b9ed
type: Opaque
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~ » k get secret tempo-simplest-cookie-proxy -o yaml                                                                                                                                                                                                                                                                                   ploffay@fedora
apiVersion: v1
data:
  session_secret: Q2IycVByZjU2WXcxR09MMC92UWJvdz09
kind: Secret
metadata:
  creationTimestamp: "2024-08-27T14:06:28Z"
  labels:
    app.kubernetes.io/component: query-frontend-proxy
    app.kubernetes.io/instance: simplest
    app.kubernetes.io/managed-by: tempo-operator
    app.kubernetes.io/name: tempo
  name: tempo-simplest-cookie-proxy
  namespace: ploffay
  ownerReferences:
  - apiVersion: tempo.grafana.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: TempoStack
    name: simplest
    uid: 139aeb6f-2397-4b1c-8d6a-48571d83fa22
  resourceVersion: "466149"
  uid: f1a51906-3158-47c9-a3e6-93529777b9ed
type: Opaque
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~ » k get secret tempo-simplest-cookie-proxy -o yaml                                                                                                                                                                                                                                                                                   ploffay@fedora
apiVersion: v1
data:
  session_secret: ZmlQMStCaC9HQmRKYnE0b0ZkOSt0UT09
kind: Secret
metadata:
  creationTimestamp: "2024-08-27T14:06:28Z"
  labels:
    app.kubernetes.io/component: query-frontend-proxy
    app.kubernetes.io/instance: simplest
    app.kubernetes.io/managed-by: tempo-operator
    app.kubernetes.io/name: tempo
  name: tempo-simplest-cookie-proxy
  namespace: ploffay
  ownerReferences:
  - apiVersion: tempo.grafana.com/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: TempoStack
    name: simplest
    uid: 139aeb6f-2397-4b1c-8d6a-48571d83fa22
  resourceVersion: "466160"
  uid: f1a51906-3158-47c9-a3e6-93529777b9ed
type: Opaque
Rajpratik71 commented 3 weeks ago

For me "query-frontend" pod remained in CrashLoopBackOff .

pod/tempo-gi-query-frontend-6b5bff6dd4-4tj8w                          2/3     CrashLoopBackOff   36 (2m3s ago)   161m

with below error

oc logs pod/tempo-gi-query-frontend-6b5bff6dd4-4tj8w oauth-proxy
2024/10/16 15:13:27 main.go:145: unable to decode review: invalid character 's' looking for beginning of value
andreasgerstmayr commented 3 weeks ago

For me "query-frontend" pod remained in CrashLoopBackOff .

pod/tempo-gi-query-frontend-6b5bff6dd4-4tj8w                          2/3     CrashLoopBackOff   36 (2m3s ago)   161m

with below error

oc logs pod/tempo-gi-query-frontend-6b5bff6dd4-4tj8w oauth-proxy
2024/10/16 15:13:27 main.go:145: unable to decode review: invalid character 's' looking for beginning of value

Can you open a separate issue and post your TempoStack CR?

Rajpratik71 commented 3 weeks ago

For me "query-frontend" pod remained in CrashLoopBackOff .

pod/tempo-gi-query-frontend-6b5bff6dd4-4tj8w                          2/3     CrashLoopBackOff   36 (2m3s ago)   161m

with below error

oc logs pod/tempo-gi-query-frontend-6b5bff6dd4-4tj8w oauth-proxy
2024/10/16 15:13:27 main.go:145: unable to decode review: invalid character 's' looking for beginning of value

Can you open a separate issue and post your TempoStack CR?

I think issue was related to this one only.

After disabling / removing route from CR , CrashLoopBackOff stopped. I was using Redhat Official Operator on Openshift 4.14.36.

andreasgerstmayr commented 2 weeks ago

For me "query-frontend" pod remained in CrashLoopBackOff .

pod/tempo-gi-query-frontend-6b5bff6dd4-4tj8w                          2/3     CrashLoopBackOff   36 (2m3s ago)   161m

with below error

oc logs pod/tempo-gi-query-frontend-6b5bff6dd4-4tj8w oauth-proxy
2024/10/16 15:13:27 main.go:145: unable to decode review: invalid character 's' looking for beginning of value

Can you open a separate issue and post your TempoStack CR?

I think issue was related to this one only.

After disabling / removing route from CR , CrashLoopBackOff stopped. I was using Redhat Official Operator on Openshift 4.14.36.

unable to decode review: invalid character 's' looking for beginning of value sounds like a typo in the sar value of the CR, that's why we need the entire CR to be able to debug this issue. This issue is about infinite reconciliation, the issue you posted is about a crash loop. The area of the error might be the same, but the root cause is different.