search

SigNoz / signoz

SigNoz is an open-source observability platform native to OpenTelemetry with logs, traces and metrics in a single application. An open-source alternative to DataDog, NewRelic, etc. 🔥 🖥. 👉 Open source Application Performance Monitoring (APM) & Observability tool
https://signoz.io
Other
18.31k stars 1.17k forks source link

Helm based install on openshift is failing as some pod got stuck in init stage #5600

Open Rajpratik71 opened 1 month ago

Rajpratik71 commented 1 month ago

Bug description

After helm install, some pods remain in Init stage.

pratikraj@Pratiks-MacBook-Pro common % kubectl create ns platform && helm --namespace platform install signoz signoz/signoz
namespace/platform created
coalesce.go:237: warning: skipped value for zookeeper.initContainers: Not a table.
W0729 20:34:11.107784   42735 warnings.go:70] would violate PodSecurity "restricted:v1.24": hostPort (container "signoz-k8s-infra-otel-agent" uses hostPorts 13133, 4317, 4318, 8888), allowPrivilegeEscalation != false (container "signoz-k8s-infra-otel-agent" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "signoz-k8s-infra-otel-agent" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volumes "varlog", "varlibdockercontainers" use restricted volume type "hostPath"), runAsNonRoot != true (pod or container "signoz-k8s-infra-otel-agent" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "signoz-k8s-infra-otel-agent" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0729 20:34:11.850003   42735 warnings.go:70] would violate PodSecurity "restricted:v1.24": unrestricted capabilities (container "zookeeper" must set securityContext.capabilities.drop=["ALL"]), seccompProfile (pod or container "zookeeper" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0729 20:34:11.850584   42735 warnings.go:70] would violate PodSecurity "restricted:v1.24": allowPrivilegeEscalation != false (containers "signoz-alertmanager-init", "signoz-alertmanager" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "signoz-alertmanager-init", "signoz-alertmanager" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "signoz-alertmanager-init" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "signoz-alertmanager-init", "signoz-alertmanager" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
NAME: signoz
LAST DEPLOYED: Mon Jul 29 20:33:36 2024
NAMESPACE: platform
STATUS: deployed
REVISION: 1
NOTES:
1. You have just deployed SigNoz cluster:

- frontend version: '0.50.0'
- query-service version: '0.50.0'
- alertmanager version: '0.23.5'
- otel-collector version: '0.102.2'
- otel-collector-metrics version: '0.102.2'

2. Get the application URL by running these commands:

  export POD_NAME=$(kubectl get pods --namespace platform -l "app.kubernetes.io/name=signoz,app.kubernetes.io/instance=signoz,app.kubernetes.io/component=frontend" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:3301 to use your application"
  kubectl --namespace platform port-forward $POD_NAME 3301:3301

If you have any ideas, questions, or any feedback, please share on our Github Discussions:
  https://github.com/SigNoz/signoz/discussions/713
pratikraj@Pratiks-MacBook-Pro common % 
pratikraj@Pratiks-MacBook-Pro common %

Pods in init stage

pratikraj@Pratiks-MacBook-Pro common % 
pratikraj@Pratiks-MacBook-Pro common % oc get po,svc,pvc -n platform                                
NAME                                                    READY   STATUS     RESTARTS   AGE
pod/signoz-clickhouse-operator-6d95579cd9-9pkx4         2/2     Running    0          8m33s
pod/signoz-frontend-64c68d5b6c-hdlsp                    0/1     Init:0/1   0          8m33s
pod/signoz-k8s-infra-otel-deployment-579bcdb68c-k8m5r   1/1     Running    0          8m33s
pod/signoz-otel-collector-689c6c4bb5-pmdsn              0/1     Init:0/1   0          8m33s
pod/signoz-otel-collector-metrics-7dcf984d77-f5rcb      0/1     Init:0/1   0          8m33s
pod/signoz-query-service-0                              0/1     Init:0/1   0          8m33s
pod/signoz-schema-migrator-init-dqwtk                   0/1     Init:0/2   0          8m32s

NAME                                         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                            AGE
service/chi-signoz-clickhouse-cluster-0-0    ClusterIP   None             <none>        9000/TCP,8123/TCP,9009/TCP                                         8m
service/signoz-alertmanager                  ClusterIP   172.30.4.3       <none>        9093/TCP                                                           8m36s
service/signoz-alertmanager-headless         ClusterIP   None             <none>        9093/TCP                                                           8m36s
service/signoz-clickhouse                    ClusterIP   172.30.12.83     <none>        8123/TCP,9000/TCP                                                  2m37s
service/signoz-clickhouse-operator-metrics   ClusterIP   172.30.94.206    <none>        8888/TCP                                                           8m36s
service/signoz-frontend                      ClusterIP   172.30.99.44     <none>        3301/TCP                                                           8m36s
service/signoz-k8s-infra-otel-agent          ClusterIP   172.30.37.114    <none>        13133/TCP,8888/TCP,4317/TCP,4318/TCP                               8m36s
service/signoz-k8s-infra-otel-deployment     ClusterIP   172.30.122.138   <none>        13133/TCP                                                          8m36s
service/signoz-otel-collector                ClusterIP   172.30.58.100    <none>        14250/TCP,14268/TCP,8081/TCP,8082/TCP,8888/TCP,4317/TCP,4318/TCP   8m36s
service/signoz-otel-collector-metrics        ClusterIP   172.30.141.116   <none>        13133/TCP                                                          8m36s
service/signoz-query-service                 ClusterIP   172.30.9.90      <none>        8080/TCP,8085/TCP,4320/TCP                                         8m36s
service/signoz-zookeeper                     ClusterIP   172.30.123.12    <none>        2181/TCP,2888/TCP,3888/TCP                                         8m36s
service/signoz-zookeeper-headless            ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP                                         8m36s

NAME                                                                                  STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
persistentvolumeclaim/data-signoz-zookeeper-0                                         Bound    pvc-22d6a98e-5eb0-4b0b-9068-2938b015fc91   8Gi        RWO            rook-ceph-block   8m35s
persistentvolumeclaim/data-volumeclaim-template-chi-signoz-clickhouse-cluster-0-0-0   Bound    pvc-3d041f47-92ca-4ea5-a55f-5bb622226b58   20Gi       RWO            rook-ceph-block   8m31s
persistentvolumeclaim/signoz-db-signoz-query-service-0                                Bound    pvc-e55aa83c-3766-442a-b873-d24d84005ebd   1Gi        RWO            rook-ceph-block   8m35s
persistentvolumeclaim/storage-signoz-alertmanager-0                                   Bound    pvc-29d826e4-e7cc-41d4-baca-577072ef9d9b   100Mi      RWO            rook-ceph-block   8m35s
pratikraj@Pratiks-MacBook-Pro common % 
pratikraj@Pratiks-MacBook-Pro common %

Expected behavior

After, Helm install all pods should be in running state.

How to reproduce

  1. kubectl create ns platform && helm --namespace platform install signoz signoz/signoz
  2. oc get po,svc,pvc -n platform

Version information

Additional context

Events shows same error / warning which received in helm install

pratikraj@Pratiks-MacBook-Pro common % 
pratikraj@Pratiks-MacBook-Pro common % oc get events -n platform
LAST SEEN   TYPE      REASON                   OBJECT                                                                                MESSAGE
22m         Normal    SuccessfulCreate         statefulset/chi-signoz-clickhouse-cluster-0-0                                         create Claim data-volumeclaim-template-chi-signoz-clickhouse-cluster-0-0-0 Pod chi-signoz-clickhouse-cluster-0-0-0 in StatefulSet chi-signoz-clickhouse-cluster-0-0 success
21m         Warning   FailedCreate             statefulset/chi-signoz-clickhouse-cluster-0-0                                         create Pod chi-signoz-clickhouse-cluster-0-0-0 in StatefulSet chi-signoz-clickhouse-cluster-0-0 failed error: pods "chi-signoz-clickhouse-cluster-0-0-0" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "staging-c-staging-db2-scc": Forbidden: not usable by user or serviceaccount, provider "mongodb-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: []int64{101}: 101 is not an allowed group, provider restricted-v2: .initContainers[0].runAsUser: Invalid value: 101: must be in the ranges: [1000720000, 1000729999], provider restricted-v2: .containers[0].runAsUser: Invalid value: 101: must be in the ranges: [1000720000, 1000729999], provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "noobaa-db": Forbidden: not usable by user or serviceaccount, provider "ibm-restricted-scc": Forbidden: not usable by user or serviceaccount, provider "noobaa-endpoint": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "rook-ceph": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "ibm-privileged-scc": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount, provider "rook-ceph-csi": Forbidden: not usable by user or serviceaccount]
18m         Warning   FailedCreate             statefulset/chi-signoz-clickhouse-cluster-0-0                                         create Pod chi-signoz-clickhouse-cluster-0-0-0 in StatefulSet chi-signoz-clickhouse-cluster-0-0 failed error: pods "chi-signoz-clickhouse-cluster-0-0-0" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "staging-c-staging-db2-scc": Forbidden: not usable by user or serviceaccount, provider "mongodb-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: []int64{101}: 101 is not an allowed group, provider restricted-v2: .initContainers[0].runAsUser: Invalid value: 101: must be in the ranges: [1000720000, 1000729999], provider restricted-v2: .containers[0].runAsUser: Invalid value: 101: must be in the ranges: [1000720000, 1000729999], provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "noobaa-db": Forbidden: not usable by user or serviceaccount, provider "ibm-restricted-scc": Forbidden: not usable by user or serviceaccount, provider "noobaa-endpoint": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "rook-ceph": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "ibm-privileged-scc": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount, provider "rook-ceph-csi": Forbidden: not usable by user or serviceaccount]
11m         Info      UpdateCompleted          clickhouseinstallation/signoz-clickhouse                                              Update ConfigMap platform/chi-signoz-clickhouse-common-configd
10m         Info      ReconcileCompleted       clickhouseinstallation/signoz-clickhouse                                              reconcile completed successfully, task id: 26dab86d-354f-48a3-a545-00302b94d005
21m         Info      CreateCompleted          clickhouseinstallation/signoz-clickhouse                                              Create StatefulSet platform/chi-signoz-clickhouse-cluster-0-0 - completed
22m         Info      ReconcileStarted         clickhouseinstallation/signoz-clickhouse                                              Reconcile Host start. Host: 0-0 ClickHouse version running: host is a new one, version is not not applicable
21m         Info      CreateCompleted          clickhouseinstallation/signoz-clickhouse                                              Create StatefulSet platform/chi-signoz-clickhouse-cluster-0-0 - completed
22m         Info      ReconcileStarted         clickhouseinstallation/signoz-clickhouse                                              reconcile started, task id: 26dab86d-354f-48a3-a545-00302b94d005
22m         Info      CreateStarted            clickhouseinstallation/signoz-clickhouse                                              Update StatefulSet(platform/chi-signoz-clickhouse-cluster-0-0) - started
22m         Info      UpdateCompleted          clickhouseinstallation/signoz-clickhouse                                              Update ConfigMap platform/chi-signoz-clickhouse-deploy-confd-cluster-0-0
16m         Error     ReconcileFailed          clickhouseinstallation/signoz-clickhouse                                              FAILED to reconcile StatefulSet: chi-signoz-clickhouse-cluster-0-0 CHI: signoz-clickhouse
21m         Info      CreateStarted            clickhouseinstallation/signoz-clickhouse                                              Update StatefulSet(platform/chi-signoz-clickhouse-cluster-0-0) - started
22m         Info      CreateCompleted          clickhouseinstallation/signoz-clickhouse                                              Create ConfigMap platform/chi-signoz-clickhouse-common-usersd
22m         Info      CreateCompleted          clickhouseinstallation/signoz-clickhouse                                              Create ConfigMap platform/chi-signoz-clickhouse-common-configd
22m         Info      ReconcileStarted         clickhouseinstallation/signoz-clickhouse                                              Reconcile Host 0-0 started
11m         Info      UpdateCompleted          clickhouseinstallation/signoz-clickhouse                                              Update Service success: platform/signoz-clickhouse
11m         Info      ReconcileInProgress      clickhouseinstallation/signoz-clickhouse                                              add CHI to monitoring
22m         Info      UpdateCompleted          clickhouseinstallation/signoz-clickhouse                                              Update ConfigMap platform/chi-signoz-clickhouse-common-configd
16m         Error     ReconcileFailed          clickhouseinstallation/signoz-clickhouse                                              FAILED to update err: crud error - should abort
11m         Warning   ReconcileCompleted       clickhouseinstallation/signoz-clickhouse                                              Reconcile Host completed. Host: 0-0 Failed to get ClickHouse version: failed to query
22m         Info      ReconcileStarted         clickhouseinstallation/signoz-clickhouse                                              reconcile started, task id: 802520f0-d326-4f24-9bf3-4c781ebd0ba4
16m         Info      ReconcileFailed          clickhouseinstallation/signoz-clickhouse                                              reconcile completed unsuccessfully, task id: 802520f0-d326-4f24-9bf3-4c781ebd0ba4
16m         Info      UpdateCompleted          clickhouseinstallation/signoz-clickhouse                                              Update ConfigMap platform/chi-signoz-clickhouse-common-configd
22m         Info      CreateCompleted          clickhouseinstallation/signoz-clickhouse                                              Create ConfigMap platform/chi-signoz-clickhouse-deploy-confd-cluster-0-0
21m         Info      CreateCompleted          clickhouseinstallation/signoz-clickhouse                                              OK Create Service: platform/chi-signoz-clickhouse-cluster-0-0
22m         Info      UpdateInProgress         clickhouseinstallation/signoz-clickhouse                                              Update StatefulSet(platform/chi-signoz-clickhouse-cluster-0-0) switch from Update to Recreate
22m         Info      UpdateCompleted          clickhouseinstallation/signoz-clickhouse                                              Update ConfigMap platform/chi-signoz-clickhouse-common-usersd
16m         Info      CreateCompleted          clickhouseinstallation/signoz-clickhouse                                              Create Service platform/signoz-clickhouse
21m         Info      UpdateInProgress         clickhouseinstallation/signoz-clickhouse                                              Update StatefulSet(platform/chi-signoz-clickhouse-cluster-0-0) switch from Update to Recreate
11m         Info      ProgressHostsCompleted   clickhouseinstallation/signoz-clickhouse                                              [now: 2024-07-29 15:14:47.552683964 +0000 UTC m=+416213.525664902] ProgressHostsCompleted: 1 of 1
22m         Info      UpdateCompleted          clickhouseinstallation/signoz-clickhouse                                              Update ConfigMap platform/chi-signoz-clickhouse-common-configd
10m         Info      UpdateCompleted          clickhouseinstallation/signoz-clickhouse                                              Update ConfigMap platform/chi-signoz-clickhouse-common-usersd
22m         Info      CreateStarted            clickhouseinstallation/signoz-clickhouse                                              Create StatefulSet platform/chi-signoz-clickhouse-cluster-0-0 - started
21m         Info      CreateStarted            clickhouseinstallation/signoz-clickhouse                                              Create StatefulSet platform/chi-signoz-clickhouse-cluster-0-0 - started
22m         Info      CreateStarted            clickhouseinstallation/signoz-clickhouse                                              Update StatefulSet(platform/chi-signoz-clickhouse-cluster-0-0) - started
11m         Info      ReconcileInProgress      clickhouseinstallation/signoz-clickhouse                                              remove items scheduled for deletion
22m         Normal    Provisioning             persistentvolumeclaim/data-signoz-zookeeper-0                                         External provisioner is provisioning volume for claim "platform/data-signoz-zookeeper-0"
22m         Normal    ExternalProvisioning     persistentvolumeclaim/data-signoz-zookeeper-0                                         waiting for a volume to be created, either by external provisioner "rook-ceph.rbd.csi.ceph.com" or manually created by system administrator
22m         Normal    ProvisioningSucceeded    persistentvolumeclaim/data-signoz-zookeeper-0                                         Successfully provisioned volume pvc-22d6a98e-5eb0-4b0b-9068-2938b015fc91
22m         Normal    ExternalProvisioning     persistentvolumeclaim/data-volumeclaim-template-chi-signoz-clickhouse-cluster-0-0-0   waiting for a volume to be created, either by external provisioner "rook-ceph.rbd.csi.ceph.com" or manually created by system administrator
22m         Normal    Provisioning             persistentvolumeclaim/data-volumeclaim-template-chi-signoz-clickhouse-cluster-0-0-0   External provisioner is provisioning volume for claim "platform/data-volumeclaim-template-chi-signoz-clickhouse-cluster-0-0-0"
22m         Normal    ProvisioningSucceeded    persistentvolumeclaim/data-volumeclaim-template-chi-signoz-clickhouse-cluster-0-0-0   Successfully provisioned volume pvc-3d041f47-92ca-4ea5-a55f-5bb622226b58
22m         Normal    SuccessfulCreate         statefulset/signoz-alertmanager                                                       create Claim storage-signoz-alertmanager-0 Pod signoz-alertmanager-0 in StatefulSet signoz-alertmanager success
2m5s        Warning   FailedCreate             statefulset/signoz-alertmanager                                                       create Pod signoz-alertmanager-0 in StatefulSet signoz-alertmanager failed error: pods "signoz-alertmanager-0" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "staging-c-staging-db2-scc": Forbidden: not usable by user or serviceaccount, provider "mongodb-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: []int64{65534}: 65534 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 65534: must be in the ranges: [1000720000, 1000729999], provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "noobaa-db": Forbidden: not usable by user or serviceaccount, provider "ibm-restricted-scc": Forbidden: not usable by user or serviceaccount, provider "noobaa-endpoint": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "rook-ceph": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "ibm-privileged-scc": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount, provider "rook-ceph-csi": Forbidden: not usable by user or serviceaccount]
2m5s        Normal    NoPods                   poddisruptionbudget/signoz-clickhouse-cluster                                         No matching pods found
22m         Normal    Scheduled                pod/signoz-clickhouse-operator-6d95579cd9-9pkx4                                       Successfully assigned platform/signoz-clickhouse-operator-6d95579cd9-9pkx4 to worker1.gi-tracing-poc.cp.fyre.ibm.com
22m         Normal    AddedInterface           pod/signoz-clickhouse-operator-6d95579cd9-9pkx4                                       Add eth0 [10.254.21.167/22] from ovn-kubernetes
22m         Normal    Pulling                  pod/signoz-clickhouse-operator-6d95579cd9-9pkx4                                       Pulling image "docker.io/altinity/clickhouse-operator:0.21.2"
22m         Normal    Pulled                   pod/signoz-clickhouse-operator-6d95579cd9-9pkx4                                       Successfully pulled image "docker.io/altinity/clickhouse-operator:0.21.2" in 7.040314004s (7.040326304s including waiting)
22m         Normal    Created                  pod/signoz-clickhouse-operator-6d95579cd9-9pkx4                                       Created container signoz-clickhouse-operator
22m         Normal    Started                  pod/signoz-clickhouse-operator-6d95579cd9-9pkx4                                       Started container signoz-clickhouse-operator
22m         Normal    Pulling                  pod/signoz-clickhouse-operator-6d95579cd9-9pkx4                                       Pulling image "docker.io/altinity/metrics-exporter:0.21.2"
22m         Normal    Pulled                   pod/signoz-clickhouse-operator-6d95579cd9-9pkx4                                       Successfully pulled image "docker.io/altinity/metrics-exporter:0.21.2" in 3.901561387s (3.901596639s including waiting)
22m         Normal    Created                  pod/signoz-clickhouse-operator-6d95579cd9-9pkx4                                       Created container signoz-clickhouse-metrics-exporter
22m         Normal    Started                  pod/signoz-clickhouse-operator-6d95579cd9-9pkx4                                       Started container signoz-clickhouse-metrics-exporter
22m         Normal    SuccessfulCreate         replicaset/signoz-clickhouse-operator-6d95579cd9                                      Created pod: signoz-clickhouse-operator-6d95579cd9-9pkx4
22m         Normal    ScalingReplicaSet        deployment/signoz-clickhouse-operator                                                 Scaled up replica set signoz-clickhouse-operator-6d95579cd9 to 1
22m         Normal    Provisioning             persistentvolumeclaim/signoz-db-signoz-query-service-0                                External provisioner is provisioning volume for claim "platform/signoz-db-signoz-query-service-0"
22m         Normal    ExternalProvisioning     persistentvolumeclaim/signoz-db-signoz-query-service-0                                waiting for a volume to be created, either by external provisioner "rook-ceph.rbd.csi.ceph.com" or manually created by system administrator
22m         Normal    ProvisioningSucceeded    persistentvolumeclaim/signoz-db-signoz-query-service-0                                Successfully provisioned volume pvc-e55aa83c-3766-442a-b873-d24d84005ebd
22m         Normal    Scheduled                pod/signoz-frontend-64c68d5b6c-hdlsp                                                  Successfully assigned platform/signoz-frontend-64c68d5b6c-hdlsp to worker1.gi-tracing-poc.cp.fyre.ibm.com
22m         Normal    AddedInterface           pod/signoz-frontend-64c68d5b6c-hdlsp                                                  Add eth0 [10.254.21.166/22] from ovn-kubernetes
22m         Normal    Pulling                  pod/signoz-frontend-64c68d5b6c-hdlsp                                                  Pulling image "docker.io/busybox:1.35"
22m         Normal    Pulled                   pod/signoz-frontend-64c68d5b6c-hdlsp                                                  Successfully pulled image "docker.io/busybox:1.35" in 3.256069282s (3.256079022s including waiting)
22m         Normal    Created                  pod/signoz-frontend-64c68d5b6c-hdlsp                                                  Created container signoz-frontend-init
22m         Normal    Started                  pod/signoz-frontend-64c68d5b6c-hdlsp                                                  Started container signoz-frontend-init
22m         Normal    SuccessfulCreate         replicaset/signoz-frontend-64c68d5b6c                                                 Created pod: signoz-frontend-64c68d5b6c-hdlsp
22m         Normal    ScalingReplicaSet        deployment/signoz-frontend                                                            Scaled up replica set signoz-frontend-64c68d5b6c to 1
2m5s        Warning   FailedCreate             daemonset/signoz-k8s-infra-otel-agent                                                 Error creating: pods "signoz-k8s-infra-otel-agent-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "staging-c-staging-db2-scc": Forbidden: not usable by user or serviceaccount, provider "mongodb-scc": Forbidden: not usable by user or serviceaccount, spec.volumes[1]: Invalid value: "hostPath": hostPath volumes are not allowed to be used, spec.volumes[2]: Invalid value: "hostPath": hostPath volumes are not allowed to be used, provider restricted-v2: .containers[0].containers[0].hostPort: Invalid value: 13133: Host ports are not allowed to be used, provider restricted-v2: .containers[0].containers[0].hostPort: Invalid value: 8888: Host ports are not allowed to be used, provider restricted-v2: .containers[0].containers[0].hostPort: Invalid value: 4317: Host ports are not allowed to be used, provider restricted-v2: .containers[0].containers[0].hostPort: Invalid value: 4318: Host ports are not allowed to be used, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "noobaa-db": Forbidden: not usable by user or serviceaccount, provider "ibm-restricted-scc": Forbidden: not usable by user or serviceaccount, provider "noobaa-endpoint": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "rook-ceph": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "ibm-privileged-scc": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount, provider "rook-ceph-csi": Forbidden: not usable by user or serviceaccount]
22m         Normal    Scheduled                pod/signoz-k8s-infra-otel-deployment-579bcdb68c-k8m5r                                 Successfully assigned platform/signoz-k8s-infra-otel-deployment-579bcdb68c-k8m5r to worker1.gi-tracing-poc.cp.fyre.ibm.com
22m         Normal    AddedInterface           pod/signoz-k8s-infra-otel-deployment-579bcdb68c-k8m5r                                 Add eth0 [10.254.21.169/22] from ovn-kubernetes
22m         Normal    Pulling                  pod/signoz-k8s-infra-otel-deployment-579bcdb68c-k8m5r                                 Pulling image "docker.io/otel/opentelemetry-collector-contrib:0.88.0"
22m         Normal    Pulled                   pod/signoz-k8s-infra-otel-deployment-579bcdb68c-k8m5r                                 Successfully pulled image "docker.io/otel/opentelemetry-collector-contrib:0.88.0" in 5.431497688s (5.431505638s including waiting)
22m         Normal    Created                  pod/signoz-k8s-infra-otel-deployment-579bcdb68c-k8m5r                                 Created container signoz-k8s-infra-otel-deployment
22m         Normal    Started                  pod/signoz-k8s-infra-otel-deployment-579bcdb68c-k8m5r                                 Started container signoz-k8s-infra-otel-deployment
22m         Normal    SuccessfulCreate         replicaset/signoz-k8s-infra-otel-deployment-579bcdb68c                                Created pod: signoz-k8s-infra-otel-deployment-579bcdb68c-k8m5r
22m         Normal    ScalingReplicaSet        deployment/signoz-k8s-infra-otel-deployment                                           Scaled up replica set signoz-k8s-infra-otel-deployment-579bcdb68c to 1
22m         Normal    Scheduled                pod/signoz-otel-collector-689c6c4bb5-pmdsn                                            Successfully assigned platform/signoz-otel-collector-689c6c4bb5-pmdsn to worker1.gi-tracing-poc.cp.fyre.ibm.com
22m         Normal    AddedInterface           pod/signoz-otel-collector-689c6c4bb5-pmdsn                                            Add eth0 [10.254.21.168/22] from ovn-kubernetes
22m         Normal    Pulling                  pod/signoz-otel-collector-689c6c4bb5-pmdsn                                            Pulling image "docker.io/groundnuty/k8s-wait-for:v2.0"
22m         Normal    Pulled                   pod/signoz-otel-collector-689c6c4bb5-pmdsn                                            Successfully pulled image "docker.io/groundnuty/k8s-wait-for:v2.0" in 4.928466235s (4.928489286s including waiting)
22m         Normal    Created                  pod/signoz-otel-collector-689c6c4bb5-pmdsn                                            Created container signoz-otel-collector-migrate-init
22m         Normal    Started                  pod/signoz-otel-collector-689c6c4bb5-pmdsn                                            Started container signoz-otel-collector-migrate-init
22m         Normal    SuccessfulCreate         replicaset/signoz-otel-collector-689c6c4bb5                                           Created pod: signoz-otel-collector-689c6c4bb5-pmdsn
22m         Normal    Scheduled                pod/signoz-otel-collector-metrics-7dcf984d77-f5rcb                                    Successfully assigned platform/signoz-otel-collector-metrics-7dcf984d77-f5rcb to worker2.gi-tracing-poc.cp.fyre.ibm.com
22m         Normal    AddedInterface           pod/signoz-otel-collector-metrics-7dcf984d77-f5rcb                                    Add eth0 [10.254.15.93/22] from ovn-kubernetes
22m         Normal    Pulling                  pod/signoz-otel-collector-metrics-7dcf984d77-f5rcb                                    Pulling image "docker.io/groundnuty/k8s-wait-for:v2.0"
22m         Normal    Pulled                   pod/signoz-otel-collector-metrics-7dcf984d77-f5rcb                                    Successfully pulled image "docker.io/groundnuty/k8s-wait-for:v2.0" in 3.397796113s (3.397804893s including waiting)
22m         Normal    Created                  pod/signoz-otel-collector-metrics-7dcf984d77-f5rcb                                    Created container signoz-otel-collector-metrics-migrate-init
22m         Normal    Started                  pod/signoz-otel-collector-metrics-7dcf984d77-f5rcb                                    Started container signoz-otel-collector-metrics-migrate-init
22m         Normal    SuccessfulCreate         replicaset/signoz-otel-collector-metrics-7dcf984d77                                   Created pod: signoz-otel-collector-metrics-7dcf984d77-f5rcb
22m         Normal    ScalingReplicaSet        deployment/signoz-otel-collector-metrics                                              Scaled up replica set signoz-otel-collector-metrics-7dcf984d77 to 1
22m         Normal    ScalingReplicaSet        deployment/signoz-otel-collector                                                      Scaled up replica set signoz-otel-collector-689c6c4bb5 to 1
22m         Warning   FailedScheduling         pod/signoz-query-service-0                                                            0/8 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/8 nodes are available: 8 Preemption is not helpful for scheduling..
22m         Normal    Scheduled                pod/signoz-query-service-0                                                            Successfully assigned platform/signoz-query-service-0 to worker1.gi-tracing-poc.cp.fyre.ibm.com
22m         Normal    SuccessfulAttachVolume   pod/signoz-query-service-0                                                            AttachVolume.Attach succeeded for volume "pvc-e55aa83c-3766-442a-b873-d24d84005ebd"
22m         Normal    AddedInterface           pod/signoz-query-service-0                                                            Add eth0 [10.254.21.170/22] from ovn-kubernetes
22m         Normal    Pulled                   pod/signoz-query-service-0                                                            Container image "docker.io/busybox:1.35" already present on machine
22m         Normal    Created                  pod/signoz-query-service-0                                                            Created container signoz-query-service-init
22m         Normal    Started                  pod/signoz-query-service-0                                                            Started container signoz-query-service-init
22m         Normal    SuccessfulCreate         statefulset/signoz-query-service                                                      create Claim signoz-db-signoz-query-service-0 Pod signoz-query-service-0 in StatefulSet signoz-query-service success
22m         Normal    SuccessfulCreate         statefulset/signoz-query-service                                                      create Pod signoz-query-service-0 in StatefulSet signoz-query-service successful
22m         Normal    Scheduled                pod/signoz-schema-migrator-init-dqwtk                                                 Successfully assigned platform/signoz-schema-migrator-init-dqwtk to worker2.gi-tracing-poc.cp.fyre.ibm.com
22m         Normal    AddedInterface           pod/signoz-schema-migrator-init-dqwtk                                                 Add eth0 [10.254.15.94/22] from ovn-kubernetes
22m         Normal    Pulling                  pod/signoz-schema-migrator-init-dqwtk                                                 Pulling image "docker.io/busybox:1.35"
22m         Normal    Pulled                   pod/signoz-schema-migrator-init-dqwtk                                                 Successfully pulled image "docker.io/busybox:1.35" in 2.715409223s (2.715420093s including waiting)
22m         Normal    Created                  pod/signoz-schema-migrator-init-dqwtk                                                 Created container signoz-schema-migrator-init
22m         Normal    Started                  pod/signoz-schema-migrator-init-dqwtk                                                 Started container signoz-schema-migrator-init
22m         Normal    SuccessfulCreate         job/signoz-schema-migrator-init                                                       Created pod: signoz-schema-migrator-init-dqwtk
22m         Normal    SuccessfulCreate         statefulset/signoz-zookeeper                                                          create Claim data-signoz-zookeeper-0 Pod signoz-zookeeper-0 in StatefulSet signoz-zookeeper success
2m5s        Warning   FailedCreate             statefulset/signoz-zookeeper                                                          create Pod signoz-zookeeper-0 in StatefulSet signoz-zookeeper failed error: pods "signoz-zookeeper-0" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider "staging-c-staging-db2-scc": Forbidden: not usable by user or serviceaccount, provider "mongodb-scc": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .spec.securityContext.fsGroup: Invalid value: []int64{1001}: 1001 is not an allowed group, provider restricted-v2: .containers[0].runAsUser: Invalid value: 1001: must be in the ranges: [1000720000, 1000729999], provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "noobaa-db": Forbidden: not usable by user or serviceaccount, provider "ibm-restricted-scc": Forbidden: not usable by user or serviceaccount, provider "noobaa-endpoint": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "rook-ceph": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "ibm-privileged-scc": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount, provider "rook-ceph-csi": Forbidden: not usable by user or serviceaccount]
22m         Normal    ExternalProvisioning     persistentvolumeclaim/storage-signoz-alertmanager-0                                   waiting for a volume to be created, either by external provisioner "rook-ceph.rbd.csi.ceph.com" or manually created by system administrator
22m         Normal    Provisioning             persistentvolumeclaim/storage-signoz-alertmanager-0                                   External provisioner is provisioning volume for claim "platform/storage-signoz-alertmanager-0"
22m         Normal    ProvisioningSucceeded    persistentvolumeclaim/storage-signoz-alertmanager-0                                   Successfully provisioned volume pvc-29d826e4-e7cc-41d4-baca-577072ef9d9b
pratikraj@Pratiks-MacBook-Pro common % 
pratikraj@Pratiks-MacBook-Pro common %
welcome[bot] commented 1 month ago

Thanks for opening this issue. A team member should give feedback soon. In the meantime, feel free to check out the contributing guidelines.

Rajpratik71 commented 1 month ago

looks like pods are failing due to "SCC" restriction of OpenShift.

Similar "SCC" needs to be created and configured SCC for eBPF