Clickhouse operator errors out and causes other pods not to start

[fclaudiopalmeira]

I have deployed Signoz on AKS (Azure Kubernetes Service) by following the oficial Signoz Installation guide, but unfortunately the clickhouse-operator pod has some errors in it, checking the logs for it I get:

2022/11/18 02:09:43 Shellcode Length:  46
2022/11/18 02:09:43 Overlapping Sections in Generated Elf: .rodata
2022/11/18 02:09:43 Start section header table at: 308668f
2022/11/18 02:09:43 Error writing file: open /clickhouse-operator: permission denied

Again under the same pod while checking the logs for the metrics-exporte i get:

2022/11/18 02:09:43 Shellcode Length:  46
2022/11/18 02:09:43 Start section header table at: 2996a50
2022/11/18 02:09:43 Error writing file: open /metrics-exporter: permission denied

And both instrumentation give similar errors:

{"level":"info","ts":1668737577.2842069,"caller":"process/discover.go:37","msg":"process not found yet, trying again soon","exe_path":"/clickhouse-operator"}

and

{"level":"info","ts":1668737609.4530525,"caller":"process/discover.go:37","msg":"process not found yet, trying again soon","exe_path":"/metrics-exporter"}

And it cascades on series of errors: The alertmanager can't find the query-service which in turn doens't start because it can't find ClickHouseDB also the Otel-Collector fails with a similar permission denied error:

2022/11/18 02:15:45 Shellcode Length:  46
2022/11/18 02:15:46 Start section header table at: a5006ba
2022/11/18 02:15:46 Error writing file: open /signoz-collector: permission denied

but oddly enough under the init it shows:

clickhouse ready, starting otel collector now

Everything seems to be related to that permission denied error. here is the list of my storage classes in there:

PS C:\Users\myuser> kubectl get sc
NAME                    PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
azurefile               file.csi.azure.com   Delete          Immediate              true                   186d
azurefile-csi           file.csi.azure.com   Delete          Immediate              true                   186d
azurefile-csi-premium   file.csi.azure.com   Delete          Immediate              true                   186d
azurefile-premium       file.csi.azure.com   Delete          Immediate              true                   186d
default (default)       disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   186d
managed                 disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   186d
managed-csi             disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   186d
managed-csi-premium     disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   186d
managed-premium         disk.csi.azure.com   Delete          WaitForFirstConsumer   true                   186d

I have tried: azurefile-csi-premium managed-csi-premium None worked. Any insight on this issue

[pranay01]

@fclaudiopalmeira Whats the version of k8s you are using in EKS?

[fclaudiopalmeira]

HI @pranay01 I am using AKS and it is version 1.24.3