Self Hosted Signoz deployed on AWS EKS failing Clickhouse Authentication.

[lazyboson]

Bug description

I have an EKS cluster running with version 1.25. I have created seperate managednodegroup in cluster say otel. then i have tried to install helm chart on EKS. i have also created required storage class[Screenshot attached].

Here is my override-values.yaml

frontend:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: instance
          operator: In
          values:
          - otel

alertmanager:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: instance
          operator: In
          values:
          - otel

otelCollector:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: instance
          operator: In
          values:
          - otel

schemaMigrator:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: instance
          operator: In
          values:
          - otel

global:
  storageClass: gp2-resizable
  cloud: aws

clickhouse:
  installCustomStorageClass: true

Now query-service is crashing with following logs -

{"level":"WARN","timestamp":"2024-04-22T12:01:37.208Z","caller":"query-service/main.go:141","msg":"No JWT secret key is specified."}
{"level":"INFO","timestamp":"2024-04-22T12:01:37.258Z","caller":"license/manager.go:128","msg":"No active license found, defaulting to basic plan"}
{"level":"INFO","timestamp":"2024-04-22T12:01:37.259Z","caller":"app/server.go:138","msg":"Using ClickHouse as datastore ..."}
{"level":"INFO","timestamp":"2024-04-22T12:01:37.259Z","caller":"clickhouseReader/options.go:109","msg":"Connecting to Clickhouse","at":"my-release-clickhouse:9000","MaxIdleConns":50,"MaxOpenConns":100,"DialTimeout":5000}
{"level":"FATAL","timestamp":"2024-04-22T12:01:37.261Z","caller":"clickhouseReader/reader.go:145","msg":"failed to initialize ClickHouse","error":"error connecting to primary db: code: 516, message: admin: Authentication failed: password is incorrect, or there is no user with such name.","stacktrace":"go.signoz.io/signoz/pkg/query-service/app/clickhouseReader.NewReader\n\t/home/runner/work/signoz/signoz/pkg/query-service/app/clickhouseReader/reader.go:145\ngo.signoz.io/signoz/ee/query-service/app/db.NewDataConnector\n\t/home/runner/work/signoz/signoz/ee/query-service/app/db/reader.go:29\ngo.signoz.io/signoz/ee/query-service/app.NewServer\n\t/home/runner/work/signoz/signoz/ee/query-service/app/server.go:139\nmain.main\n\t/home/runner/work/signoz/signoz/ee/query-service/main.go:146\nruntime.main\n\t/opt/hostedtoolcache/go/1.21.9/x64/src/runtime/proc.go:267"}

here is query-service describe. which matches with default user and password of clickhouse as passed in your helm chart values.yaml.

I am stuck any suggestion will be helpful.

[welcome[bot]]

Thanks for opening this issue. A team member should give feedback soon. In the meantime, feel free to check out the contributing guidelines.