[bitnami/grafana-loki] Loki return (500) empty ring

HuyDo-95g commented 9 months ago

Name and Version

bitnami/grafana-loki 2.11.2

What architecture are you using?

None

What steps will reproduce the bug?

Deploy the helm chart loki verion: 2.11.2 with the following values.yaml:

nodeSelector:
nodepool: "monitoring"

serviceAccount:
create: true
annotations:
  eks.amazonaws.com/role-arn: "arn:aws:iam::12345678:role/role_name"

loki:
overrideConfiguration: 
  storage_config:
    max_parallel_get_chunk: 10000
    max_chunk_batch_size: 50000
    aws:
      region: eu-central-1
      insecure: false
      s3: s3://region/s3-name
      sse_encryption: true
      sse:
        type: SSE-KMS
        kms_key_id: <key_id>
    boltdb_shipper:
      shared_store: s3

promtail:
enabled: false

querier:
resources:
  limits:
    cpu: 2
    memory: 2Gi
  requests:
    cpu: 1
    memory: 1Gi
persistence:
  enabled: false
extraEnvVars:
  - name: LOG_LEVEL
    value: error

queryFrontend:
resources:
  limits:
    cpu: 2
    memory: 2Gi
  requests:
    cpu: 1
    memory: 1Gi
extraEnvVars:
  - name: LOG_LEVEL
    value: error

compactor:
persistence:
  enabled: false
extraEnvVars:
  - name: LOG_LEVEL
    value: error

ingester:
extraEnvVars:
  - name: LOG_LEVEL
    value: error

distributor:
extraEnvVars:
  - name: LOG_LEVEL
    value: error

Deploy grafana datasource as below:

apiVersion: integreatly.org/v1alpha1
kind: GrafanaDataSource
metadata:
  name: loki
spec:
  name: loki
  datasources:
    - name: Loki
      type: loki
      url: http://loki-query-frontend:3100
      access: proxy
      isDefault: false
      editable: true
      jsonData:
        maxLines: 300
        timeout: 300

Go to grafana UI datasource.
Select loki then run "Save & Test"

What is the expected behavior?

Run successfully without any issue

What do you see instead?

I see

and

Additional information

Environment:

Infrastructure: k8s 1.25
Deployment tool: helm
network: istio-proxy injected as a sidecar in every pods
Logs are sent from container with image: grafana/fluent-bit-plugin-loki

javsalgar commented 9 months ago

Hi,

In order to discard this is an issue with the cloud provider, did you try in another cluster like minikube?

HuyDo-95g commented 9 months ago

Hey there! I gave a shot at setting up a similar configuration in Azure, using Blob as storage. Ran into a snag there too. Also, encountered a similar hiccup when setting up Grafana Tempo previously.

Mauraza commented 9 months ago

Hi @HuyDo-95g,

Could you check this issue https://github.com/grafana/loki/issues/7614 too?

HuyDo-95g commented 9 months ago

Hi @Mauraza I have checked that out. Seem like it does not help anything

Mauraza commented 9 months ago

Hi @HuyDo-95g,

But that is an open issue in grafana-loki, not relative to the bitnami chart. Let's see if they can solve the bug.

HuyDo-95g commented 8 months ago

Hi @Mauraza Look like grafana-loki team has not solve the bug yet. Are there anything we can do from bitnami side?

Mauraza commented 8 months ago

Hi @HuyDo-95g,

If it is a bug of grafana-loki, we can't do too much.

github-actions[bot] commented 7 months ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

HuyDo-95g commented 7 months ago

The issue is still existed.

Mauraza commented 7 months ago

I will put this issue 'on hold' until the bug reported in https://github.com/grafana/loki/issues/7614 is fixed.

bitnami / charts