[Loki] Loki supports Istio?

[kingbj940429]

I'm using Loki with helm chart.

But, there is a problem about istio

I already read Install on Istio But, It's not helpful..

If not use istio sidecar, It works well!

Loki Chart version : 5.42.0 Istio Version : 1.18

loki-read logs

################################################################################
# Loki
################################################################################
loki:
  auth_enabled: false

#  podLabels:
#    sidecar.istio.io/inject: "false"

  readinessProbe:
    httpGet:
      path: /ready
      port: http-metrics
    initialDelaySeconds: 30
    timeoutSeconds: 1
    periodSeconds: 10
    failureThreshold: 30

  commonConfig:
    replication_factor: 1

  compactor:
    compaction_interval: 10m
    retention_delete_delay: 2h
    retention_delete_worker_count: 150
    retention_enabled: true
    shared_store: s3
    working_directory: /var/loki/compactor

  schemaConfig:
    configs:
      - from: 2022-11-21
        store: boltdb-shipper
        object_store: s3
        schema: v11
        index:
          prefix: loki_index_
          period: 24h

  storage_config:
    boltdb_shipper:
      active_index_directory: /var/loki/index
      cache_location: /var/loki/cache
      cache_ttl: 24h
      shared_store: s3
    hedging:
      at: "250ms"
      max_per_second: 20
      up_to: 3

  storage:
    type: s3
    s3:
      region: ap-northeast-2
    bucketNames:
      chunks: eks-loki-logs
      ruler: eks-loki-logs
      admin: eks-loki-logs

  memcached:
    chunk_cache:
      enabled: true
    results_cache:
      enabled: true

#  memberlistConfig:
#    join_members:
#      - loki-memberlist:7946

################################################################################
# Write
################################################################################
write:
  replicas: 1

#  podLabels:
#    sidecar.istio.io/inject: "false"

  extraArgs:
    - -validation.max-entries-limit=20000
    - -querier.engine.timeout=5m
    - -querier.query-timeout=5m
    - -server.http-read-timeout=5m
    - -server.http-write-timeout=5m

  resources:
    requests:
      cpu: 250m
      memory: 512Mi
    limits:
      cpu: 500m
      memory: 1Gi

  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 6

  persistence:
    volumeClaimsEnabled: true
    size: 50Gi
    storageClass: efs-sc
    enableStatefulSetAutoDeletePVC: true

################################################################################
# Read
################################################################################
read:
  replicas: 1

#  podLabels:
#    sidecar.istio.io/inject: "false"

  extraArgs:
    - -validation.max-entries-limit=20000
    - -querier.engine.timeout=5m
    - -querier.query-timeout=5m
    - -server.http-read-timeout=5m
    - -server.http-write-timeout=5m

  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: 1000m
      memory: 2Gi

  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 6

  persistence:
    enableStatefulSetAutoDeletePVC: true
    size: 50Gi
    storageClass: efs-sc

################################################################################
# Backend
################################################################################
backend:
  replicas: 1

#  podLabels:
#    sidecar.istio.io/inject: "false"

  resources:
    requests:
      cpu: 250m
      memory: 512Mi
    limits:
      cpu: 500m
      memory: 1Gi

  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 6

  persistence:
    volumeClaimsEnabled: true
    size: 50Gi
    storageClass: efs-sc

sidecar:
  resources:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      cpu: 100m
      memory: 512Mi

################################################################################
# Gateway
################################################################################
gateway:
  enabled: true

#  podLabels:
#    sidecar.istio.io/inject: "false"

  replicas: 1

  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 6

  resources:
    requests:
      cpu: 250m
      memory: 512Mi
    limits:
      cpu: 500m
      memory: 1Gi

################################################################################
# Canary
################################################################################
test:
  enabled: false

monitoring:
  rules:
    labels:
      release: prometheus

  serviceMonitor:
    labels:
      release: prometheus

  selfMonitoring:
    enabled: false

    grafanaAgent:
      installOperator: false

  lokiCanary:
    enabled: false

[cyriltovena]

👋

It should yes, not sure how your istio is configured but the log message seems to say that something is wrong with the ring which is made using memberlist.

This happens on port 7946 I think, and it uses UDP which means the doc you pointed out seems wrong. Can you change that to see if it makes any difference ?

[kingbj940429]

this is loki-memberlist service manifest

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: loki
    meta.helm.sh/release-namespace: monitoring-system
  creationTimestamp: "2024-02-04T03:29:37Z"
  labels:
    app.kubernetes.io/instance: loki
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: loki
    app.kubernetes.io/version: 2.9.3
    helm.sh/chart: loki-5.42.0
  name: loki-memberlist
  namespace: monitoring-system
  resourceVersion: "20078658"
  uid: 629eb006-f50d-4440-91ea-58730128e653
spec:
  clusterIP: None
  clusterIPs:
  - None
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: tcp
    port: 7946
    protocol: UDP #-- changed TCP to UDP
    targetPort: http-memberlist
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/instance: loki
    app.kubernetes.io/name: loki
    app.kubernetes.io/part-of: memberlist
  sessionAffinity: None
  type: ClusterIP

this is logs when changed TCP to UDP

this is ring status - when click refresh button, show different result