search

grafana / helm-charts

Apache License 2.0
1.67k stars 2.29k forks source link

[loki-simple-scalable] Please provide ingest configuration:ingestion_rate_xx per_stream_rate_limit_xx #1479

Open liuxuzxx opened 2 years ago

liuxuzxx commented 2 years ago

When we use loki-simple-scalable loki-read and loki-write replication count is 3! when the log speed is 30MB/s the error is occure:

10.192.0.7 - - [09/Jun/2022:10:06:38 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"
10.0.2.129 - - [09/Jun/2022:10:06:38 +0000]  429 "POST /loki/api/v1/push HTTP/1.1" 227 "-" "promtail/2.5.0" "-"
10.192.0.3 - - [09/Jun/2022:10:06:38 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"
10.0.2.129 - - [09/Jun/2022:10:06:38 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"
10.192.0.3 - - [09/Jun/2022:10:06:38 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"
10.192.0.7 - - [09/Jun/2022:10:06:39 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"
10.192.0.7 - - [09/Jun/2022:10:06:39 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"
10.192.0.8 - - [09/Jun/2022:10:06:39 +0000]  429 "POST /loki/api/v1/push HTTP/1.1" 227 "-" "promtail/2.5.0" "-"
10.192.0.3 - - [09/Jun/2022:10:06:39 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"
10.0.2.129 - - [09/Jun/2022:10:06:39 +0000]  429 "POST /loki/api/v1/push HTTP/1.1" 227 "-" "promtail/2.5.0" "-"
10.192.0.3 - - [09/Jun/2022:10:06:39 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"
10.192.0.7 - - [09/Jun/2022:10:06:39 +0000]  429 "POST /loki/api/v1/push HTTP/1.1" 227 "-" "promtail/2.5.0" "-"
10.192.0.4 - - [09/Jun/2022:10:06:39 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"
10.192.0.7 - - [09/Jun/2022:10:06:39 +0000]  429 "POST /loki/api/v1/push HTTP/1.1" 227 "-" "promtail/2.5.0" "-"
10.192.0.3 - - [09/Jun/2022:10:06:40 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"
10.0.2.129 - - [09/Jun/2022:10:06:40 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"
10.192.0.7 - - [09/Jun/2022:10:06:40 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"
10.192.0.8 - - [09/Jun/2022:10:06:40 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"
10.192.0.3 - - [09/Jun/2022:10:06:40 +0000]  429 "POST /loki/api/v1/push HTTP/1.1" 227 "-" "promtail/2.5.0" "-"
10.192.0.7 - - [09/Jun/2022:10:06:40 +0000]  429 "POST /loki/api/v1/push HTTP/1.1" 227 "-" "promtail/2.5.0" "-"
10.192.0.7 - - [09/Jun/2022:10:06:40 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"
10.192.0.7 - - [09/Jun/2022:10:06:40 +0000]  204 "POST /loki/api/v1/push HTTP/1.1" 0 "-" "promtail/2.5.0" "-"

the loki-gateway return http 429(Too man request)

liuxuzxx commented 2 years ago

Please proviate ingrestion_rate_xx and per_stream_rate_limit_xx configuration, the default values is 4MB/s , when the replication count is 3,one loki-write can use 1.33MB/s speed!

  auth_enabled: false
    common:
      path_prefix: /var/loki
      replication_factor: 3
      storage:
        s3:
          ......
    limits_config:
      ......

      #add the configuration
      #每个实例只会分到1.3MB/s
      ingestion_rate_mb: 100
      ingestion_burst_size_mb: 150
      per_stream_rate_limit: "100MB"
      per_stream_rate_limit_burst: "300MB"
liuxuzxx commented 2 years ago

and the values.yaml:

loki:
  ......
  config: |
    {{- if .Values.enterprise.enabled}}
    {{- tpl .Values.enterprise.config . }}
    {{- else }}
    auth_enabled: {{ .Values.loki.authEnabled }}
    {{- end }}

    server:
      http_listen_port: 3100
      grpc_listen_port: 9095

    memberlist:
      join_members:
        - {{ include "loki.name" . }}-memberlist

    {{- if .Values.loki.commonConfig}}
    common:
    {{- toYaml .Values.loki.commonConfig | nindent 2}}
      storage:
      {{- include "loki.commonStorageConfig" . | nindent 4}}
    {{- end}}

    limits_config:
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
      max_cache_freshness_per_query: 10m
      split_queries_by_interval: 15m
      # add rate configuration
      ingestion_rate_mb: 100
      ingestion_burst_size_mb: 150
      per_stream_rate_limit: "100MB"
      per_stream_rate_limit_burst: "300MB"
liuxuzxx commented 2 years ago

@trevorwhitney

liuxuzxx commented 2 years ago

Can the loki-simple-scalable loki-gateway provider /metircs prometheus proxy

trevorwhitney commented 2 years ago

Is this ask here just to add better limits to the default config? I like this idea, but I think there's a bit more work to be done here in terms of a cluster planning tools. Do you have an issue overriding the config for now?

liuxuzxx commented 2 years ago

Yes ,the values.yaml must provide ingestion_rate_xx config information ,and must provide ingestor config information:

ingester:
  flush_check_period: 5s
  flush_op_timeout: 100m
liuxuzxx commented 2 years ago

And how to collect loki-read-xx and loki-write-xx metrics? Can the loki-gateway give a metrics path proxy?

liuxuzxx commented 2 years ago

When the storage type is local,the config information is:

      path_prefix: /var/loki
      replication_factor: 3
      storage:
        filesystem:
          chunks_directory: /var/loki/chunks
          rules_directory: /var/loki/rules

the path_prefix is loki-write-xx write path and the chunks_directory is sharedPath?