Open dmitry-mightydevops opened 3 years ago
I even went inside the node running loki, inside the loki container and this is the config:
PID USER TIME COMMAND
1 loki 0:03 /usr/bin/loki -config.file=/etc/loki/loki.yaml
46 loki 0:00 ash
60 loki 0:00 ps aufx
/ $ cat /etc/loki/loki.yaml
auth_enabled: false
chunk_store_config:
max_look_back_period: 0s
compactor:
shared_store: filesystem
working_directory: /data/loki/boltdb-shipper-compactor
ingester:
chunk_block_size: 262144
chunk_idle_period: 3m
chunk_retain_period: 1m
lifecycler:
ring:
kvstore:
store: inmemory
replication_factor: 1
max_transfer_retries: 0
limits_config:
enforce_metric_name: false
ingestion_burst_size_mb: 20
ingestion_rate_mb: 10
reject_old_samples: true
reject_old_samples_max_age: 168h
schema_config:
configs:
- from: "2020-10-24"
index:
period: 24h
prefix: index_
object_store: filesystem
schema: v11
store: boltdb-shipper
server:
http_listen_port: 3100
storage_config:
boltdb_shipper:
active_index_directory: /data/loki/boltdb-shipper-active
cache_location: /data/loki/boltdb-shipper-cache
cache_ttl: 24h
shared_store: filesystem
filesystem:
directory: /data/loki/chunks
table_manager:
retention_deletes_enabled: false
retention_period: 0s/ $
So these values got applied (from the HELM chart), but I still get that original error I reported. I saw this discussion https://community.grafana.com/t/discarding-promtail-log-entries-en-masse/41128 but apparently something else I have missed.
@dmitry-mightydevops try the per_stream_rate_limit
https://grafana.com/docs/loki/latest/configuration/#limits_config
My experimental config, which seemed to help get rid of the 429 code. Might come in handy.
retention_period: 72h
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
max_cache_freshness_per_query: 10m
split_queries_by_interval: 15m
# for big logs tune
per_stream_rate_limit: 512M
per_stream_rate_limit_burst: 1024M
cardinality_limit: 200000
ingestion_burst_size_mb: 1000
ingestion_rate_mb: 10000
max_entries_limit_per_query: 1000000
max_label_value_length: 20480
max_label_name_length: 10240
max_label_names_per_series: 300
@dmitry-mightydevops
I came along with same issue, do you have fixed it? Could you share your config? Thanks.
Used the loki chart https://github.com/grafana/helm-charts/blob/main/charts/loki/values.yaml
get the following in my promtail pods:
values.promtail.yaml
values.loki.yaml
and so adding the following two lines into limits_config had no effect.
I'm using latest charts