search

hrxi / tracing-loki

A tracing layer for shipping logs to Grafana Loki
Apache License 2.0
50 stars 20 forks source link

HTTP status client error (429 Too Many Requests) #24

Open zohnannor opened 1 year ago

zohnannor commented 1 year ago

I am getting this:

2023-10-09T13:04:07.696099Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=1 backoff_time=0ns error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:07.702310Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=2 backoff_time=500ms error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:09.162408Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=1 backoff_time=0ns error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:09.176558Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=1 backoff_time=0ns error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:09.178456Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=2 backoff_time=500ms error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:10.427208Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=1 backoff_time=0ns error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:10.431438Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=2 backoff_time=500ms error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:11.967807Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=1 backoff_time=0ns error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
2023-10-09T13:04:11.970397Z ERROR                        main tracing_loki: couldn't send logs to loki error_count=2 backoff_time=500ms error=HTTP status client error (429 Too Many Requests) for url (http://localhost:3100/loki/api/v1/push)
let (loki_layer, loki_task_ctl, loki_task) = tracing_loki::builder()
    .label("host", gethostname::gethostname().to_str().unwrap())
    .unwrap()
    .build_controller_url(
        tracing_loki::url::Url::parse("http://localhost:3100").unwrap(),
    )
    .unwrap();
tokio::spawn(loki_task);

loki.yml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

query_range:
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

schema_config:
  configs:
    - store: tsdb
      object_store: filesystem
      schema: v12
      index:
        prefix: index_
        period: 24h

analytics:
  reporting_enabled: false
zohnannor commented 1 year ago

I have increased the limit:


limits_config:
  per_stream_rate_limit: 128MB
  per_stream_rate_limit_burst: 128MB
  ingestion_rate_mb: 128
  ingestion_burst_size_mb: 128

And it doesn't error anymore.

However, I noticed in the code that BackgroundTask is essentially a tight loop. The process consumes 100% of one CPU core.

image

Can I configure it so that it doesn't?

And no logs are being sent to Loki?

image