Hi, I got some weird problem when using the loki in simple scalable mode.
Here is my architecture:1 loki-read docker container, 2 loki-write docker containers which receive data from 2 vectors respectively and save data to s3, all run at one linux system.
The problem is: when I generate 1k logs and save them to s3 through loki system, I query these data from grafana, somtimes I get 280 logs, somtimes I get 720 logs, 280 + 720 = 1k.
I retry generate&query many times, I get 550/450, 1k/0(sometimes 1k, sometimes 0), 342/658...
it seems like the logs are saved in two parts at s3, and the query will get logs round-robin at these parts.
Hi, I got some weird problem when using the loki in simple scalable mode.
Here is my architecture:1 loki-read docker container, 2 loki-write docker containers which receive data from 2 vectors respectively and save data to s3, all run at one linux system.
The problem is: when I generate 1k logs and save them to s3 through loki system, I query these data from grafana, somtimes I get 280 logs, somtimes I get 720 logs, 280 + 720 = 1k. I retry generate&query many times, I get 550/450, 1k/0(sometimes 1k, sometimes 0), 342/658... it seems like the logs are saved in two parts at s3, and the query will get logs round-robin at these parts.
Here is my loki config.
auth_enabled: false
target: write/read
server: http_listen_port: 3300 grpc_listen_port: 9300
memberlist: bind_port: 7300 join_members: ["localhost:7200" for read, "localhost:7300" for write, "localhost:7301" for write]
schema_config: configs:
common: path_prefix: /tmp/loki replication_factor: 1 storage: s3: endpoint: xxx insecure: true bucketnames: xxx region: xxx access_key_id: xxx secret_access_key: xx s3forcepathstyle: true http_config: insecure_skip_verify: true ring: kvstore: store: memberlist
query_range: results_cache: cache: embedded_cache: enabled: true max_size_mb: 100
limits_config: enforce_metric_name: false reject_old_samples: true reject_old_samples_max_age: 336h ingestion_rate_mb: 400 ingestion_burst_size_mb: 200 per_stream_rate_limit: 50MB
ingester: max_chunk_age: 336h
chunk_store_config: max_look_back_period: 336h
table_manager: retention_deletes_enabled: true retention_period: 336h
query_scheduler: max_outstanding_requests_per_tenant: 2048
Happy to get help.