Open Amfiass opened 1 year ago
andrewkwok2022
commented
1 year ago hello @Amfiass, I have the similar issue (https://community.grafana.com/t/post-loki-api-v1-push-500-response-empty-ring-and-context-canceled/75544) as you and still no idea how to fix it. Kindly tell me if you find out the solution.
Schmoho
commented
1 year ago I am having the same issue. Any hint or help would be appreciated.
Amfiass
commented
1 year ago @Schmoho @andrewkwok2022 The problem is probably in the key-value store. A possible solution is to change storage to consul or etcd. We solved the problem by deploying the Loki cluster in the kubernetes, since the definition of the members of the ring occurs through the kubernets API.
Schmoho
commented
1 year ago I'm using inmemory kv and any kinds of cloud storage are not an option here
javier-torres
commented
1 year ago Hello @Schmoho @andrewkwok2022
In the official documentation said this:
In this mode the component microservices of Loki are bundled into two targets: -target=read and -target=write. The BoltDB compactor service will run as part of the read target.
You only need deploy section compactor y read node. The config file must be different for -target=read and -target=write.
I had the same problem an change this its works fine.
This is mi basic config for two options:
### -target=read
# Custom Config
auth_enabled: false
server:
http_listen_port: 3100
memberlist:
join_members:
- grafana-loki-read.poc.internal:7946
- grafana-loki-write.poc.internal:7946
compactor:
working_directory: /loki/compactor
shared_store: s3
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: memberlist
storage_config:
boltdb_shipper:
active_index_directory: /loki/boltdb-shipper-active
cache_location: /loki/boltdb-shipper-cache
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
shared_store: s3
aws:
s3: s3://eu-west-1
bucketnames: bucket-name
schema_config:
configs:
- from: 2020-07-01
store: boltdb-shipper
object_store: aws
schema: v11
index:
prefix: index_
period: 24h
ruler:
alertmanager_url: http://localhost:9093### -target=write
# Custom Config
auth_enabled: false
server:
http_listen_port: 3100
memberlist:
join_members:
- grafana-loki-read.poc.internal:7946
- grafana-loki-write.poc.internal:7946
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: memberlist
storage_config:
boltdb_shipper:
active_index_directory: /loki/boltdb-shipper-active
cache_location: /loki/boltdb-shipper-cache
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
shared_store: s3
aws:
s3: s3://eu-west-1
bucketnames: bucket-name
schema_config:
configs:
- from: 2020-07-01
store: boltdb-shipper
object_store: aws
schema: v11
index:
prefix: index_
period: 24h
ruler:
alertmanager_url: http://localhost:9093
sazima
commented
1 year ago same issue
remram44
commented
1 year ago Also have this issue. I am running Loki as a single container with common: ring: kvstore: store: inmemory
remram44
commented
1 year ago Is there documentation for how this "ring" thing is configured? What is the difference between memberlist and inmemory?
A search for inmemory through the entire documentation has like 2 hits. Judging by this issue, this seems like a critical component which could probably be described somewhere?
HuyDo-95g
commented
10 months ago I also have a same issue.
sojjan1337
commented
9 months ago running loki-distributed. promtail says: caller=client.go:419 component=client host=[loki.test.com](http://loki.test.com/) msg="error send ing batch, will retry" status=500 tenant= error="server returned HTTP status 500 Internal Server Error (500): empty ring"
This error msg comes up as soon as loki is down, why can't promtail reconnect correctly when loki comes up again? If i reinstall promtail again after loki is up, then i have no issues. Or it's isnt a problem, because i can retreive logs as usual.
Vinaum8
commented
4 months ago same error here
LucaIcaro
commented
3 days ago Same error with monolith and this conf:
# authentication is handled by reverse proxy
auth_enabled: false
server:
http_listen_port: 3101 # using non default port and leaving the default to nginx
grpc_listen_port: 9096
log_level: warn
grpc_server_max_concurrent_streams: 1000
common:
instance_addr: 127.0.0.1
path_prefix: /loki/data
storage:
filesystem:
chunks_directory: /loki/data/chunks
rules_directory: /loki/data/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100
schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: s3
schema: v13
index:
prefix: index_
period: 24h
storage_config:
tsdb_shipper:
active_index_directory: /loki/data/index
cache_location: /loki/data/index_cache
aws:
s3: s3://my-s3-bucket
s3forcepathstyle: true
region: eu-west-1
pattern_ingester:
enabled: true
metric_aggregation:
enabled: true
loki_address: localhost:3101
ruler:
alertmanager_url: http://localhost:9093
frontend:
encoding: protobufUsing this loki:
loki, version 3.2.0 (branch: k218, revision: 659f5421)
build user: root@003ce357cdf4
build date: 2024-09-18T16:21:52Z
go version: go1.22.6
platform: linux/amd64
tags: netgoEDIT: solved by changing the configuration of the s3 to this:
schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: **aws**
schema: v13
index:
prefix: index_
period: 24h
...
storage_config:
tsdb_shipper:
active_index_directory: /loki/data/index
cache_location: /loki/data/index_cache
aws:
s3: s3://**eu-west-1**
bucketnames: mybucketI followed the examples to make the first configuration (https://grafana.com/docs/loki/latest/configure/examples/configuration-examples/#2-s3-cluster-exampleyaml) that ended with a mistake.
Hello!
Faced a problem when installing a loki cluster. The cluster consists of 5 nodes: 1) Nginx proxy 2) loki reader x2 3) loki writer x2
All have the same configuration file installed with a difference in the target item:
All healthchecks pass, the service starts. According to the logs, Loki was able to locate all the members of the ring. But when trying to connect the cluster to grafana, a 500 error:
Loki: Internal Server Error. 500. empty ringIn the service logs I see the following:
At the same time, the installation of loki on one server works correctly, logs are written and read.
Tell me, please, where to dig, I can not find any problems.
Also, I tried to connect the distributor to the loki, the logging does not happen either. A 404 error is thrown without any explanation:
At the same time, the s3 storage is available, another lock is written to it in parallel, which is located on a separate server.