thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
12.79k stars 2.04k forks source link

Receive component got "AlreadyExists desc = store" error. #6036

Open JayChanggithub opened 1 year ago

JayChanggithub commented 1 year ago

Hi Thanos developers.

I saw the issue https://github.com/thanos-io/thanos/issues/5407 regarding "AlreadyExists desc = store" error. I'd like to follow up whether this issue resolved. If so how to i updated to fix. Because i got the same issue today. The below is my configuration of receive component. Thanos image version is "v0.25.0".

    spec:
      containers:
      - args:
        - receive
        - --grpc-address=0.0.0.0:10901
        - --http-address=0.0.0.0:10902
        - --remote-write.address=0.0.0.0:19291
        - --receive.replication-factor=1
        - --objstore.config-file=/etc/thanos/objectstorage.yaml
        - --tsdb.path=/var/thanos/receive
        - --tsdb.retention=12h
        - --label=receive_replica="$(NAME)"
        - --label=receive="true"
        - --receive.hashrings-file=/etc/thanos/thanos-receive-hashrings.json
        - --receive.local-endpoint=$(NAME).thanos-receive.thanos.svc.cluster.local:10901
        env:
        - name: NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        image: thanosio/thanos:v0.25.0
k logs -f prometheus-prometheus-0  -n cic-system prometheus | grep 'remote' 

ts=2023-01-11T08:32:10.546Z caller=dedupe.go:112 component=remote level=error remote_name=7885dc url=http://thanos-receive.xxxxxxxxx/api/v1/receive msg="non-recoverable error" count=500 exemplarCount=0 err="server returned HTTP status 409 Conflict: forwarding request to endpoint thanos-receive-0.thanos-receive.thanos.svc.cluster.local:10901: rpc error: code = AlreadyExists desc = store locally for endpoint thanos-receive-0.thanos-receive.thanos.svc.cluster.local:10901: conflict"

ts=2023-01-11T09:21:37.374Z caller=dedupe.go:112 component=remote level=warn remote_name=7885dc url=http://thanos-receive.xxxxxxxxx/api/v1/receive msg="Failed to send batch, retrying" err="Post \"http://thanos-receive.xxxxxxxxxxxxxx/api/v1/receive\": http: server closed idle connection"
sharathfeb12 commented 1 year ago

Seeing this issue on v0.31.0 as well.

fpetkovski commented 1 year ago

This is likely not a problem, it just means that the sample already exists in the receiver.