Closed Rohlik closed 4 months ago
Do you have stats about queue sizes on the other targets?
Yes, I have. This is a graph from that day (22.6.2021) and it is visible that during that time (~13:30) other carbon-c-relays have queue size bellow limit (25k).
Can you zoom in, e.g. from 13:20 to 13:40 to show the queue sizes? Not that this is in any way more of a fact, but what you're looking at are samples, and any peak pressure would cause this. It feels though as if the other queues should have handled the excess.
Here we go:
It seems the relay cannot push out metrics fast enough. It wouldn't surprise me if TLS adds some of it here,
I wonder if the relay is disconnecting here, which would make for constantly connecting again. Is your workload batch-driven, or should there be a constant load of influx metric?
I don't have access to that environment anymore 😐, but I can say that the workload is constant for sure.
ok, thanks, I'll close this ticket then because we cannot reproduce nor confirm any more
Hello @grobian, We have tens of servers with the same carbon-c-relay config and still some of them have problem and dropping metrics:
Logs during the carbon-c-relay restart:
Client config:
On client side and also on remote proxy servers we have carbon-c-relay.
I am not fully understand why client carbon-c-relay don't try send metrics to different destination instead of dropping metrics. Can you explain it to me?
PS: Interesting fact is that client node
XY
have problem just with remote serverproxy10
(the first log snippet) and clent nodeXZ
have problem only withproxy08
.