search

graphite-project / carbon

Carbon is one of the components of Graphite, and is responsible for receiving metrics over the network and writing them down to disk using a storage backend.
http://graphite.readthedocs.org/
Apache License 2.0
1.51k stars 490 forks source link

Metric upload unreliable without ACK in protocol #842

Closed alespour closed 4 years ago

alespour commented 5 years ago

I cannot seem to find a way to reliable upload a large bulk of metrics with limited cache and updates per second with flow control in use. With such configuration, carbon-cache is supposed to "sockets over which metrics are received will temporarily stop accepting data", right? But in one-way communication without ACKs, client has no way to find that nor to be sure at all that data were accepted. So it can appear it has successfully sent all the data (probably filling some network buffers on both sides), then close the connection and if the receiver is too slow to read all the data before client-side socket is definitely gone, the data is lost.

I'm testing version 1.1.4 and I can configure carbon-cache in a way that this will unlikely happen, but that does not seem enough robust solution, without proper flow control.

piotr1212 commented 5 years ago

There are ACKs in the TCP layer. I don't think the server's receive buffers get lost when the client disconnects, not sure but would really surprise me if so. And why would a client close the connection before it receives it's tcp ack? What would an application layer ack add if the client just closes it's connection before receiving the tcp ack?

I'm not against adding ACKs to the protocol. If you have suggestions how to do this let us know.

alespour commented 5 years ago

I know very little about TCP, so I am probably just wrong. I'm just looking for explanation why I am losing data with enabled flow control in carbon. What prevents the client from closing the connection before everything is tcp acked? Does close() block? Also, on the server side, I doubt that receive buffers are valid and socket can be read from when client end is gone for good. In case carbon is temporarily not accepting data, client unnecessarily keeps sending them until gets blocked because buffers are full. Another case for app-level ACK could be error responses eg. when clients sends malformed data.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.