search

tikv / client-go

Go client for TiKV
Apache License 2.0
279 stars 219 forks source link

Support show bytes sent/received on the connection #627

Open lance6716 opened 1 year ago

lance6716 commented 1 year ago

Due to some billing requirements, I want to know how many bytes this client sent/received with TiKV. Currrently I want to wrap a net.Conn with customized Read Write method to add the IO byte number to an atomic variable, like

https://github.com/pingcap/tiflow/blob/7f27730235f94859692b9da561f50c491e483427/dm/syncer/dbconn/utils.go#L102-L127

Is this OK? can you give some advice

lance6716 commented 1 year ago

@Defined2014 @xiongjiwei ptal

Defined2014 commented 1 year ago

Is this added to every TCP connection? Will it be too heavy to add atomic variables here which have a big impact on performance?

lance6716 commented 1 year ago

Is this added to every TCP connection? Will it be too heavy to add atomic variables here which have a big impact on performance?

It should be an option like WithDialer or something, but when it enables, all connections to the endpoint should use the atomic counter.

And I think the performance is not a big problem, because IO duration is much higher than atomic operations

sticnarf commented 1 year ago

It sounds okay but I'm not sure whether it is a better idea to count at upper layers for more flexibility. If one client could support multiple tenants, counting bytes by connection may be not enough.

Defined2014 commented 1 year ago

It sounds okay but I'm not sure whether it is a better idea to count at upper layers for more flexibility. If one client could support multiple tenants, counting bytes by connection may be not enough.

Agree. Do you have a spec about this requirement? I am not fully understand about this feature.

According to my understanding, a TiDB-Server may only have one tenant, do we need to count traffic by TCP connection, maybe by instance is enough (not sure)?.

lance6716 commented 1 year ago

As my plan, it's used by lightning to import data to tikv. We want to know the network IO of this import task

sticnarf commented 1 year ago

Is precision so important that we must wrap the connection? The protobuf bytes or the bytes you just want to import (oh it will) don't seem to differ a lot from the real IO bytes 🤔

disksing commented 1 year ago

Is precision so important that we must wrap the connection? The protobuf bytes ~or the bytes you just want to import~ (oh it will) don't seem to differ a lot from the real IO bytes 🤔

good point. I'm thinking of another example: if there are problems with environment and lead to network reconnection or data retransmission, we should not charge users for it.