search

libp2p / js-libp2p

The JavaScript Implementation of libp2p networking stack.
https://libp2p.io
Other
2.31k stars 442 forks source link

Implement statistics to support Connection Management #157

Closed hacdias closed 6 years ago

hacdias commented 6 years ago

In order to implement stats/bw on js-ipfs, we need to have some way to measure the bandwidth that comes in and out of js-ipfs. It should also support peer and protocol basis metrics.

Some refs:

/cc @pgte @diasdavid

pgte commented 6 years ago

It turns out I also need global stats to implement this connection manager API: https://github.com/libp2p/js-libp2p-connection-manager/tree/master#api

pgte commented 6 years ago

@diasdavid what would be, in your opinion, the best place in the libp2p stack to monitor total traffic (if that's even possible)?

daviddias commented 6 years ago

It can be a module that implements that functionality and gets plugged into js-libp2p.

pgte commented 6 years ago

@diasdavid and how would it hook itself into watching the traffic?

pgte commented 6 years ago

@diasdavid ping :)

daviddias commented 6 years ago

@pgte wanna make a suggestion for that one?

pgte commented 6 years ago

OK, took me a bit to figure out how switch is handling all the outgoing and incoming connections, but here goes:

Some requirements: we need to observe unencrypted traffic in protocol-specific streams so we can compute total (unencrypted) traffic by peer, protocol, direction (upload / download) and the combinations of these ({ peer, protocol }, {peer, direction}, {peer, protocol, direction}).

In my guess, we need the following hooks on js-libp2p-switch:

On self-initiated connections:

On incoming connections:

@diasdavid thoughts?

daviddias commented 6 years ago

we need to observe unencrypted traffic in protocol-specific streams so we can compute total (unencrypted)

Unencrypted? I guess you mean that you need to observe traffic as it reaches the socket before going to SECIO (which adds a lot of overhead). The traffic won't be in plaintext once the SECIO handshake happens though.

One other option is piggybacking on the fact that we already do Connection wraps to propagate things like Observed Addrs, we can also keep a reference to the socket so that we can measure that directly.

pgte commented 6 years ago

@diasdavid what I mean is the exact opposite: inside the encrypted stream is an unencrypted stream. I think we should not measure the encryption overhead. Also, not sure what go-ipfs does, we should miimc that..

@diasdavid about pigging back on Connection, what would be the insertion points?

daviddias commented 6 years ago

what I mean is the exact opposite: inside the encrypted stream is an unencrypted stream.

I see, you meant to measure the outbound throughput, before encryption and after encryption.

pigging back on Connection, what would be the insertion points?

A Connection can be both the wrap around the socket or a wrap around a muxed stream. We should have a way to visualize how much traffic the stream is contribution to the overall traffic over the socket. libp2p-switch has access to all of these, it can do that work of attaching this readers to the Connectiob object.

pgte commented 6 years ago

@diasdavid yup, and Ithink I want inbound traffic too, also discounting the encryption overhead.

I see the insertion points on dialled connections here, but I don't see that when accepting a new connection. @diasdavid Am I missing something?

daviddias commented 6 years ago

Hope this answers your question:

Incoming Connection via socket (direct from transport):

Incoming Connection via stream (from the muxer):

pgte commented 6 years ago

@diasdavid Ha, multistream-select already returns a Connection. That's what I was missing. Thanks!

daviddias commented 6 years ago

I see this is all done now -- https://github.com/libp2p/js-libp2p#switch-stats-api --, great work @pgte ❤️

daviddias commented 4 years ago

Lot's of relevant Metrics conversations from go-libp2p land