search

filecoin-project / boost

Boost is a tool for Filecoin storage providers to manage data storage and retrievals on Filecoin.
Other
111 stars 67 forks source link

Add opt in metrics #365

Open jacobheun opened 2 years ago

jacobheun commented 2 years ago

It would be helpful for debugging and improvements if we were able to collect metrics from Boost, such as deal success rates, errors, system information, etc.

This should be opt in only, and we could add this to the settings page with clear information about what we're collecting and where it's being collected.

In order to increase visibility into Boost and understand how Boost is used and what its performance is on SPs systems, we'd like to add functionality to Boost so that SPs can opt-in and periodically send statistics and metrics to a centralized system maintained by the development team.

We would then be able to track a measure a lot of data that is not available on-chain, such as:

Given that Boost is exposed on the internet, the opt-in metrics could be implemented as either push- or pull- based.

Opt-in push metrics

As an example of push- based system we could implement something similar to go-ethereum: https://github.com/ethereum/go-ethereum/tree/master/metrics

Each SP would then periodically emit metrics to our centralized time-series db backend.

Opt-in pull metrics

As an example of pull- based system we could implement a metrics package similar to Lotus: https://github.com/filecoin-project/lotus/blob/master/metrics/metrics.go

Additionally to the metrics package, we'd need a way to announce and discover every endpoint for an SP who has enabled opt-in metrics, so that we know to pull them periodically.

Issue Tracking

Refs

dirkmc commented 2 years ago

@jacobheun to follow up with data systems team about the metrics tooling they have implemented and if we can reuse it