man-group / arctic

High performance datastore for time series and tick data
https://arctic.readthedocs.io/en/latest/
GNU Lesser General Public License v2.1
3.05k stars 583 forks source link

Improve store get_info with collection data for VersionStore #544

Open dimosped opened 6 years ago

dimosped commented 6 years ago

We need to improve the available meta-data we keep for the data usage of a given library.

Extend capabilities of get_info with :

The user may create standardized Arctic Prometheus metrics for the above, with option to start prometheus web server ready for scraping.

The goal is to help understanding space usage providing insights for data eligible for deletion.

This could also help with solutions of multi tier storage (e.g. hybrid mongo(where most recently accessed data go) and nfs/s3(slower easier to maintain, slower retrieval))

dimosped commented 6 years ago

Use Prometheus in your Python code: https://github.com/prometheus/client_python

Spin up your own local Prometheus server: https://prometheus.io/docs/prometheus/latest/getting_started/