tensorlakeai / indexify

A realtime and indexing and structured extraction engine for Unstructured Data to build Generative AI Applications
https://getindexify.ai
Apache License 2.0
665 stars 70 forks source link

Introduce Metrics-tracking / Telemetry #121

Open yenicelik opened 9 months ago

yenicelik commented 9 months ago

We have logging to stdout through tracing which uses anysc tokio under the hood. We want something similar that we can hook up to Prometheus or Grafana to show analyzed data. For this we can use https://github.com/open-telemetry/opentelemetry-rust

Amongst others, following items should be tracked (please edit this issue as more items come to mind):

common system-level information:

yenicelik commented 9 months ago

Addressed partially by https://github.com/diptanu/indexify/pull/123 metrics like disk usage, CPU usage, context switches are missing