search

opendatahub-io-contrib / data-mesh-pattern

Data Mesh Pattern
https://opendatahub-io-contrib.github.io/data-mesh-pattern
Apache License 2.0
24 stars 15 forks source link

Integrate data mesh over platform observability #57

Open caldeirav opened 1 year ago

caldeirav commented 1 year ago

Integrate with OpenTelemetry to export metrics, logs, and traces from the platform (as well as potentially Kepler) into data mesh ingestion. For this we focus on technical stacks to be used long term by our engineering team for metrics / logs / traces collection in the platform.

Metrics: Prometheus / Thanos Logs: Loki / Vector Traces: Jaeger / OpenTelemetry

The proposed approach would create a single layer of data delivery for metrics, logs and traces for the data collected and stored (potentially via ingestion through Trino / Iceberg).

caldeirav commented 1 year ago

Working on an initial POC for data mesh on application observability and also reservation system for an airline customer. This is to create an event database to be used as source for AI models for anomaly detection.

caldeirav commented 11 months ago

@jpaulrajredhat as discussed, let's start to introduce the observability components into the data mesh deployment through this issue please. The end state we are looking at is metrics, logs, traces stored for long term queries in Minio and ability to query them through Trino. This will be the starting point of AIOps pipelines in the future.

zagaos commented 11 months ago

@jpaulrajredhat as discussed, let's start to introduce the observability components into the data mesh deployment through this issue please. The end state we are looking at is metrics, logs, traces stored for long term queries in Minio and ability to query them through Trino. This will be the starting point of AIOps pipelines in the future.

sure , I'll do this week.

jpaulrajredhat commented 11 months ago

@jpaulrajredhat as discussed, let's start to introduce the observability components into the data mesh deployment through this issue please. The end state we are looking at is metrics, logs, traces stored for long term queries in Minio and ability to query them through Trino. This will be the starting point of AIOps pipelines in the future.

sure , I'll do this week.

MinIO , Airflow components are already in datamesh repo , only thing we need to add OpenTelemetry collector , Elastic search APM components and Kafka.

for single point of view , Elastic search APM is the right option or we need to build custom dashboard