This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.
The epoch time (milliseconds) would be useful in case you want to know exactly the time your metrics are captured in spark job rather than rely on the time message from Kafka coming.
Model: Spark measure -> Kafka -> Telegraf -> InfluxDB. Telegraf parses your epoch time and uses it as the time writing to InfluxDB.
LucaCanali
commented
2 years ago
The epoch time (milliseconds) would be useful in case you want to know exactly the time your metrics are captured in spark job rather than rely on the time message from Kafka coming.
Model: Spark measure -> Kafka -> Telegraf -> InfluxDB. Telegraf parses your epoch time and uses it as the time writing to InfluxDB.