search

sparsecode / DaFlow

Apache-Spark based Data Flow(ETL) Framework which supports multiple read, write destinations of different types and also support multiple categories of transformation rules.
Other
26 stars 13 forks source link

Configure etl feed metrics stats publishing #4

Closed abhioncbr closed 6 years ago

abhioncbr commented 6 years ago

Currently etl_framework support two frameworks for publishing stats for the ETL feed.

  1. Entries in hive table for each feed run.
  2. Publishing stats on Prometheus.

Right now, it is tightly coupled and code needs a separation from the feed code so that based on job_static_param stats will be published.