twitter / summingbird

Streaming MapReduce with Scalding and Storm
https://twitter.com/summingbird
Apache License 2.0
2.14k stars 267 forks source link

Make it easy to add metrics to SummerConstructor #711

Open pankajroark opened 7 years ago

pankajroark commented 7 years ago

There are couple things that make adding metrics for replacement SummerBuilder very difficult.

  1. The metrics for regular summer have the node name in them, e.g: summingbird.Tail/tuplesIn. The overridden summer has to know the nodename to use the same metric names.
  2. An implicit job id is required for creating summingbird counters. Most jobs don't set job id by default and are forced to create one for this case.

Instead Counter should be passed to SummerBuilder. SummerBuilder is seldom used so one possibility is to accept breaking backward compatibility in this case. Otherwise a new option can be defined and this one deprecated.