search

Shopify / camus

Kafka->HDFS pipeline from LInkedIn. It is a mapreduce job that does distributed data loads out of Kafka.
7 stars 4 forks source link

Timestamp parse failure #98

Closed olessia closed 6 years ago

olessia commented 6 years ago

Ticket: https://github.com/Shopify/camus-project/issues/13

Pass in an extra flag to the record wrapper that indicates whether the timestamp was parsed correctly. Count how many times this happens and publish the metric to statsd.

drdee commented 6 years ago

Won't this lead to excessive logging?

On Oct 26, 2017, at 09:32, Diogo Terror notifications@github.com wrote:

@dterror commented on this pull request.

In camus-kafka-coders/src/main/java/com/linkedin/camus/etl/kafka/coders/JsonStringMessageDecoder.java:

@@ -120,15 +120,18 @@ else if (timestampFormat.equals("ISO-8601")) { } } }

  • else {
  • log.warn("Message does not have the field " + timestampField); // TODO gotcha

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

dterror-zz commented 6 years ago

not much worse than we have now and it's for us to better understand it so we can remove the excessive logging we currently have.

drdee commented 6 years ago

👍