search

Shopify / camus

Kafka->HDFS pipeline from LInkedIn. It is a mapreduce job that does distributed data loads out of Kafka.
7 stars 4 forks source link

Do not replace dots with underscores in topic names when constructing filename #17

Closed drdee closed 10 years ago

drdee commented 10 years ago

Some of our topic names contain dots. When generating the workingFilename, Camus replaces all dots in a topic with underscores. This complicates our migration away from Albert to Camus. Hence let's stop doing this.

In general, there has been a discussion in the Kafka community whether dots, underscores and dashes should be banned from topic names as it complicates writing JMX / Mbeans based monitoring tools for Kafka. The current consensus seems to be that dots, underscores and dashes are okay.

See: http://search-hadoop.com/m/4TaT4SHanu1&subj=Re+MBeans+dashes+underscores+and+KAFKA+1481 http://search-hadoop.com/m/4TaT4Jf1Jz1&subj=Re+Need+Document+and+Explanation+Of+New+Metrics+Name+in+New+Java+Producer+on+Kafka+Trunk https://issues.apache.org/jira/browse/KAFKA-1481

@yagnik @airhorns @wvanbergen

wvanbergen commented 10 years ago

:shipit: long live our fork.

airhorns commented 10 years ago

:shipit:

yagnik commented 10 years ago

:ship: I'm sad that we have to make our own fork for this.