search

iconara / rubydoop

Write Hadoop jobs in JRuby
220 stars 33 forks source link

Counters in a MapReduce job to track anything #14

Closed deivinsontejeda closed 10 years ago

deivinsontejeda commented 10 years ago

I tried to find a way to use counter into my jobs but did not see any way to add this one. So, In the current Rubydoop's version it's possible add this counter to track anything?

iconara commented 10 years ago

You can work with counters the same way you work with counters in the Java API. The context parameter given to to #map and #reduce is a standard Hadoop Mapper.Context or Reducer.Context, which have a method called getCounter that returns a counter object.

Almost everything in Rubydoop works like in the Java Hadoop APIs. We try to hide as little as possible, and use the Java objects coming from Hadoop as much as possible, so if it can be done in the Java API it is almost always possible in Rubydoop.

deivinsontejeda commented 10 years ago

Yep, but Counters you need define them and then you can get them with getCounter. So, Certainly with context one will get access to Counter (and others things), but how do you define a new counter?

iconara commented 10 years ago

Sorry, I don’t think I understand the question. Counters don’t need to be created AFAIK, I mean you can define labels for them and in Java you use enums, but you can create any counter with just a string as a name.

deivinsontejeda commented 10 years ago

Yeah, works fine!

context.get_counter("BadRecord", "INVALID_COLUMN").increment(1)
iconara commented 10 years ago

Great! Maybe I should add this as an example.

deivinsontejeda commented 10 years ago

I could help you with some example, basically the whole MapReduce that I have developed at this moment work with Rubydoop :)

iconara commented 10 years ago

Cool, examples would be very welcome. Put them in the examples dir and send a pull request. Just keep each example focused on one singe topic.

On 17 okt 2013, at 15:50, Deivinson Tejeda notifications@github.com wrote:

I could help you with some example, basically the whole MapReduce that I have developed at this moment work with Rubydoop :)

— Reply to this email directly or view it on GitHub.