search

LinkedInAttic / white-elephant

Hadoop log aggregator and dashboard
Other
191 stars 63 forks source link

Make it work with Hadoop2 #15

Open ykk1431 opened 10 years ago

ykk1431 commented 10 years ago

Hi ,

Any plan to make it work with Hadoop2. I am using cdh4.2 with MR1 and white-elephant is not working for me and i think the issue is during parsing logs. I wish i am a developer to fix this myself.

-Thanks

matthayes commented 10 years ago

Yea we will add support for Hadoop2 at some point. What is your issue with MR1 though? Can you include any error messages you're seeing?

bilsch commented 10 years ago

Just curious, saying it does not work on hadoop2 - does that mean no support for yarn mode but running under mr1 mode is supported/ok? ( since its the same api, I would have thought this to be the case )

matthayes commented 10 years ago

The log parsing jobs are written using MR1, which I presume is still available. So you're right that if the same api is available it should be possible to run the jobs. I suspect there may be a problem parsing the logs though. I think in Hadoop 2 with yarn the log format may have changed. They may not even be stored the same way. I haven't looked into this but this is what I heard.

On Mon, Nov 11, 2013 at 6:07 PM, Bill Schwanitz notifications@github.comwrote:

Just curious, saying it does not work on hadoop2 - does that mean no support for yarn mode but running under mr1 mode is supported/ok? ( since its the same api, I would have thought this to be the case )

— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/white-elephant/issues/15#issuecomment-28262077 .