search

LinkedInAttic / white-elephant

Hadoop log aggregator and dashboard
Other
191 stars 63 forks source link

Categorize jobs as Hive, Pig, JMR, etc. #1

Open jghoman opened 11 years ago

jghoman commented 11 years ago

It would be nice to be able to see what type of job is being listed. This would allow one to break cluster usage down by framework. One can identify Hive jobs via the hive.query param in the job conf. Pig has a similar property.

matthayes commented 11 years ago

I think this is a great idea :)

epicycle commented 11 years ago

We'd like to be able to collect arbitrary metadata from our jobs to be able to query by them, the type of job being one but possibly even more. For example set some arbitrary other identifier in a job configuration and then being able to query on that in the web ui. Obviously we'd have to tell the mapper / reducer processes what those keys were but it seems reasonable and if done right very flexible.

matthayes commented 11 years ago

Yea this sounds like it would be very useful indeed.

On Thu, Aug 29, 2013 at 10:59 AM, Sean notifications@github.com wrote:

We'd like to be able to collect arbitrary metadata from our jobs to be able to query by them, the type of job being one but possibly even more. For example set some arbitrary other identifier in a job configuration and then being able to query on that in the web ui. Obviously we'd have to tell the mapper / reducer processes what those keys were but it seems reasonable and if done right very flexible.

— Reply to this email directly or view it on GitHubhttps://github.com/linkedin/white-elephant/issues/1#issuecomment-23509488 .