search

damballa / parkour

Hadoop MapReduce in idiomatic Clojure.
Apache License 2.0
257 stars 19 forks source link

Getting log output from mappers when using conf/local-mr! #13

Closed mjwillson closed 9 years ago

mjwillson commented 9 years ago

Apologies if this is a general hadoop question rather than something parkour-specific, but I'm finding it pretty much impossible to get any log output (stdout / stderr etc) from my mappers/reducers when running parkour jobs from the REPL in local mode.

I've tried manually setting various config properties, e.g.

  (conf/assoc!
   (conf/local-mr!)
   "yarn.log.dir" "/my/log/dir"
   "yarn.nodemanager.log-dirs" "/my/log/dir")

But no joy. No logs anywhere under /usr/local/hadoop (where I have the distribution installed) either.

Is there something obvious I could be missing here? Is there anything about the way that parkour starts these jobs which would prevent them from logging / any special prodding which might be needed?

mjwillson commented 9 years ago

OK so it turns out that in local mode the mapper / reducer log output goes directly to the process's STDOUT / STDERR, just that these weren't showing up in my remote nREPL.

Sorry for the interruption! :)

llasram commented 9 years ago

No worries -- glad there was an easy solution!