search

cleanzr / dblink

Distributed Bayesian Entity Resolution in Apache Spark
Other
57 stars 9 forks source link

Controlling log #7

Closed YathishK closed 5 years ago

YathishK commented 5 years ago

log4j.properties controls the log generated in Spark driver and worker nodes. can we change
log4j.rootCategory=INFO, console, file to log4j.rootCategory=WARN, console in log4j.properties file

resteorts commented 5 years ago

Why are you wishing to change this?

YathishK commented 5 years ago

The application with sample size of 10K run creating huge log on driver and worker nodes. Often times the browser is not responding, so by limiting the log messages to warning and errors, it would be easy to navigate.

resteorts commented 5 years ago

@YathishK Could you clarify what you mean by browser? Are you not running things from the command line? Perhaps I'm not fully understanding your meaning. I would not recommend opening log files in a browser. But again, I may not understand your question.

ngmarchant commented 5 years ago

log4j.properties controls the log generated in Spark driver and worker nodes. can we change log4j.rootCategory=INFO, console, file to log4j.rootCategory=WARN, console in log4j.properties file

Yes, you can set up the logging however you like. I think we were using that configuration for debugging purposes.

YathishK commented 5 years ago

image

stderr will take us to the log file.

Thank you