oliver-tarrant-tessella
commented
5 years ago Notes have been added to the wiki here: https://github.com/ukaea/piezo/wiki/SparkDevNotes#log-files. Investigations have shown that logging for all spark applications is done through Log4j. Configuration is possible and can be set as part of the spark manifest as outlined in the wiki. A link to an example template file is included but we have not explicitly altered the default logging in our system as the default parameters have been appropriate for all tasks we have tried and without specific use cases we are unable to configure the logs to those that are needed.
Job log files may roll over when a set size or time is reached. This is not yet an issue, since we are running small example jobs, but could become one when the system is put into production. The thresholds for time/size may be configurable - if this is the case, then documenting what the defaults are and how they could be configured is the first step.