Make DSpace logs rotate using org.apache.log4j.DailyRollingFileAppender

[pmather]

Logs produced by VTechWorks DSpace rotate log files via the application using the log4j framework. The logging is defined via log4j properties files. Currently, different log files are rotated using a daily file rotation logger. The logs for "dspace.log", "checker.log", and "cocoon.log" are rotated using org.dspace.app.util.DailyFileAppender (defined in /dspace/config/log4j.properties). Solr logs are rotated using org.apache.log4j.DailyRollingFileAppender (defined in /dspace/config/log4j-solr.properties). Finally, "handle-plugin.log" is using org.apache.log4j.DailyRollingFileAppender (defined in /dspace/config/log4j-handle-plugin.properties)

Although both logging methods superficially achieve the same ends (rotating a main log file on a daily basis) they differ subtly in precise details. With org.apache.log4j.DailyRollingFileAppender, when the base file is rotated it is renamed with a suffix indicating the date its contents represent. The base file is then truncated to zero size and continues to be the current active log file. With org.dspace.app.util.DailyFileAppender, the base file never exists. Logging is always done to a base file name with a date suffix. When a daily rotation period rolls around, a new log file is created with the new daily date suffix.

The main practical difference between the two schemes is that with org.apache.log4j.DailyRollingFileAppender the base log file (e.g., "solr.log") is always "today's logs" whereas with org.dspace.app.util.DailyFileAppender the fixed concept of "today's logs" does not exist. (You need to know today's date and then use that to find the appropriate file.)

The VT Minimum Security Standards (https://it.vt.edu/content/dam/it_vt_edu/policies/Minimum-Security-Standards.pdf) mandate the following for medium and high risk servers: "Forward logs to a remote log server. Use of the university's centralized log server is recommended." We currently do this for our servers and use filebeat to ship the logs. With org.apache.log4j.DailyRollingFileAppender, we can get filebeat to monitor and ship logs from a single file. When searching log entries in Kibana, only a single log file name need be searched over time. With org.dspace.app.util.DailyFileAppender, we have to have filebeat monitor and ship logs from all files matching the base file + date suffix pattern. When searching these in Kibana, many different files must be searched. It's easier to construct dashboards and queries in the org.apache.log4j.DailyRollingFileAppender case.

Given all the above, it would be preferable if the log files currently rotated using org.dspace.app.util.DailyFileAppender could be switched to rotate via org.apache.log4j.DailyRollingFileAppender (or some other rotation than maintains a fixed base log file).

[alawvt]

@pmather, it seems like we have the dspace and solr log rotation working by other means. Shall I close this issue?

[pmather]

This issue concerns logs, not stats, so the recent sharding of stats in the Solr cores would not be a reason to close this issue in my opinion.

It may be that DSpace log rotation has changed in DSpace 7, so this issue may be superseded by events. However, it would be nice if @keithgee could take a look as to whether the current logging class could be substituted for the one that Solr is using.

A perhaps more important issue that arose after I filed this issue is the log4j vulnerability. Although we did confirm our current DSpace is not vulnerable, we need to address the reason why it is not vulnerable...

[alawvt]

Thanks, @pmather, our agenda for the DevOps meeting next week is pretty full but we'll keep this on the backlog.