Default configuration is incompatible with logrotate

[nibroc]

This cookbooks sets up log rotation for *.log and *.out files, but the way Tomcat writes out files with the default configuration, log rotation isn't quite right. In particular, Tomcat writes out the access log, the catalina log (not catalina.out), and the localhost log with date suffixies. Since log rotate expects to operate on a single file name, it sees these files as unrelated and thus doesn't ever purge old ones.

In other words, Tomcat is writing out log files like catalina.2017-10-07.log, catalina.2017-10-08.log, catalina.2017-10-09.log, etc. The log rotation configuration will have these correctly gzipped and whatnot, but they will never be purged. There will only ever be 1 rotated file of each name. We've noticed on some of our nodes that we still have files months old because of this, and we occasionally run into disk space issues.

I would expect Tomcat to be configured by this cookbook to use simply catalina.log, localhost_access_log.txt, etc instead of these date suffixed names.

[nibroc]

Would there be interest in a pull request to fix this? Or is the current behavior desired for some reason?

[bbaugher]

I'm not certain that this cookbook actually configures tomcat to do that by default. To make the access log change it would require configuring the server.xml access log valve. I believe to make changes to the catalina log file would require changing the logging.properties file. By default this cookbook won't change either of those unless specified

[nibroc]

The cookbook just uses the server.xml and logging.properties that are bundled with the Tomcat archive. For example, here's the relevant snippets from a typical Tomcat 7 archive.

server.xml:

        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log." suffix=".txt"
               pattern="%h %l %u %t &quot;%r&quot; %s %b" />

logging.properties:

2localhost.org.apache.juli.FileHandler.level = FINE
2localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
2localhost.org.apache.juli.FileHandler.prefix = localhost.

Though this doesn't look like it, the FileHandler configuration defaults are such that they're effectively the same as the AccessLogValve in terms of naming.

And then there's the logrotate configuration cerner_tomcat sets up:

"/var/log/tomcat/server-8080/*.out" "/var/log/tomcat/server-8080/*.log" {
  daily
  maxsize 100M
  rotate 3
  size 10M
  notifempty
  copytruncate
  compress
}

This looks right at first, but the problem is that the Tomcat files are using date suffixes. This confuses logrotate since it expects them to be the same file always (e.g. localhost.log instead of localhost.2017-10-01.log, localhost.2017-10-02.log, etc). So, instead of actually rotating, what ends up happening is you end up with localhost.2017-10-01.log.gz.1, localhost.2017-10-02.log.gz.1, localhost.2017-10-03.log.gz.1, etc and none of these actually get purged.

If cerner_tomcat is going to setup logrotate, it also needs to overwrite the default logger settings in logging.properties and server.xml.

For example:

server.xml

        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
               prefix="localhost_access_log" suffix=""
               rotatable="false" />

logging.properties

2localhost.org.apache.juli.FileHandler.level = FINE
2localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
2localhost.org.apache.juli.FileHandler.prefix = localhost
2localhost.org.apache.juli.FileHandler.suffix = .log
2localhost.org.apache.juli.FileHandler.rotatable = false

[nibroc]

After thinking about it more, I think it might make more sense for cerner_tomcat to not use logrotate but instead just setup a cron job. That way, it could just clean out the old files Tomcat leaves around instead of needing to overwrite server.xml and logging.properties.

That would save the hassle of having to support both Tomcat 7 and 8 quirks for logging.properties. It's also arguably a bit more correct since logrotate has to use the copytruncate strategy with Tomcat, and that's prone to a race condition that can lose logs.

If it helps by the way, feel free to contact me internally to Cerner, and I can show you some of our nodes where logrotate is running, but rotation is wrong.

[bbaugher]

Yes the logrotate settings are probably a copy from how we usually configure our server/access logs since they don't seem to make sense with the default

[nibroc]

After thinking on it more, I think the best option would probably be to use logrotate for catalina.out, and a cron job for the other log files. This would avoid the race condition that logrotate has for localhost.log, catalina.log, etc. The only problem is that Tomcat doesn't handle rotation itself for catalina.out, so logrotate would still be necessary for that (though in theory catalina.out should be empty -- but in practice it of course isn't).

Would there be any interest in a pull request that changes logrotate to only handle catalina.out and adds a cron job for the other files?

[bbaugher]

Yes I think that would be good