Log rotate

[nathansmonk]

I see the file to set the log rotate settings for nginx and php-fpm, but I don't see how these are applied. My instance does not appear to be rotating, could you shed any light on this please?

[jeff-kilbride]

@nathansmonk Take a look at the 99_custom_logging.sh file in:

.ebextensions/setup/hooks/appdeploy/pre

That's where the logrotate files are setup. Also, keep in mind they are setup to rotate based on size, not on time. So, your logs won't rotate until they reach 10MB in size. You can adjust those settings in the files in the:

.ebextensions/setup/support/conf

directory.

[nathansmonk]

Ah ok, thanks that is useful.

I have a further questions though: I see in set up that cron is used to rotate the nginx logs -

080_healthd_nginx_cron_conf:
  command: cp -f .ebextensions/setup/config/cron/healthd.nginx.conf /etc/cron.hourly/cron.logcleanup.elasticbeanstalk.healthd.nginx.conf
090_cron_executable:
  command: chmod +x /etc/cron.hourly/cron.logcleanup.elasticbeanstalk.healthd.nginx.conf

But the same is not done for php-fpm - is that correct? If so, was that intentional? How would you set it up to rotate php-fpm logs automatically?

[jeff-kilbride]

The healthd logs are only implemented for nginx. It's basically an easy to parse web response log that Elastic Beanstalk uses to see if the web server / app is healthy. The cron job just removes old log files every hour. Elastic Beanstalk has it's own system for rotating the regular nginx and php-fpm logs. That's what these lines are doing in that custom_logging file:

if [ -f /etc/logrotate.d/nginx ]
then
    /opt/elasticbeanstalk/bin/log-conf -n'nginx' -l'/var/log/nginx/*' -f /opt/elasticbeanstalk/support/conf/nginx.logrotate.conf
    rm -f /etc/logrotate.d/nginx
fi

if [ -f /etc/logrotate.d/php-fpm-7.1 ]
then
    /opt/elasticbeanstalk/bin/log-conf -n'php-fpm' -l'/var/log/php-fpm/7.1/*log' -f /opt/elasticbeanstalk/support/conf/php-fpm.logrotate.conf
    rm -f /etc/logrotate.d/php-fpm-7.1
fi

They remove the cron-based log rotation and use the log-conf binary from EB.

This stuff was all cribbed from a Node.js Elastic Beanstalk server. Node uses nginx as a proxy by default. So, most of the nginx related config was copied verbatim.

[nathansmonk]

So are you saying therefore, that EB does the log rotation for php-fpm and should rotate by default?

[jeff-kilbride]

Yes. Here is my log directory for php:

# cd /var/log/php-fpm/7.1/
total 404K
drwxrwx--- 2 webapp root   4.0K Jun 17 03:26 ./
drwxr-xr-x 3 webapp root   4.0K Mar 23 20:32 ../
-rw------- 1 root   root    11K Jun 19 14:49 error.log
-rw------- 1 root   root    26K May 27 02:08 error.log-20180527
-rw------- 1 root   root    26K Jun  3 02:29 error.log-20180603
-rw------- 1 root   root    26K Jun 10 03:21 error.log-20180610
-rw------- 1 root   root    26K Jun 16 21:01 error.log-20180617
-rw-r--r-- 1 webapp webapp  42K Jun 19 18:48 www-error.log
-rw-r--r-- 1 webapp webapp 203K Jun 17 03:12 www-error.log-20180617

and this is nginx:

[2018-06-19 18:55:36]-[root@ip-172-31-17-220]-[/var/log/nginx]
# ll
total 5.6M
drwx------ 3 nginx nginx 4.0K Jun 19 03:22 ./
drwxr-xr-x 8 root  root  4.0K Jun 17 03:26 ../
-rw-r--r-- 1 nginx nginx 2.4M Jun 19 18:55 access.log
-rw-r--r-- 1 nginx nginx 305K Jun 10 03:51 access.log-20180610.gz
-rw-r--r-- 1 nginx nginx 289K Jun 11 03:06 access.log-20180611.gz
-rw-r--r-- 1 nginx nginx 316K Jun 12 03:21 access.log-20180612.gz
-rw-r--r-- 1 nginx nginx 325K Jun 13 03:08 access.log-20180613.gz
-rw-r--r-- 1 nginx nginx 353K Jun 14 03:47 access.log-20180614.gz
-rw-r--r-- 1 nginx nginx 337K Jun 15 04:44 access.log-20180615.gz
-rw-r--r-- 1 nginx nginx 295K Jun 16 03:19 access.log-20180616.gz
-rw-r--r-- 1 nginx nginx 307K Jun 17 03:26 access.log-20180617.gz
-rw-r--r-- 1 nginx nginx 337K Jun 18 03:31 access.log-20180618.gz
-rw-r--r-- 1 nginx nginx 345K Jun 19 03:22 access.log-20180619.gz
-rw-r--r-- 1 nginx nginx    0 Jun 18 03:31 error.log
-rw-r--r-- 1 nginx nginx 5.0K Jun  7 03:27 error.log-20180607.gz
-rw-r--r-- 1 nginx nginx 4.2K Jun  8 03:40 error.log-20180608.gz
-rw-r--r-- 1 nginx nginx 5.4K Jun  9 03:35 error.log-20180609.gz
-rw-r--r-- 1 nginx nginx  471 Jun 10 03:51 error.log-20180610.gz
-rw-r--r-- 1 nginx nginx  359 Jun 11 03:06 error.log-20180611.gz
-rw-r--r-- 1 nginx nginx  437 Jun 13 03:08 error.log-20180613.gz
-rw-r--r-- 1 nginx nginx  263 Jun 15 04:44 error.log-20180615.gz
-rw-r--r-- 1 nginx nginx  238 Jun 16 03:19 error.log-20180616.gz
-rw-r--r-- 1 nginx nginx  187 Jun 17 03:26 error.log-20180617.gz
-rw-r--r-- 1 nginx nginx  303 Jun 18 03:31 error.log-20180618.gz
drwxr-xr-x 2 nginx nginx 4.0K Jun 19 18:01 healthd/

[2018-06-19 18:55:39]-[root@ip-172-31-17-220]-[/var/log/nginx]
# ll healthd/
total 48K
drwxr-xr-x 2 nginx nginx 4.0K Jun 19 18:01 ./
drwx------ 3 nginx nginx 4.0K Jun 19 03:22 ../
-rw-r--r-- 1 nginx nginx  36K Jun 19 18:55 application.log.2018-06-19-18

Notice, the php logs are only being rotated once a week -- probably because they aren't very big. However, Elastic Beanstalk is doing some other stuff here, because it doesn't exactly follow the logrotate configuration. For example, the php logs are not being compressed and they are rotating weekly, even though they are smaller than 10MB.

If you really want to dive into this, you're going to need to look at the /opt/elasticbeanstalk/bin/log-conf file and the ruby file it is calling. I don't pretend to know the intricacies of log rotation in that kind of detail. I merely setup the config files the same way they are setup on other EB platforms.

[jeff-kilbride]

FYI, there was a bug preventing the Elastic Beanstalk logrotate config from being setup correctly on initial app deployments -- it would work correctly for any deployment after the first. It has been corrected in my latest PR #13.