Rotate logs to S3

[iandees]

1. Rotate logs to S3
   • Use an S3 workflow to delete logs after 90(?) days
   • Point embed.censusreporter.org S3 website logs to this same S3 bucket
2. Use papertrail for live logs from the app server

[iandees]

I signed up for papertrail.

I added the syslog host/port that they gave to the EC2 box's rsyslog config:

echo "*.*          @logs4.papertrailapp.com:1234" > /etc/rsyslog.d/60-papertrail.conf
sudo service rsyslog restart

At this point I started to get messages in papertrail.

I added the following to /etc/rsyslog.d/50-censusreporter-nginx.conf to poll the access logs and dump them to syslog (and thus to papertrail). From https://petermolnar.eu/log-nginx-to-rsyslog/.

$ModLoad imfile

$InputFileName /var/log/nginx/api.censusreporter.org.error.log
$InputFileTag nginx_error_api:
$InputFileStateFile stat-nginx-error-api
$InputFileSeverity error
$InputFileFacility local6
$InputFilePollInterval 1
$InputRunFileMonitor

$InputFileName /var/log/nginx/api.censusreporter.org.access.log
$InputFileTag nginx_access_api:
$InputFileStateFile stat-nginx-access-api
$InputFileSeverity notice
$InputFileFacility local6
$InputFilePollInterval 1
$InputRunFileMonitor

$InputFileName /var/log/nginx/censusreporter.org.error.log
$InputFileTag nginx_error_django
$InputFileStateFile stat-nginx-error-django
$InputFileSeverity error
$InputFileFacility local6
$InputFilePollInterval 1
$InputRunFileMonitor

$InputFileName /var/log/nginx/censusreporter.org.access.log
$InputFileTag nginx_access_django
$InputFileStateFile stat-nginx-access-django
$InputFileSeverity notice
$InputFileFacility local6
$InputFilePollInterval 1
$InputRunFileMonitor

[iandees]

I also set up a filter in Papertrail to ignore healthcheck requests. There were a bunch of requests that came in from New Relic that were cluttering things up.

[iandees]

Next up here is to:

• [x] Log gunicorn logs to syslog (http://docs.gunicorn.org/en/stable/settings.html#syslog)
• [ ] Give access to Joe and Ryan
• [ ] Have Papertrail archive to S3
• [ ] Check to see how much data is used in Papertrail (how much would we be paying?)
• [ ] Change logrotate settings so that we only keep logs on the box for a day (or a week?). Especially /var/log/syslog, /var/log/nginx/*