How to store log files for harvest jobs on production

[adborden]

Instances in Elastic Beanstalk come and go. Any files stored on the instance will only survive until Beanstalk decides to replace the instance. Because of this, we need to store log files elsewhere.

Here are some options to look into:

• Email the job report as soon as its done
• Store the job report in the database
• Store the job report as a file in Django Storage object, which needs to be moved to S3 or similar

[adborden]

For the Django Storage option, Django allows custom storage engines. django-storages provides several backends including S3.

[adborden]

I split out the work for S3/Django Storage to another issue since that needs to happen regardless of this one. I think in the long term we want to store the job reports on s3 or in the database, but for the short term, emailing the report should be fine.

[adborden]

Another option would be to just log to the django logger, and then figure out how to pull the logs from Beanstalk which is something we should do anyway for debugging.