Create checklist for production setups

[holtgrewe]

Is your feature request related to a problem? Please describe. Documenting best practices for production deployments would help admins to create varfish installations that are suitable for production.

Describe the solution you'd like Checklist should be created and part of the deployment.

Describe alternatives you've considered N/A

Additional context N/A

[holtgrewe]

• apply postgresql tunings from postgres config of docker-compose (according to manual), otherwise the low number of locks will prevent the high partition table count from working
• postgres database should be placed on SSD/NVME storage
• sufficient memory for database but also upload storage (minio S3)
• monitor free storage for postgres database and upload; file systems commonly start to have degrading performance at 80% fill ratio and even worse about 90%
• monitor out-of-memory kills (e.g., celery jobs)
• run database backups, excluding background database and mass variant data (TODO: add list / example shell commands for this)
• run S3 raw file backups

[holtgrewe]

• setup sentry to get traceback of crashes

[holtgrewe]

• logging with systemd (notes)
  • https://docs.docker.com/config/containers/logging/journald/
  • https://gist.github.com/mosquito/b23e1c1e5723a7fd9e6568e5cf91180f