Updates and fixes for the existing Docker-based hosting capability

[sfpurdy]

Changes

Admin Credentials

Added the ability to set admin credentials using environment variables so they don't have to be set once everything is running. This allows for unattended deployments such as in CI/CD pipelines.

• bin/celery-entry.sh
  • Added to set the admin username and password using environment variables instead of having to do it manually after the containers start up. Depends on the updated version of the flask install command.
• enferno/commands.py
  • Added username and password options to the install command
  • It will still prompt for the username and password if they aren't provided

PostgreSQL Connection Dropping

There was an error that caused the PostgreSQL connection to drop when the cached connection goes stale. The fix to the error required the pool_pre_ping=True configuration to be passed to SQL Alchemy per the SQLAlchemy Documentation. This fix uses the SQLALCHEMY_ENGINE_OPTIONS option of the flask-sqlalchemy module to set the pool_pre_ping configuration as outlined in the Module Docs

• enferno/tasks/__init__.py

  • Added celery.conf.update({'database_engine_options': cfg.SQLALCHEMY_ENGINE_OPTIONS}) to set the value

• enferno/settings.py

  • Added a config for SQLALCHEMY_ENGINE_OPTIONS that defaults to True

Redis Configuration

The existing configs for Redis were not working when running in Docker Compose. This adds configuration items to fix the issue and make the naming a bit more consistent.

• enferno/settings.py
  • Breaks out configuration items for Redis hostname, password, and port so we can build a REDIS_URL_BASE to use for the other value defaults
  • Added the REDIS_BROKER_DB configuration to set the database used in the default value for celery_broker_url
  • Added the REDIS_DB configuration to set the database used in the default value for REDIS_URL
  • Added the REDIS_RESULT_DB configuration to set the database used in the default value for result_backend
  • Set celery_broker_url and result_backend using the new Redis hostname and port values by default but still allow CELERY_BROKER_URL to override it when set
  • Added SESSION_DB_REDIS to set the session DB used in the default value for SESSION_REDIS
  • Build the SESSION_REDIS string using the separate hostname, port, and new SESSION_DB_REDIS values to ensure they are consistent but still allow SESSION_REDIS to override it when set

Dockerfile

• Updated the image to python:3.8.16-slim because numpy 1.22.0 isn't supported on Python 3.7
• Changed the apt-get install command
  • Removed python-pip as it's not a package in the updated Docker image
  • Added the exiftool installation as that's a requirement based on this issue
  • Added build-essential as the uwgi package instructions require it
• Added the following environment variables so the Redis connection could be configured
  • REDIS_HOSTNAME
  • REDIS_PASSWORD
  • REDIS_PORT
  • REDIS_BROKER_DB
  • REDIS_DB
  • REDIS_RESULT_DB
  • SESSION_DB_REDIS
• Removed the following env variables as they are built using the separate REDIS_CELERY_HOSTNAME and REDIS_CELERY_PASSWORD config items
  • CELERY_RESULT_BACKEND
  • CELERY_BROKER_URL
• Removed SESSION_REDIS and added SESSION_DB_REDIS env vars as REDIS_SESSION is now built using the separate hostname and port values for the Redis server

PostgreSQL Service

• Adds a Dockerfile for PostgreSQL (postgres/Dockerfile)
• Updates the PostgreSQL version to 14.2
• Installs the postgresql-14-postgis-3 package during the Docker build stage so it doesn't need to be added later

Docker Compose Changes

Adds the ability to run in a Docker Swarm cluster as well as incorporating changes to run more effectively using Docker Compose.

• docker-compose.override.yml
  • Docker Compose Override files are automatically used when running using the default command (docker-compose up -d). Using it here so the local, development specific settings aren't run in a Docker Swarm context
  • Adds the optional nginx service using a profile so it doesn't run by default
• docker-compose.yml
  • Moves the nginx service section to the override file as it's not usually needed during local development or in a Docker Swarm
  • Utilizes named volumes for many of the mapped volumes such as the Postgres /opt/postgres/data directory as that is a bit more user-friendly than mapping local volumes
  • Adds environment variables for the mapped ports so they can be changed in the .env file
  • Updated the command for the celery service as the -A flag has to be set at the global command level
  • Removed the links fields as those are automatically setup when running in Docker Compose so specifying them in the links field is redundant
• docker-compose.swarm.yml
  • Adds image tags and volumes to run in a Docker Swarm cluster
  • Adds a COMPOSE_REGISTRY environment variable for setting the URL for the private Docker Registry to use to store and fetch images
  • Adds a volume mapping for storing automatic PostgreSQL backups
  • Changes many of the volume maps from named volumes to mapped ones as named volumes don't work in Docker swarm without storage plugins. This allows you to just use directories shared on all Swarm nodes using something like NFS to store files

Miscellaneous Changes

• Removed extra newlines in various files, but this is just personal preference.
• enferno/admin/models.py
  • Fixed minor typo in a comment
• enferno/admin/templates/admin/actors.html
  • Fixed a typo in the formTitle return statement, changing this.translations.newActr_ to this.translations.newActor_
• import-data/create-labels.sql
  • Added to automatically create labels for war crimes
• .env-sample
  • Added the new, separate Redis configuration values
  • Commented out SESSION_REDIS, CELERY_BROKER_URL, and CELERY_RESULT_BACKEND as using the separate values for Redis hostname and such make things a bit more consistent; left an explanation for when it might be good to uncomment them for use
  • Added the other new configuration values for PostgreSQL, Docker, and the initial admin user credentials

[sjacgit]

Many thanks for your contribution. We'll review the changes and respond in due course. However, we'll likely to ask to amend this PR or submit a new one as he Docker changes are in conflict with the upcoming new version we're planning to release shortly. We'll revert back shortly. Again, thank you so much.

[sfpurdy]

Many thanks for your contribution. We'll review the changes and respond in due course. However, we'll likely to ask to amend this PR or submit a new one as he Docker changes are in conflict with the upcoming new version we're planning to release shortly. We'll revert back shortly. Again, thank you so much.

Okay, sounds good. I can update this PR once you publish the changes.

[sjacgit]

Thanks for your patience. The changes are now push to the repo. Could you please review as agreed and revert back?

[sfpurdy]

Thanks for your patience. The changes are now push to the repo. Could you please review as agreed and revert back?

Yup, I'll review and update this PR.