Celery

[andyneff]

Finalize working support for celery executor and multiprocess logging

Breaks

• terra.settings.terra.zone can be misrepresented in ThreadPoolExecutor, resulting in log messages appearing to come from the wrong zone.

Added

• Added settings.terra.zone to settings
• Added zone to logging
• Added color formatter to logging
• Added Debug4 level to logging
• Added terra.tests.demo app and service for testing config file in.
• Terra demo app auto added to celery apps
• TERRA_KEEP_TEMP_DIR env var to keep temp autogenerated files for debugging
• Celery has a fourth zone known as task_controller for the celery worker's parent
• Executors have a volume map translation too
• Task kwargs and returns values are can have map translation applied to them
• ConfigurationWarning
• Each run of a terra app gets assigned a UUID, settings.terra.uuid
• All multi processing processes use a TCP server for sending logs, using settings.logging.server.hostname and settings.logging.server.port
• Settings can be pickled
• terra.logging.LogRecordStreamHandler to log out to TCP
• terra.logging.LogRecordSocketReceiver to run a threaded server to receive TCP log messages
• Terra's own task class, terra.task.TerraTask

Changes

• TERRA_CELERY_MAIN_NAME is now auto deteremined by default. It only has to be set if the main App CLI is a built-in (aka compiled python module) which is not typical.
• Computes are now responsible for configuring and reconfiguring the logger for controller and runners zones
• Executors are now responsible for configuring and reconfiguring the logger for task zones
• Executor and compute volume maps are stored in settings now.
• Settings files dumped during processing contain the uuid, to make determining which run is which easier.
• All executors (including thread/process) inherit from terra.executor.base.BaseExecutor
• Celery CLI is no longer called directly, instead terra.executor.celery should be called
• Celery logging facilities are disabled, only terra logging used now
• terra.logging.get_logger -> terra.logging.getLogger
• Default format includes hostname and zone.
• Runner and tasks no longer output log messages to stderr. Instead the master controller will output the runner's and task's messages (received via TCP) itself, to stderr.
• Uncaught exceptions stack traces now have a header with the hostname and zone
• Uncaught exceptions are no longer printed twice due to logging
• logging file_handler -> main_log_handler
• hostname, zone, and other logging settings set with filters instead of formatters or "extra" variables, it is a more universal entrypoint for adding values
• Tasks should now be decorated with terra.task.shared_task

Bug Fixes

• Fixed new? bug on CI with flake8 being run in the wrong directory, not finding the config file all of a sudden
• Fixed typos in Redis command secret file variable names, causing the wrong variable to be used
• Added celery as a dependency of terra
• Removed invalid password characters from auto generated redis password
• Docker compute use non-TTY containers, to fix a stdout nl/cr bug
• Remove duplicated volumes in singularity
• Not setting the virtualenv_dir in virtualenv compute now issues a warning
• Fixed tilde expantion in settings.
• Celery shutdown uses a tuple of self._futures to create a copy of futures and prevent a race condition
• Catch file not existing when deleting tmp_file

Testing

• Check to see if any test permanently modifies os.environ by accident
• Refactors test cases so settings, logging and other specical cases are all handled by Test Cases classes in terra.tests.utils
• TestSettingsUnconfiguredCase prepares terra.core.settings, but leaves it unconfigured
• TestSettingsConfiguredCase, same as TestSettingsUnconfiguredCase, but configures it using an empty dictionary.
• TestLoggerCase - An Unconfigured Settings case, that uses a modified temp dir function so that all temp files are created in the test dir, preventing /tmp from being spammed with temp terra log files. Also mocks the TCP server so the TCP server thread is not actually started.
• TestComputeCase - Allows for a compute backend to be loaded
• TestExecutorCase - Allows for an executor backend to be loaded.
• TestSignalCase - Enables signals. By default, signals are disabled for all unit tests.
• TestLoggerConfigureCase - Test logging with settings configured and signals enabled, but TCP Server is still disabled.