While working on writing integration tests for deadline-cloud-worker-agent, I realized there could be some improvements to our test library to make it easier to write tests
What was the solution? (How)
Rework the entire package. See detailed changes below.
Also fixed the add-copyright-headers.sh script and made the copyright header test run by default
What is the impact of this change?
It's a little easier to use this package for integration tests (specifically for Worker agent)
The EC2 implementation is similar to our original fixture but has been changed to use the Worker agent installer to do most of the configuration
Added a Docker container implementation so integ tests can be run on a local Docker container
Note that this mode does not support installing the systemd service, because that's not recommended in Docker containers. See Worker agent PR for addition of --no-install-service option to its installer
Added ability to copy over files from host machine to Worker agent environment (EC2 instance or Docker container). Currently, this provides a generic way to facilitate:
Using a dev build of the Worker agent
Using a local Deadline Cloud model file
fixtures.py: The number of fixtures has been reduced since the functionality has moved into objects that are returned by the fixtures. Currently, we have:
deadline_client: Our shim layer boto client that is configurable with DEADLINE_ENDPOINT env var
codeartifact: Stores info about the CodeArtifact repo to pull Python deps from based on env vars. Currently only used for Worker.
service_model_s3_object: Allows specifying the model file to use which is stored in S3 via DEADLINE_SERVICE_MODEL_S3_URI. Currently only used for Worker.
bootstrap_resources: Deploys the CloudFormation stack for bootstrapping Worker resources. Only used for Worker.
deadline_resources: Deploys an opinionated set of Deadline Cloud resources geared towards the Worker.
worker: Deploys a Worker.
deploy_job_attachment_resources: The existing fixture used by Job Attachments integ tests
job_attachment_manager: Changed to use new deadline/ and cloudformation/ code.
I tried to keep the public interface as similar as I could to the existing interface to make it easy to upgrade the Job Attachments test to this new version. Hopefully it's just a few renames
models.py: Contains common dataclasses. Open to suggestions for the module name.
pytest_hooks.py: Defines pytest hooks to enhance live logging for integ tests
Without this, the live logging messages are prefixed with the filename + line in this package where the log message is emitted. I found that to be not helpful and changed it to prefix messages with the test currently being run. There is still an issue where it creates double prefixes when tests fail.
deadline_manager.py: Replaced by code in deadline/
constants.py:
Constants were moved closer to where they are needed and removed the need to share them
Env var configurations were moved to fixtures.py
util.py: Changed to contain utility functions used across the package
test/unit: Updated/added unit tests. Added moto as a test dependency to make testing easier.
Given the size of this PR, I recommend quickly glazing over the unit tests
What was the problem/requirement? (What/Why)
While working on writing integration tests for deadline-cloud-worker-agent, I realized there could be some improvements to our test library to make it easier to write tests
What was the solution? (How)
add-copyright-headers.shscript and made the copyright header test run by defaultWhat is the impact of this change?
It's a little easier to use this package for integration tests (specifically for Worker agent)
How was this change tested?
Was this change documented?
Yes, added to README.md and an example configuration file
Is this a breaking change?
Yes
Changes
cf_templates/: Replaced by code incloudformation/cloudformation/: Models CloudFormation templates and handles deploymentcfn.py: Core model for CloudFormation resourcesjob_attachments_bootstrap_stack.py: Replacement for old CFN template undercf_templates/worker_bootstrap_stack.py: Stack deploying bootstrap resources for Worker agent deploymentdeadline/: Models Deadline Cloud resources. Currently, this includes:client.py: The existing shim layer we had for the boto client to handle API changesstubs.py: The existing stub for the boto client that we use in some unit testsresources.py: Models Deadline Cloud resources (e.g. Farm, Queue, Fleet, Job, etc.)worker.py: Handles deploying a Worker agent.--no-install-serviceoption to its installerfixtures.py: The number of fixtures has been reduced since the functionality has moved into objects that are returned by the fixtures. Currently, we have:deadline_client: Our shim layer boto client that is configurable withDEADLINE_ENDPOINTenv varcodeartifact: Stores info about the CodeArtifact repo to pull Python deps from based on env vars. Currently only used for Worker.service_model_s3_object: Allows specifying the model file to use which is stored in S3 viaDEADLINE_SERVICE_MODEL_S3_URI. Currently only used for Worker.bootstrap_resources: Deploys the CloudFormation stack for bootstrapping Worker resources. Only used for Worker.deadline_resources: Deploys an opinionated set of Deadline Cloud resources geared towards the Worker.worker: Deploys a Worker.deploy_job_attachment_resources: The existing fixture used by Job Attachments integ testsjob_attachment_manager: Changed to use newdeadline/andcloudformation/code.models.py: Contains common dataclasses. Open to suggestions for the module name.pytest_hooks.py: Defines pytest hooks to enhance live logging for integ testsdeadline_manager.py: Replaced by code indeadline/constants.py:fixtures.pyutil.py: Changed to contain utility functions used across the packagetest/unit: Updated/added unit tests. Addedmotoas a test dependency to make testing easier.