Closed marwan37 closed 4 months ago
[!IMPORTANT]
Auto Review Skipped
Auto reviews are disabled on this repository.
Please check the settings in the CodeRabbit UI or the
.coderabbit.yaml
file in this repository.To trigger a single review, invoke the
@coderabbitai review
command.
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?
@marwan37 you'll need to add a step to the ci.yaml
file so that the new integration tests runs, something like:
ubuntu-integration-test:
needs: ubuntu-setup-python-environment # possibly remove if it doesn't need this?
strategy:
matrix:
os: [ubuntu-latest]
python-version: ["3.9", "3.11"]
fail-fast: false
uses: ./.github/workflows/aws-integration-test.yml
with:
os: ${{ matrix.os }}
python-version: ${{ matrix.python-version }}
secrets: inherit
Otherwise it won't run with PRs etc..
Update: I kept encountering failures with the new workflow file during the aws-modular
integration test, due to issues with capturing outputs in the runner environment. A Stack Overflow post suggested using terraform-bin
instead of terraform
and disabling terraform_wrapper
. This change finally solved the issue and correctly captured the stack-yaml-path
as shown below:
- name: Output Stack YAML Path
id: set_output
run: |
OUTPUT=$(terraform-bin output -raw stack-yaml-path)
echo "stack_yaml_path=$OUTPUT" >> $GITHUB_OUTPUT
working-directory: src/mlstacks/terraform/aws-modular
env:
terraform_wrapper: false
An adjustment was also necessary to use an absolute path in the test script:
- name: Run Tests to Verify Resource Provisioning
run: |
STACK_YAML_PATH="${{ steps.set_output.outputs.stack_yaml_path }}"
ABSOLUTE_PATH="${GITHUB_WORKSPACE}/src/mlstacks/terraform/aws-modular/${STACK_YAML_PATH}"
../../../../tests/integration/aws-modular/verify_stack.sh "$ABSOLUTE_PATH"
working-directory: src/mlstacks/terraform/aws-modular
Finally, I simplified the integration test setup by replacing tflocal
with a manual override approach. This involved using an _override.tf
file in tests/integration
to override the aws
provider without setting any resource configurations to preserve the flexibility of our .tfvars
files. The _override.tf
file is copied to the appropriate mlstacks/terraform
module for each test and is removed afterwards. This simplifies the CI configuration in ci.yml
to:
localstack-aws-integration-test:
uses: ./.github/workflows/aws-integration-test.yml
secrets: inherit
Neither Python nor a specific OS is needed to run the test, and it can run concurrently with others.
@marwan37 seems like one of the scripts needs to be made an executable? At least that seems to be the failure?
Yep! Fixed and repushed.
@marwan37 thanks as always for your contribution!
Describe changes
I implemented a LocalStack integration for MLStacks to create a local testing environment for AWS deployments, addressing the objectives outlined in issue #136. This includes setting up LocalStack to emulate AWS services (e.g., S3, DynamoDB), creating a test environment within
tests/integration
for running integration tests, and adding a new workflow file in.github/workflows
to execute these tests. The aim was to enhance development efficiency and reliability by allowing local testing in a simulated AWS environment.Pre-requisites
Please ensure you have done the following:
develop
and the open PR is targetingdevelop
. If your branch wasn't based on develop read Contribution guide on rebasing branch to develop.Types of changes
Detailed Description
Following the task description, I developed a POC using LocalStack to emulate AWS services relevant to MLStacks deployments, focusing on S3 and DynamoDB for
aws-remote-state
, and S3 with Skypilot enabled foraws-modular
. This involved:tests/integration
with subdirectories foraws-modular
andaws-remote-state
, encapsulating all LocalStack integration test configurations..tfvars
files and scripts (verify_stack.sh
) to dynamically configure and verify AWS resources provisioned via LocalStack.aws-integration-test.yml
) to automate setup, execution, and cleanup of LocalStack integration tests.docker-compose.localstack.yml
, not utilized in the workflow file, but available for developers to use and customize for quick LocalStack setup.tests/integration
directory to provide documentation for setting up, executing, and understanding the LocalStack integration tests within MLStacks.I propose integrating the
aws-integration-test.yml
workflow into the mainci.yml
, like this:This allows us to run LocalStack integration tests before proceeding with AWS-specific tests, and without cluttering the main CI pipeline.
Additional Context
This integration aims to provide a framework for future testing enhancements, and serves as a basis for LocalStack-based tests within MLStacks.
Note: This integration does not include EKS based services due to the limitations of LocalStack's Community Edition, as discussed with @strickvl. Future enhancements will be revisited upon the completion of issue #140.