Closed brianherrera closed 2 years ago
This looks good and I approve this direction. Some suggestions:
Thanks for the feedback. I'll update the RFC with these points in addition to the feedback received in the build sig meeting. I'll submit the PR to add it to the sig-build repo when the final draft is ready.
Summary:
The purpose of this feature is to provide the O3DE Jenkins Pipeline setup to developers. This includes maintainers/contributors that will submit improvements to the O3DE Jenkins pipeline and game developers that want to use the AR pipeline for their own project.
The goal is to move the pipeline configuration from an internal process to a public one managed by the Build SIG with all the required files stored in a repo under the O3DE org. The current setup has a combination of CloudFormation stacks and manually deployed resources managed by an internal Amazon team.
The release of this feature will not change the behavior of the AR pipeline, but it will allow the O3DE community to maintain and contribute changes to the Jenkins setup.
What is the relevance of this feature?
This is important because it allows developers to contribute changes to the pipeline setup and enables the following:
Feature design description:
All the scripts and configs required to setup our Jenkins infrastructure will be packaged in a repo. A pipeline will automatically deploy the required resources when updates are merged in.
A CDK package will also be included to easily deploy the service to AWS. This includes a CodePipeline instance that will be used to build, test, and deploy the service.
Use cases:
Update the Jenkins server setup: a user will clone the repo and make changes (e.g. install a new plugin, update the jenkins version, etc.). Then they can test these changes locally by building the docker image and running the container on their local machine. After testing, the user can then submit a PR with their changes. After the changes are reviewed and merged, the pipeline will take the change, deploy it to staging, then finally to production.
Use the AR for game projects: a user will fork the repo and make any customizations they need to the Jenkins server setup. Using this configuration they can easily setup the AR pipeline to run on their project. They can host Jenkins on their own server or a cloud provider using the templates provided.
Technical design description:
Jenkins config components:
scripts/build/Jenkins/Jenkinsfile
in the O3DE repo.AWS Hosting components:
Deployment workflow:
The core part of this setup is the docker container used to run the Jenkins server. To build the image, we pull down the latest LTS version of the Jenkins docker image and run our Dockerfile with our customizations.
Testing Deployments
Here are the options to test changes before they are deployed to production:
cdk synth
to validate changes to the CDK packages prior to deployment and usecdk deploy
to test the changes in their own AWS account.Rollback
In the event that issues are identified in the prod stack, a rollback mechanism will be required to redeploy the last known good version.
These issues should be identified with automated mechanisms to rollback quickly and effortlessly. We should also provide a manual rollback mechanism if issue are identified outside of automated testing.
CDK Pipelines doesn't provide a built-in mechanism to rollback yet, however the CodeDeploy resource within the Jenkins ECS stack can be utilized to perform a rollback if necessary.
Deployment Permissions
For simplicity, a manual approval step will be added to promote changes from staging to prod. We will need to investigate additional mechanisms to promote these changes. This includes maintenance windows, merge queues, etc. We will need to work with the build SIG to identify the users that will have access to promote changes.
Additionally, a subset of build SIG maintainers will need direct access to the AWS account hosting the pipeline. This will be required in the event an investigation requires access to the AWS account to resolve an issue.
What are the advantages of the feature?
Jenkinsfile
in the o3de repo, developers can use this setup to run the same pipe on their project and customize as needed.What are the disadvantages of the feature?
Are there any alternatives to this feature?
An alternative is to migrate our pipeline workflows to GitHub actions. While we have plans to utilize this mechanism for smaller automated workflows, there are some limitations that prevent us from migrating the entire pipeline to GH actions. This is mainly due to storage and compute restrains for O3DE builds. It's possible to utilize self-hosted runners, so we would still maintain some of our current build infrastructure with this solution. That said, there are planned investigations into how we can integrate GH actions into our pipeline.
How will users learn this feature?
Are there any open questions?
main
will be created for our production deployments and branch protection will be enabled to require approvals and successful checks. The pipeline used for deployment will also have additional protections when moving changes from staging to prod.