GitHub Self-Hosted Runners Implementation

[AErmie]

As part of supporting more secure application infrastructure deployments, leveraging technologies like Private Link Endpoints, VNet Service Endpoints, and VNet Integration, etc. we need to explore how to support using self-hosted Agents/Runners for pipeline workflow execution. This would allow the Runner to operate within a private network, and thus be able to support deployment of, and interaction with, private-only resources (ie. Azure resources that do not have public access enabled).

Part of this work initially started with the Azure Startup Sample Application (Serverless) example.

[!NOTE] The Azure Startup Sample Application (Serverless) repo currently has separate sub-module code for deploying the networking required for the GitHub self-hosted Runner environment, along with code for the Azure Container App Jobs-based implementation of self-hosted runners.

This code should be moved into its own repository, so that it can be more easily iterated on, and centrally used and referenced by various Ministry teams.

Implementation Options

• Custom created Terraform code (this is the current implementation to-date)
• Azure Verified Module for CI/CD Agents and Runners

[!NOTE] The current self-hosted runner implementation utilizes Container Apps, which does not support running docker commands.

We need to investigate options for running Docker-in-Docker (DIND), and if the Container App method works with this. This would remove the need to also create and maintain a VM-based runner approach (via VM Scale Sets). If we do need to explore a VM-based option, a good starting point would be the GitHub Actions Runner Images, and Azure Virtual Machine Scale Set agents repos.

Reference articles:

• Creating a Docker in Docker (DinD) container with any base image
• Azure DevOps Agent: Azure Container Instance with Private Azure Container Registry
• Get Ahead with Self-Hosted Agents and Container Apps Jobs

In addition to the custom created Terraform code, and Azure Verified Module implementation options, we also need to explore implementation and usage of GitHub-hosted runners in a private network.

• About Azure private networking for GitHub-hosted runners in your organization

[!NOTE] We are currently working with the Security, and Developer Experience (DevEx) teams to explore this option, as it is implemented and configured at the GitHub Organization level.

Acceptance Criteria

Generated by Zenhub AI

• [ ] Scenario: Implementation of self-hosted runners for private network
  • Given a self-hosted runner implementation is required for private networks
  • When using Terraform code or Azure Verified Module to create the necessary infrastructure
  • Then the runner should be able to operate within a private network and interact with private resources
  • [ ] Scenario: Explore Docker-in-Docker (DIND) support for Container Apps
  • Given the current self-hosted runner implementation uses Container Apps
  • When investigating options for running Docker-in-Docker within Container Apps
  • Then determine if this is a viable option, or if a VM-based approach is necessary
  • [ ] Scenario: Evaluate GitHub-hosted runners in private network
  • Given the need to explore implementation and usage of GitHub-hosted runners in a private network
  • When working with Security and Developer Experience teams to evaluate this option
  • Then determine if this is a viable alternative, or if self-hosted runners are required

[AErmie]

Met with DevEx team (who is responsible for the GitHub Enterprise Organization). Will continue to explore options, including the GitHub-managed runners, and include the DevEx team when ready to test.

[AErmie]

The DevEx team met with the GitHub Account Team, and was able to confirm that there is no ETA for when GitHub-managed runners will be supported in the Canada Azure regions.

[AErmie]

Continued testing the Azure Verified Module for CI/CD Agents and Runners for GitHub Runners, but encountered several errors/issues, as has been reported in the following GitHub Issue: [AVM Module Issue]: 500 Internal Server Error When Creating Container App Job.