GitHub Runners

[andreasisnes]

Description

VNet-Integrated GitHub Actions Runners for Secure Azure Resource Access

Summary

@Altinn/team-authorization and @Dialogporten teams run specific tasks via GitHub Actions that require direct communication with Azure resources. These resources are only accessible within an Azure Virtual Network (VNet) and are not exposed to the internet.

• Dialogporten requires access to Azure Key Vault for retrieving secrets during their GitHub Actions workflows. They have raised concerns about the security implications of exposing the Key Vault to the internet.

• @Altinn/team-authorization has set up a PostgreSQL database server and needs to execute specific SQL scripts for user and role configuration post-deployment. This requires direct communication with the database, which cannot be facilitated through ARM APIs but must be done via direct network access.

Proposed Solution

To address these challenges, we propose setting up VNet-integrated, self-hosted GitHub runners for each Altinn product team. This setup would enable secure, private communication between GitHub Actions and the necessary Azure resources within the VNet, bypassing the need for internet exposure.

Design Considerations

1. Isolation and Flexibility:

   • Each Altinn product team should have its own dedicated VM for running GitHub Actions. This ensures:
     • DNS Resolution: DNS Private Links are used to resolve private IPs. Each private link needs to be associated with specific VNets for correct DNS resolution. Using shared runners complicates this, as it requires multiple private links to be associated with a single VNet.
     • Team Independence: Altinn product teams should operate independently without needing to interact with each other’s resources through GitHub Actions.
     • Customization: Teams can configure their GitHub runners as needed without impacting the setup of other teams.

2. Deployment Process:

   • Each team will deploy their own GitHub runner VMs by utilizing a Terraform module, which is to be maintained by @Altinn/team-platform.
   • The Terraform module will include a comprehensive README to guide teams on configuring and creating their GitHub runner instances.

References

• Adding a self-hosted runner to a repository - GitHub Documentation

Security Considerations

• Forks of public repository can potentially run dangerous code on self-hosted runner machine by creating a pull request that executes the code in a workflow.

For more information, see "About self-hosted runners."

No response

[andreasisnes]

WIP

[tjololo]

In your design considerations you say that each team should have it's own VM for running Github Actions. I think we should investigate the possibility to use Azure Container Apps instead. Pros of using container apps:

• Possible to implement scale to zero so we don't pay for idling VMs
• "Limitless" scaling, we can leverage the scaling capabilities of azure container apps to scale out if there is a sudden surge in actions
• No need for updating and maintaining VMs
• Clean workers for each run of the github actions

Cons of using container apps

• More complex initial setup
• Last time I tested it there was some issues when trying to authenticate against github to register the runner
• No possibility to debug the runner after it has completed its work, as the container app is deleted (could also be a nice security feature)

Not a complete list of pros and cons but a start

[altinnadmin]

@tjololo You beat me, I was planning to give the same feedback 👍

Also, is this something that makes it impossible to share IaC as part of the public product repos?

[andreasisnes]

Sure. I also think Container apps is a good solution, if we can produce a seamless setup for container apps, then, I'm very supportive of it! I think it's a requirement that self-hosted GitHub runners are isolated to each product due to.

• Altinn Products have different set of environments and infrastructure. I do not believe all teams / products share network (please, correct me if i'm wrong), thus, resources are not in the same network at all. Introducing shared runners forcing them to share network which is a poor incentive to start sharing network.

• We must have github runners for each environments due to CIDR blocks and environment isolation.

Container apps is kubernetes with vm nodes at root, idle VMs are inevitable. There is a small limitation with Container apps

• jobs don't support running Docker in containers. Any steps in your workflows that use Docker commands will fail when run on a self-hosted runner or agent in a Container Apps job.

But yes, if we can make it seamless with container apps that would be nice. @altinnadmin another incentive to start automating github as self-hosted runner are created we also need to configure repositories to use these runners :smile_cat:

@altinnadmin i do believe that the fork policy for altinn repositories is that fork requests of repos must be accepted by an admin?

There is a 3. solution, where you can deploy the NIC of github runner into an azure vnet, it looks elegant, but i requires enterprise license. https://docs.github.com/en/enterprise-cloud@latest/admin/configuring-settings/configuring-private-networking-for-hosted-compute-products/about-azure-private-networking-for-github-hosted-runners-in-your-enterprise

[altinnadmin]

i do believe that the fork policy for altinn repositories is that fork requests of repos must be accepted by an admin?

Nah, anyone can fork any public repo on github. The problem is when a PR is created from the fork, and we automatically run an action on a self-hosted runner before review has been done.

[andreasisnes]

So if anyone can fork a repository, then it is possible to do the following. fork, create a new github action, create a PR. in the pipeline they can specify any AT environment in github actions (AT environments don't require review) and read any repo (environment) / organization variables and secrets from a repository.

[tjololo]

I think secrets in github are protected. They should not be available from forks, but the runner would be possible to leverage

[andreasisnes]

@tjololo, but what about identity federation, will you have access to the github token then you can initiate token exchange with azure

[tjololo]

@andreasisnes yes, but federations also take repository and branch, tag or environment into consideration so they should if setup correctly be safe also

[tjololo]

I have a working POC in my personal github/azure setup using Azure Container App Job to run self-hosted runners. This setup will start a container when there is a job queued in github that matches the labels for the runner that is added to a repo. Once a job is completed the container is discarded (ephemeral runner). This should limit the possibility to "dirty" the self-hosted runner for subsequent jobs. While testing it seems like when a job is queued is will have to wait for a minute before the runner is started and ready to accept work.

The benefits with this setup, as I see them:

• Each team can have their own Azure Container App Env that is isolated in its own virtual network
• Each job is run on a fresh container limiting the possibility for a job to dirty the context of a subsequent job
• Containers and runners are only started when there is a job queued and we don't pay for idling cpus/memory
• team-platform maintains a base image for our runners. If a team needs to extend this the can either roll their own or extend on the base image, but they will have the responsibility to maintain the image and container apps job setup
• team-platform can maintain most of the infrastructure and it has small to none differences between teams (only the image can wary and if a custom image is used the team is responsible for updating it)

Drawbacks:

• Docker is not available in our self-hosted setup
• One minute startup time

IMO. the drawbacks are ok as we should strive to use github-hosted runners for as much as possible. For my personal setup I also have a key vault that only is accessible from given VNets, but I work around this by adding a pre-script that adds the current runners IP to the allowed list and a post-script that removes it again. This in combination with federated identity should reduce the chances of an attacker gaining access to the key vault, as he needs to obtain the short lived credentials from the federated identity and run his attack from the same IP as the runner before the post-script is executed.

[andreasisnes]

@tjololo good work! This looks fine to me. It don't mind if the platform team maintains the container app environments, but if you could create some nice terraform / biceps template with some simple parameters we can probably deploy it ourself, but use the templates that the platform team maintains.