Open andreasisnes opened 1 month ago
WIP
In your design considerations you say that each team should have it's own VM for running Github Actions. I think we should investigate the possibility to use Azure Container Apps instead. Pros of using container apps:
Cons of using container apps
Not a complete list of pros and cons but a start
@tjololo You beat me, I was planning to give the same feedback 👍
Also, is this something that makes it impossible to share IaC as part of the public product repos?
Sure. I also think Container apps is a good solution, if we can produce a seamless setup for container apps, then, I'm very supportive of it! I think it's a requirement that self-hosted GitHub runners are isolated to each product due to.
Altinn Products have different set of environments and infrastructure. I do not believe all teams / products share network (please, correct me if i'm wrong), thus, resources are not in the same network at all. Introducing shared runners forcing them to share network which is a poor incentive to start sharing network.
We must have github runners for each environments due to CIDR blocks and environment isolation.
Container apps is kubernetes with vm nodes at root, idle VMs are inevitable. There is a small limitation with Container apps
But yes, if we can make it seamless with container apps that would be nice. @altinnadmin another incentive to start automating github as self-hosted runner are created we also need to configure repositories to use these runners :smile_cat:
@altinnadmin i do believe that the fork policy for altinn repositories is that fork requests of repos must be accepted by an admin?
There is a 3. solution, where you can deploy the NIC of github runner into an azure vnet, it looks elegant, but i requires enterprise license. https://docs.github.com/en/enterprise-cloud@latest/admin/configuring-settings/configuring-private-networking-for-hosted-compute-products/about-azure-private-networking-for-github-hosted-runners-in-your-enterprise
i do believe that the fork policy for altinn repositories is that fork requests of repos must be accepted by an admin?
Nah, anyone can fork any public repo on github. The problem is when a PR is created from the fork, and we automatically run an action on a self-hosted runner before review has been done.
So if anyone can fork a repository, then it is possible to do the following. fork, create a new github action, create a PR. in the pipeline they can specify any AT environment in github actions (AT environments don't require review) and read any repo (environment) / organization variables and secrets from a repository.
I think secrets in github are protected. They should not be available from forks, but the runner would be possible to leverage
@tjololo, but what about identity federation, will you have access to the github token then you can initiate token exchange with azure
@andreasisnes yes, but federations also take repository and branch, tag or environment into consideration so they should if setup correctly be safe also
I have a working POC in my personal github/azure setup using Azure Container App Job to run self-hosted runners. This setup will start a container when there is a job queued in github that matches the labels for the runner that is added to a repo. Once a job is completed the container is discarded (ephemeral runner). This should limit the possibility to "dirty" the self-hosted runner for subsequent jobs. While testing it seems like when a job is queued is will have to wait for a minute before the runner is started and ready to accept work.
The benefits with this setup, as I see them:
Drawbacks:
IMO. the drawbacks are ok as we should strive to use github-hosted runners for as much as possible. For my personal setup I also have a key vault that only is accessible from given VNets, but I work around this by adding a pre-script that adds the current runners IP to the allowed list and a post-script that removes it again. This in combination with federated identity should reduce the chances of an attacker gaining access to the key vault, as he needs to obtain the short lived credentials from the federated identity and run his attack from the same IP as the runner before the post-script is executed.
@tjololo good work! This looks fine to me. It don't mind if the platform team maintains the container app environments, but if you could create some nice terraform / biceps template with some simple parameters we can probably deploy it ourself, but use the templates that the platform team maintains.
Description
VNet-Integrated GitHub Actions Runners for Secure Azure Resource Access
Summary
@Altinn/team-authorization and @Dialogporten teams run specific tasks via GitHub Actions that require direct communication with Azure resources. These resources are only accessible within an Azure Virtual Network (VNet) and are not exposed to the internet.
Dialogporten requires access to Azure Key Vault for retrieving secrets during their GitHub Actions workflows. They have raised concerns about the security implications of exposing the Key Vault to the internet.
@Altinn/team-authorization has set up a PostgreSQL database server and needs to execute specific SQL scripts for user and role configuration post-deployment. This requires direct communication with the database, which cannot be facilitated through ARM APIs but must be done via direct network access.
Proposed Solution
To address these challenges, we propose setting up VNet-integrated, self-hosted GitHub runners for each Altinn product team. This setup would enable secure, private communication between GitHub Actions and the necessary Azure resources within the VNet, bypassing the need for internet exposure.
Design Considerations
Isolation and Flexibility:
Deployment Process:
References
Security Considerations
For more information, see "About self-hosted runners."
No response