Open marcofranssen opened 4 years ago
This worked for me:
https://github.com/actions/checkout/issues/116#issuecomment-644419389
Thanks @marcofranssen, this is just want I needed. this is a partial solution.
For anyone else looking, deploy keys are a partial fix.
The problem with deploy keys and a separate clone submodules step is that you need to keep the submodule ref and the ref in github actions the same, editing the setting in two places.
Personal access tokens as suggested by @beroso work, but either involve giving access to all your repos, or creating a new machine user and adding them as a collaborator, big faff.
It would be great if github could provide a proper and simple way to clone private submodules.
For anyone else having trouble with this, I have a solution that doesn't require personal access tokens but keeps the reference to the child repo commit in one place (using git submodules)
- name: clone submodule
uses: actions/checkout@v2
with:
repository: <org name>/<repo name>
path: path
ssh-key: ${{ secrets.SSH_KEY }}
persist-credentials: true
- name: checkout submodule
run: |
git submodule init
git submodule update
although the action checks out master, the git submodule
commands check out the correct commit, this avoids having to keep the ref
in github actions.
See, this is what I really want... just the persist-credentials part. then I could have
- uses: actions/deploykey@v?
with: ssh-key
and all it does is make it so the next git command or ssh command can use that key.
Instead of using an SSH key (:scream: ) you can simply use a personal access token:
- uses: actions/checkout@v2
with:
submodules: recursive
token: ${{ secrets.ACCESS_TOKEN }}
Here the ACCESS_TOKEN
variable is a personal access token
For anyone else having trouble with this, I have a solution that doesn't require personal access tokens but keeps stores the reference to the child repo commit in one play (using git submodules)
Correct me if I'm wrong, but this wont work for multiple private submodules, because each needs their own deploy key.
I ended up building a rudimentary private package manager for python to get round this problem.
I think the reason github aren't fixing it is that they think the long term solution should be their package manager(s).
I ended up just embedding the private keys for read-only deploy keys in the yml workflow files directly as ENV variables then following this advice with the hard coded strings instead of secrets (which aren't available for free in GitHub Org private repos): https://rgoswami.me/posts/priv-gh-actions/
Hacky and gross, but it works.
I ended up just embedding the private keys for read-only deploy keys in the yml workflow files directly as ENV variables then following this advice with the hard coded strings instead of secrets (which aren't available for free in GitHub Org private repos): https://rgoswami.me/posts/priv-gh-actions/
Hacky and gross, but it works.
Less hacky and gross variant, using ssh-agent
instead of writing to a file inside .ssh
:
- name: Get submodules
env:
SSH_KEY_SUBMODULE: ${{secrets. SSH_KEY_SUBMODULE}}
run: |
eval `ssh-agent -s`
ssh-add - <<< "${SSH_KEY_SUBMODULE}"; git submodule update --init --recursive
It's a travesty really that this doesn't work out of the box. It forces behaviors that should be discouraged - people fret about submodules (they shouldn't), people have to pay for dummy GitHub accounts just to make this work, people have to bend backwards to get something that is well within the minimally viable product spec for any CI environment. /smh
I tend to use deployment keys for this. Add the pub
key to the submodule repo; add the pri
key to the main repo as a base64 encoded secret. Tokens are fine but seem like they ave too much access. SSH keys (with: ssh-key:
in the action itself don't work well for me either because I have to use a global one. This is more surgical. Then something like this works:
set -Eeuo pipefail
mkdir -p ~/.ssh
ssh-keyscan -t rsa -H github.com >> ~/.ssh/known_hosts
echo $GH_ACTION_DKEY > ~/.ssh/id_rsa.b64
base64 -d -i ~/.ssh/id_rsa.b64 > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
exit 0
something like this worked for multiple private submodules.
steps:
- uses: actions/checkout@v2
with:
ssh-key: ${{secrets.SSH_KEY}}
- name: Checkout submodules
env:
GIT_SSH_COMMAND: "ssh -o StrictHostKeyChecking=no"
run: |
eval `ssh-agent -s`
echo "${{secrets.SSH_KEY_SUBMODULE1}}" | ssh-add -
git submodule update --init -- src/submodule1
ssh-add -D
echo "${{secrets.SSH_KEY_SUBMODULE2}}" | ssh-add -
git submodule update --init -- src/submodule2
ssh-add -D
eval `ssh-agent -k`
ssh -o StrictHostKeyChecking=no
This is insecure and not recommended.
The secure alternative is ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts
@fearphage true, although ssh-keyscan will accept incorrect key in case of the spoofed server. I don't like the ssh-keyscan solution because on self hosted servers this appends to the same file on each trigger.
One solution to this problem is to use ssh -o StrictHostKeyChecking=accept-new
which has similar effect as ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts
, expect it doesn't append to known_hosts file if a key is already there.
From the security perspective it may be better to ssh-keyscan
once, and save public keys in an action. Something like this:
tmpdir=$(mktemp -d)
echo "${{ secrets.GITHUB_PUBLIC }}" >> $tmpdir/known_hosts
ssh -o UserKnownHostsFile=$tmpdir/known_hosts
Then can remove temporary directory to clean things up.
I don't like the ssh-keyscan solution because on self hosted servers this appends to the same file on each trigger.
That sounds pretty bad in general. I have no experience using self-hosted runners, but the whole point of GitHub Actions (in my mind) is that every run is from a clean room environment. Reusing a dirty machine/image sounds like a security hole in and of itself.
my repository and its submodules don't support SSH ( only HTTPS ). Is there a solution for this protocol to retrieve submodules in GitHub Actions ?
@fairmonk – the trick here is to specify the submodule via a relative path when initially configuring it, that way it works when the parent module is checked out via either scheme. See https://stackoverflow.com/a/44630028/31629 for more info.
@fairmonk – the trick here is to specify the submodule via a relative path when initially configuring it, that way it works when the parent module is checked out via either scheme. See https://stackoverflow.com/a/44630028/31629 for more info.
Using relative URL didn't help here. I ended up using gh, here is a good article on how to do it: https://medium.com/@alexander.sirenko/using-github-access-token-with-submodules-5038b6d639e8
Using a PAT or SSH key means that it needs to be generated for a specific user's account. This is not a viable solution, because if that person leaves the project, and we remove their access to the project, then we need to regenerate the PAT and SSH key for another user. This is the problem we have been having, or am I misunderstanding this and there is a better way?
Using a PAT or SSH key means that it needs to be generated for a specific user's account. This is not a viable solution, because if that person leaves the project, and we remove their access to the project, then we need to regenerate the PAT and SSH key for another user.
Correct, it's not a good idea to make a teammate's GitHub account be the GitHub CI account.
What you do instead is create a new GitHub account just for CI purposes, aka machine user.
The checkout action supports checking out private repos and private submodules.
1) Create a machine user that has read-access to these private repos 2) Create a PAT for this user 3) Save this PAT to your repo's secrets
This is how you checkout private submodules:
- uses: actions/checkout@v2
with:
token: ${{ secrets.PAT }}
submodules: true
This is how you checkout multiple private repos (not as submodules):
- name: Checkout
uses: actions/checkout@v2
with:
path: main
- name: Checkout private tools
uses: actions/checkout@v2
with:
repository: my-org/my-private-tools
token: ${{ secrets.PAT }}
path: my-tools
I've sent in a PR hoping they clarify the docs.
Using a PAT or SSH key means that it needs to be generated for a specific user's account. This is not a viable solution, because if that person leaves the project, and we remove their access to the project, then we need to regenerate the PAT and SSH key for another user.
Correct, it's not a good idea to make a teammate's GitHub account be the GitHub CI account.
What you do instead is create a new GitHub account just for CI purposes, aka machine user.
This also feels like a hacky fix to me. Agree with all the people here that we shouldn't be creating "system" GH users, using up a seat (not to mention these users always get flagged in the creation process for me). It's very odd that this doesn't work as expected and it's been an issue for years.
At the end, PAT (or ssh) is the ONLY solution... So weird that Github allows that for years now... I wonder what's the wall they are hitting...
The most appropriate solution is IMHO using GitHub Apps with https://github.com/marketplace/actions/github-app-token. This allows for appropriate permissions to checkout the main repo and submodules. No "machine" account and such. This should be a built-in option.
Hi, I thought I would chip in with a solution that has been working pretty well for me.
You can use webfactory/ssh-agent action to provide individual deploy keys for multiple submodule repositories like so:
...
steps:
- uses: actions/checkout@v3
- name: Add SSH private keys for submodule repositories
uses: webfactory/ssh-agent@0.7.0
with:
ssh-private-key: |
${{ secrets.SSH_PRIVATE_KEY_SUBMODULE_1 }}
${{ secrets.SSH_PRIVATE_KEY_SUBMODULE_2 }}
- run: git submodule update --init --recursive --remote
...
You can use webfactory/ssh-agent action to provide individual deploy keys for multiple submodule repositories…
Nice to have options, @stepanjakl!
It seems this functionality really should be baked into the core product though and hopefully that will happen.
The most appropriate solution is IMHO using GitHub Apps with https://github.com/marketplace/actions/github-app-token. This allows for appropriate permissions to checkout the main repo and submodules. No "machine" account and such. This should be a built-in option.
Thanks for the link, it was the right starting point to make this work with Github Apps, which indeed seems a fairly clean way to approach this, because it:
The downside is that you need to setup and manage a Github App, and it might not be entirely obvious (to others or your future self) that the permissions for an action are set up through an app (but that's fixable by documentation, I guess).
I made this work with the below workflow file, which I'll share here in case it is useful for others too. The comment at the top has a description of how this works and what must be configured, in the workflow itself, the first two steps are relevant to this approach, the rest is just regular workflow stuff.
# This workflow handles tagged releases.
#
# Whenever a tag is pushed, this produces a clean build, creates
# a github release for the thag and attaches the build results.
#
# Authentication is a bit complicated. Actions automatically get a token
# that can access the repo they run in, but this token cannot access any
# other private repositories within the organization.
# See also https://github.com/actions/checkout/issues/287
#
# To fix this, a Github App is used. These are often publically hosted
# by a third party and users can install the app in their repo or
# organization, giving the app itself access to (selected) info in that
# repo or organization. The server the app runs on must be configured
# with a private key to authenticate to Github, with which it can
# request a short-lived "installation token" for each place where the
# app is installed. This installation token can then be used to talk to
# the normal Github API on behalf of that installation (repo or
# organization).
#
# In this case, instead of running on some server, the "app" runs inside
# a github action, using Github secrets to make the private key
# available to the action. The app (as registered in Github) is also not
# shared between many users, but you register an app just for your own
# organization, giving that app (and thus actions running your
# organization) access to all (or specific) private repositories in the
# organization.
#
# This approach only works for accessing private repositories in the
# same organization as where the workflow runs, and only if the app has
# been granted access to the repo where the workflow runs, since the
# github-app-token step uses the workflow repository to select which
# installation token to request. Cross-organization access could be made
# to work, but needs additional work.
#
# To be able to use this workflow, the following needs to be set up:
#
# - Create a Github App https://docs.github.com/en/developers/apps/building-github-apps/creating-a-github-app
# - Go to organization settings
# - Developer settings -> OAuth apps
# - New Github App
# - Under "Permissions", add "Repository -> Content -> Read-only"
#
# - Create a private key for the app (in the app's settings).
#
# - Create two secrets accessible by this workflow (e.g. organization
# secrets, with appropriate permisions):
# - REPO_READONLY_GITHUB_APP_KEY containing the generated private key
# - REPO_READONLY_GITHUB_APP_ID containing the App ID (shown in the
# app's settings).
#
# - Install the app on the organization
# - In the created app's settings -> Install App
# - Grant (read-only) access to all repositories (or selected repositories if preferred).
name: Release Workflow
on:
push:
tags:
- '*'
jobs:
build_and_publish:
name: Build and publish release
runs-on: ubuntu-latest
steps:
- name: Generate token
# Use the Github App private key to request an installation
# token with read-only access to the organization's private
# repositories. This token is then used in the checkout step
# (but not in subsequent steps that create releases and upload
# assets, those still use the default token that has write
# access to the current repository).
id: generate_token
uses: tibdex/github-app-token@v1
with:
app_id: ${{ secrets.REPO_READONLY_GITHUB_APP_ID }}
private_key: ${{ secrets.REPO_READONLY_GITHUB_APP_KEY }}
# Limit permissions to what we need (these need to be
# configured in the app settings as well).
permissions: >-
{"contents": "read"}
- name: Checkout tag
uses: actions/checkout@v3
with:
submodules: recursive
token: ${{ steps.generate_token.outputs.token }}
- name: build
run: |
make
- name: release
uses: softprops/action-gh-release@v1
with:
files: |
build-result.zip
edit: It seems Github now offers its own action that can replace the tibdex/github-app-token action. I have not tested this, but see https://github.com/actions/checkout/issues/287#issuecomment-1970552586 for an example how to use it (it is pretty much drop-in it seems).
I also realized that you can simply do this using the checkout
action to include multiple submodule deploy keys. I am surprised it hasn't been mentioned here yet.
...
steps:
- uses: actions/checkout@v3
with:
ssh-key: |
${{ secrets.SSH_PRIVATE_KEY_SUBMODULE_1 }}
${{ secrets.SSH_PRIVATE_KEY_SUBMODULE_2 }}
submodules: 'recursive'
...
Just remember the keys must be generated with a link/comment to the repository e.g.
ssh-keygen -t ed25519 -C "git@github.com:owner/repo.git"
The GH checkout can then connect the key with the correct repository.
this may be one of the few things GitLab does much better IMO, but I can appreciate (enough to suggest this be configurable rather than a new default, but not in spirit) the "but that's not secure!" protests, anyway here goes -- GitLab CI pipelines inherit permissions from the user who's action kicked off the pipeline. The ability to at least configure this behavior on a repo level seems like table stakes for GitHub to implement in core, and bikeshedding in this checkout action (read: not core) issue for another 2.5 years doesn't seem like the move.
Instead of using an SSH key (😱 ) you can simply use a personal access token:
- uses: actions/checkout@v2 with: submodules: recursive token: ${{ secrets.ACCESS_TOKEN }}
Here the
ACCESS_TOKEN
variable is a personal access token
Using a PAT in one github action can affect other github actions apparently. For example if you have a tagging/versioning step that commits to the same branch by tagging it, the default GITHUB_TOKEN prevents recursive pipeline triggers. After trying the fixes here that advocate the use of a PAT to download a submodule, in my case, the PAT stayed on for the step that committed the tag. This causes the pipeline to go into recursive builds repeatedly tagging and releasing.
Hi, I thought I would chip in with a solution that has been working pretty well for me.
You can use webfactory/ssh-agent action to provide individual deploy keys for multiple submodule repositories like so:
... steps: - uses: actions/checkout@v3 - name: Add SSH private keys for submodule repositories uses: webfactory/ssh-agent@0.7.0 with: ssh-private-key: | ${{ secrets.SSH_PRIVATE_KEY_SUBMODULE_1 }} ${{ secrets.SSH_PRIVATE_KEY_SUBMODULE_2 }} - run: git submodule update --init --recursive --remote ...
This worked for me, except that --remote
was causing it to checkout the incorrect ref (master of the submodule, not the referenced commit). Just doing git submodule update --init --recursive
got me the desired behavior
I also realized that you can simply do this using the
checkout
action to include multiple submodule deploy keys. I am surprised it hasn't been mentioned here yet.
I've checked that solution first, unfortunately these ssh keys needs to be added as deployment keys to repository that contains submodules and for some reason multiple keys failed to work :(
You approach with ssh-agent config worked!
Is this really required? Are there security reasons why a much better UX can't be provided to developers? The workflow runs with the identity of the user who triggered it, if this person has checkout privileges on submodules can't his token be authorized to checkout submodules?
I can confirm that this method no longer works reliably - it returned the following error in my case:
ERROR: Repository not found.
Error: fatal: Could not read from remote repository.
I agree with @edmondop, utilizing user's (or even organization's) access permissions automatically should be a way to go. Ideally, there could be an option to set action permissions within the GH's UI. As an avid user GH actions, I really appreciate how they simplify repository management. But I believe that simplifying those kind of low-level configurations/processes would be of a great benefit.
I also realized that you can simply do this using the
checkout
action to include multiple submodule deploy keys. I am surprised it hasn't been mentioned here yet.... steps: - uses: actions/checkout@v3 with: ssh-key: | ${{ secrets.SSH_PRIVATE_KEY_SUBMODULE_1 }} ${{ secrets.SSH_PRIVATE_KEY_SUBMODULE_2 }} submodules: 'recursive' ...
Just remember the keys must be generated with a link/comment to the repository e.g.
ssh-keygen -t ed25519 -C "git@github.com:owner/repo.git"
The GH checkout can then connect the key with the correct repository.
After an hour of wasting my time on this, I can confirm this does not work, at least not for self-hosted GitHub Enterprise.
Right now we do not use Github actions because the configuration for private submodules is just too cumbersome. We would also like this to see this working 'out of the box'. We already setup repo access with Github Teams, and then we want to stop thinking about it.
This is really sad story. I want to generate access token just in-memory for GitHub Action, because I'm in the same organisation for got sake! 🤦 https://docs.github.com/en/actions/security-guides/automatic-token-authentication
Please GitHub do something with this.
Using a PAT or SSH key means that it needs to be generated for a specific user's account.
@ScottTylerHall349 You can create repo-specific SSH keys using deploy keys.
@king-of-poppk yeah, but why do you even have to do it? It's in your organisation...
@ScottTylerHall349 You can create repo-specific SSH keys using deploy keys.
You can also create a dedicated service account if you prefer a PAT or in general managing SSH keys there. Sure it's really bad because it means one more seat to pay for on enterprise.
@king-of-poppk yeah, but why do you even have to do it? It's in your organisation...
Access might be more fine-grained than all-or-nothing within an org.
I endorse @matthijskooijman's solution. Using a Github App token is safer and more robust than using somebody's PAT.
Also: now there's an official action to handle GH App tokens. Here's how my workflow ended up:
jobs:
test-submodules:
runs-on: ubuntu-latest
steps:
- name: Get token from Github App
uses: actions/create-github-app-token@v1
id: app_token
with:
app-id: ${{ secrets.APP_ID }}
private-key: ${{ secrets.APP_PEM }}
# owner is required, otherwise the creds will fail the checkout step
owner: ${{ github.repository_owner }}
- name: Checkout from GitHub
uses: actions/checkout@v4
with:
submodules: true
token: ${{ steps.app_token.outputs.token }}
- name: Print .gitmodules
run: cat .gitmodules
In the GH app side:
contents
permission - minimumThank you for the provided workaround, @magmanu! 🙏🏼
GitHub needs to do something to make this easier, though. The below discussions are related and point to a solution where the issued GITHUB_TOKEN
needs to support tweaking on a per-job basis to include permissions up to the level of the user that initiated the run.
Deploy key solution: https://gist.github.com/doutv/54098c2c283ed8141ba961c88a2d5bb0
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Clone main repository
uses: actions/checkout@v4
- name: Add SSH private keys for submodule repositories
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: ${{ secrets.READ_ONLY_DEPLOY_KEY }}
- name: Clone submodules
run: git submodule update --init --recursive --remote
Read ssh-agent Usage section to learn how to:
Also, in .gitmodules
, change the repo url to the ssh format git@
:
[submodule "xxx"]
path = xxx
url = git@github.com:username/xxx
Doing this via ssh-agent is just overkill. It should be possible to do this via token. GitHub please introduce this feature.
Thanks @doutv, using ssh-agent is unfortunate but worked for me.
Simply defining ssh-key
with my deploy key for the submodule failed the fetch altogether, so it seems the only workaround it to split the submodule fetch separately.
+1 for this feature. I'm coming from GitLab where this type of this was easy to achieve.
After aggregating solutions here and reading some documentation I've came up with a clean solution. See webfactory/ssh-agent docs
My use case is the following, in an action in Repo A I need to clone Repo B that has a submodule Repo C. All three are private in my organization.
- name: Setup SSH Agent
uses: webfactory/ssh-agent@v0.9.0
with:
ssh-private-key: |
${{ secrets.REPO_B_DEPLOY_KEY }}
${{ secrets.REPO_C_DEPLOY_KEY }}
- name: Clone Repo B
uses: actions/checkout@v4
with:
ref: master
repository: my-org/Repo-B
submodules: 'true' # This will automatically fetch the submodule Repo C
path: repo_b
This works because webfactory/ssh-agent
creates a git-config setting that redirects git requests to the proper URL and also creates an ssh config that configures the ssh keys for each corresponding repository. This only works if you have the repository url in the comment of the corresponding key.
For example:
ssh-keygen -q -b 4096 -C git@github.com:my-org/Repo-B.git -f REPO_B_DEPLOY_KEY -t rsa
If you generate the key with puttygen export the private key with Conversions->Export OpenSSH key (force new file format)
, this will include the comment with the repo url in the private key. Otherwise the ssh-agent action can't set up the ssh config and this won't work.
I would love to see official solution from GitHub. It's should be based on Action permissions, org config of tmp token.
I would love to see official solution from GitHub. It's should be based on Action permissions, org config of tmp token.
Second this but I think "my" solution is not really hacky or sketchy and also scalable. I can now really fine tune permissions by the use of some ssh keys.
Second this but I think "my" solution is not really hacky or sketchy and also scalable. I can now really fine tune permissions by the use of some ssh keys.
AFAICS the ssh-keys approach does have limitations, in particular you can either use:
Did I get that right?
For completeness, one alternative is to use github app tokens as I previously proposed (original https://github.com/actions/checkout/issues/287#issuecomment-1315458401 and followup https://github.com/actions/checkout/issues/287#issuecomment-1830291196 showing there is now an official GH action to support this flow). Possibly a bit more hassle to set up, but possibly a bit easier to manage than per-repository deploy SSH-keys (or maybe not - haven't tried that solution).
@matthijskooijman What I've done is I created a deploy key for each of the two repositories in my scenario and added the private keys for that to the secrets in the repo that the action runs in.
After that I can simply use ${{ secrets.REPO_B_DEPLOY_KEY }}
this private key without any scripting.
I guess if you have a ton of repositories this gets pretty cumbersome.
I haven't tried the GH App approach, I guess this one could be useful if you have many repositories where you need to do this.
Currently the checkout action doesn't work with private repositories using a private submodule.
As a work-around we use the following in our workflow.
It would be good if the checkout action would support some option to provide a different SSH_KEY for private submodules. E.g.
SUBMODULE_SSH_KEY
could be an organisation level SSH Key that allows pulling the repos.