Open gthomson31 opened 3 months ago
cooperspencer
commented
3 months ago Hi, I am not that well versed in Github Actions but did you set the permissions so that the container can create files? Something like:
permissions:
contents: writeThat's the only thing that comes to mind right now.
gthomson31
commented
3 months ago Just tested out but unfortunately the same issue
The setting below refers to the default GITHUB_TOKEN that is set when deploying the workflow - I am pulling a repo specific token out from Github secrets and patching this into the config. ( Tried to also unset this token during workflow as a precaution)
permissions:
contents: write I believe the issue stems from a conflict between the GitHub runner's default image and the gickup configuration file. because the token is correctly configured, evident from its ability to fetch public/private repositories without any problems.
Will do some investigation
gthomson31
commented
3 months ago Dropping our config
source:
github:
- token: "TOKEN_IS_PATCHED"
wiki: true # includes wiki too
issues: true # back up issues, works only locally
filter:
excludeforks: true # exclude forked repositories
destination:
local:
# Export this path from Docker with a volume to make it accessible and more permanent.
- path: "PATH_IS_PATCHED"
structured: false # checks repos out like hostersite/user|organization/repo
zip: true # zips the repository after cloned and removes the repository afterwards
lfs: true # clone lfs repos, !! ATTENTION !! git and git-lfs must be installed on the system!Would you mind sharing the relevant code for the action?
gthomson31
commented
3 months ago Added the workflow and breakdown below
workflow_dispatchpush events to the main branchid-token: write: Necessary for JWT requestscontents: read: Needed for the actions/checkout to access repository contentsubuntu-latestGICKUP_VERSION set to 0.10.28actions/checkout@v4setup-tools jobubuntu-latestexample-orgactions/checkout@v4name: Deploy and Run Github Backups
on:
workflow_dispatch:
push:
branches:
- main
permissions:
id-token: write # This is required for requesting the JWT
contents: read # This is required for actions/checkout
jobs:
setup-tools:
runs-on: ubuntu-latest
outputs:
date: ${{ steps.date.outputs.RUN_DATE }}
env:
GICKUP_VERSION: 0.10.28
steps:
- name: Checkout code
uses: actions/checkout@v4
# Install Gickup and cache it
- name: Cache Gickup
id: cache-gickup
uses: actions/cache@v4
with:
path: ~/gickup
key: gickup-${{ env.GICKUP_VERSION }}
- name: Download and Extract Gickup if not cached
if: steps.cache-gickup.outputs.cache-hit != 'true'
run: |
mkdir -p ~/gickup
curl -L https://github.com/cooperspencer/gickup/releases/download/v${{ env.GICKUP_VERSION }}/gickup_${{ env.GICKUP_VERSION }}_linux_amd64.tar.gz -o ~/gickup/gickup.tar.gz
tar -xzvf ~/gickup/gickup.tar.gz -C ~/gickup
chmod +x ~/gickup/gickup
- name: Get Current date
id: date
run: echo "RUN_DATE=$(date +'%Y-%m-%d-%H-%M')" >> $GITHUB_OUTPUT
create-backup-and-upload:
needs: setup-tools
name: Backup and Upload for ${{ matrix.org }}
runs-on: ubuntu-latest
strategy:
# This will create a job for each org in the matrix
matrix:
org:
- example-org
env:
# This is the path to the backup config file for the org
BACKUP_CONFIG_FILE: .github/config/github_backups/configs/${{ matrix.org }}.yml
steps:
- name: Restore Gickup Cache
id: cache-gickup
uses: actions/cache@v4
with:
path: ~/gickup
key: gickup-${{ env.GICKUP_VERSION }}
- name: Checkout code
uses: actions/checkout@v4
# Add Gickup to PATH to make it available
- name: Add Gickup to PATH
run: echo "$HOME/gickup" >> $GITHUB_PATH
# REDACTED Setting the secret path for the org name and Patching the config file with token and path
# Run the backup
- name: Run Gickup Backup
working-directory:
run: |
gickup ${{env.BACKUP_CONFIG_FILE}}
Testing running Github API call directly in the workflow against a private repo, passing in the same token used in Gickup Config
- name: "Download using API"
run: |
echo "Downloading the repository"
curl -L \
-H "Accept: application/vnd.github+json" \
-H "Authorization: Bearer ${{ secrets[env.GH_BACKUP_TOKEN] }} " \
-H "X-GitHub-Api-Version: 2022-11-28" \
https://api.github.com/repos/${{matrix.org}}/testing-private/zipball/main > testing-private.zip
- name: "Unzip the repository and list the files"
run: |
echo "Unzipping the repository"
unzip testing-private.zip -d testing-private
folder=$(ls testing-private | grep <ORG_NAME>)
ls testing-private/$folderRun echo "Unzipping the repository"
Unzipping the repository
Archive: testing-private.zip
<REDACTED LOGS>
README.md
TEST_FILE_GICKUPConnecting into the github runner ubuntu-latest Image to run the gickup commands manually and see the log and have found the error below:
When running through it prompts for Github Username instead of passing in the token.
Using v0.10.28 - linux_amd64
2024-04-15T14:04:28Z INF starting backup for https://github.com/<org_name>/testing-private.git stage=backup
2024-04-15T14:04:28Z INF cloning testing-private path=<org_name>-2024-04-15-14-00-17 stage=locally
Username for 'https://github.com':hm... If you use git lfs it defaults to the git command and doesn't use the go-git module.
It seems like something goes wrong in this part:
https://github.com/cooperspencer/gickup/blob/main/local/local.go#L402
It seems like it doesn't add the token to the URL.
gthomson31
commented
3 months ago How strange !
I don't actually think we needed to use LFS so have disabled from our config and works fine now
cooperspencer
commented
3 months ago I am glad to hear that it worked out. I'll try to investigate what's going on.
Hi there
Wondering if anyone has seen a similar issue we are currently deploying the gickup backups to run via a Github actions workflow replicating the same process as running locally.
When running locally all repos both public and private backup without issue but when running within the actions workflow it is unable to clone private repos using the same token.
Local Run
Github Actions Log