actions-template-sync

abstract

Synchronise git repositories in an automated manner. Different git providers like GitHub (enterprise), GitLab,.. are supported as the source provider. This can help you e.g. for migration from another git provider to GitHub or if you want to mirror git repositories.

History

It is possible to create repositories within Github with GitHub templates. This is a nice approach to have some boilerplate within your repository. Over time, the template repository will get some code changes. The problem is that the already created repositories won't know about those changes. This GitHub action will help you to keep track of the template changes. The initial author of this repository faced that issue several times and decided to write a GitHub action to face that issue. Because of the nice community, several feature requests helped to go on with the development of the action. Now several other features are supported.

Features

This action is creating a pull request with the latest changes within the target repo whenever it runs with following exceptions

• there is already an open PR created with the latest changes of the source repository.
  • if there are new changes and a PR is already open, a new PR will be created (option to clean up older PRs)
• related new changes are ignored within the .templatesyncignore file
• the source repository is fully included within the target repository

flowchart LR
    github_source("fa:fa-github <b>GitHub</b> source repository <b>[private|public]</b>")
    gitlab_source("fa:fa-gitlab <b>GitLab</b> source repository <b>[private|public]</b>")
    any_source("fa:fa-git <b>Any</b> git provider <b>[private|public]</b>")
    github_target{{"fa:fa-github <b>GitHub</b> target repository <b>[private|public]</b>"}}
    github_source --> |"<b>ssh | PAT | github app</b>"| github_target
    gitlab_source --> |"<b>ssh</b>"| github_target
    any_source --> |"<b>ssh</b>"| github_target

• Sync other public or private repository (e.g. template repositories) with the current repository
• Ignore files and folders from syncing using a .templatesyncignore file
• many configuration options
• different lifecycle hooks are supported. This opens the possibility to inject custom code into the workflow with a yaml definition file.
• different git provider like GitLab, Gittea,.. as source are supported (with ssh). See .github/workflows/test_ssh_gitlab.yml for an example.
• It is not necessarily needed that source and target repository have the same base history. Because of that reason, it is possible to merge 2 totally different repositories with the help of the action.

Usage

Usage changes depending on whether the template repository is public or private, regardless of the visibility of the current repository.

Public template repository

Add this configuration to a GitHub action in the current repository:

# File: .github/workflows/template-sync.yml

on:
  # cronjob trigger
  schedule:
  - cron: "0 0 1 * *"
  # manual trigger
  workflow_dispatch:
jobs:
  repo-sync:
    runs-on: ubuntu-latest
    # https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs
    permissions:
      contents: write
      pull-requests: write

    steps:
      # To use this repository's private action, you must check out the repository
      - name: Checkout
        uses: actions/checkout@v4
        # https://github.com/actions/checkout#usage
        # uncomment if you use submodules within the repository
        # with:
        #   submodules: true

      - name: actions-template-sync
        uses: AndreasAugustin/actions-template-sync@v2
        with:
          source_repo_path: <owner/repo>
          upstream_branch: <target_branch> # defaults to main
          pr_labels: <label1>,<label2>[,...] # defaults to template_sync

You will receive a pull request within your repository if there are some changes available in the template.

Private template repository

If your current repository was created from a private template, there are several possibilities.

1. Using a GitHub app

You can create and use a GitHub App to handle access to the private template repository. To generate a token for your app you can use a separate action like tibdex/github-app-token. You have to set up the checkout step with the generated token as well.

jobs:
  repo-sync:
    runs-on: ubuntu-latest

    steps:
      - name: Generate token to read from source repo # see: https://github.com/tibdex/github-app-token
        id: generate_token
        # https://github.com/tibdex/github-app-token
        uses: tibdex/github-app-token@v2
        with:
          app_id: ${{ secrets.APP_ID }}
          private_key: ${{ secrets.PRIVATE_KEY }}

      - name: Checkout
        # https://github.com/actions/checkout#usage
        uses: actions/checkout@v4
        with:
          # submodules: true
          token: ${{ steps.generate_token.outputs.token }}

      - name: actions-template-sync
        uses: AndreasAugustin/actions-template-sync@v2
        with:
          github_token: ${{ steps.generate_token.outputs.token }}
          source_repo_path: <owner/repo>
          upstream_branch: <target_branch> # defaults to main
          pr_labels: <label1>,<label2>[,...] # defaults to template_sync

2. Using SSH

You have various options to use ssh keys with GitHub. An example is deployment keys. For our use case, write permissions are not needed. Within the current repository, where the GitHub action is enabled, add a secret (e.q. SOURCE_REPO_SSH_PRIVATE_KEY) with the content of your private SSH key. Make sure that the read permissions of that secret fulfill your use case. Set the optional source_repo_ssh_private_key input parameter. It is also possible to use a different git provider, e.g. GitLab.

jobs:
  repo-sync:
    runs-on: ubuntu-latest
    # https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs
    permissions:
      contents: write
      pull-requests: write

    steps:
      # To use this repository's private action, you must check out the repository
      - name: Checkout
        # https://github.com/actions/checkout#usage
        uses: actions/checkout@v4
        with:
          # submodules: true
          token: ${{ secrets.GITHUB_TOKEN }}

      - name: actions-template-sync
        uses: AndreasAugustin/actions-template-sync@v2
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          source_repo_path: ${{ secrets.SOURCE_REPO_PATH }} # <owner/repo>, should be within secrets
          upstream_branch: ${{ secrets.TARGET_BRANCH }} #<target_branch> # defaults to main
          pr_labels: <label1>,<label2>[,...] # defaults to template_sync
          source_repo_ssh_private_key: ${{ secrets.SOURCE_REPO_SSH_PRIVATE_KEY }} # contains the private ssh key of the private repository

3. Using a PAT

:warning: when the source repository is private using PATs, also the target repository must be private. Else it won't work.

Personal access token is an alternative to using passwords for authentication to GitHub. You can add a kind of password to your GitHub account. You need to set the scopes.

• repo -> all
• read:org

Furthermore, you need to set the access within the source repository to allow GitHub actions within the target repository. As mentioned before (you can see the note in the image) you need to set the target repository to private. settings -> actions -> general.

example workflow definition

name: actions-template-sync

on:
  # cronjob trigger At 00:00 on day-of-month 1. https://crontab.guru/every-month
  schedule:
  - cron: "0 0 1 * *"
  # manual trigger
  workflow_dispatch:

jobs:
  test-implementation-job:

    runs-on: ubuntu-latest

    steps:
      # To use this repository's private action, you must check out the repository
      - name: Checkout
        uses: actions/checkout@v4
        with:
          # submodules: true
          token: ${{ secrets.CUSTOM_GITHUB_PAT }}

      - name: Test action step PAT
        uses: AndreasAugustin/actions-template-sync@v2
        with:
          github_token: ${{ secrets.CUSTOM_GITHUB_PAT }}
          source_repo_path: ${{ secrets.SOURCE_REPO_PATH }} # <owner/repo>, should be within secrets

Action Inputs

Action Outputs

Properties that are available after the action executed.

Remarks Please consider following edge cases

• pr_branch
  • If PR branch already exists (e.g. after a 2nd run) the action won't update the branch but will still output the branch name
  • If the remote repository already contains the source repository changes the action will exit and the output variable will be undefined
  • If there are no changes the action will exit and the output variable will be undefined

Docker

There are docker images available. Please checkout How to use docker for details.

• dockerhub andyaugustin/actions-template-sync
• github andreasaugustin/actions-template-sync

Example

This repo uses this template and this action from the marketplace. See the definition here.

If you look for a more detailed guide you can have a look at

• Dev.to
• GitHub
• :heart: nice blog post

Trigger

You can use all triggers which are supported for GitHub actions

Ignore Files

Create a .templatesyncignore file. Just like writing a .gitignore file, follow the glob pattern in defining the files and folders that should be excluded from syncing with the template repository.

It can also be stored inside .github folder.

The template_sync_ignore_file_path parameter allows you to specify a path to an ignore file. This variable defaults to .templatesyncignore. Changing this allows you to support template sync with more than one repository using different ignore files.

The action will look for the path specified within . or .github/

Note: It is not possible to sync also the .templatesyncignore itself. Any changes from the template repository will be restored automatically.

Remark reading the gitglossary (pathspec section) you see a slight difference to the .gitignore file when you like to disable files you need to use :!. E.g. when you like to disable the sync for all files with exceptions, you need to do smth like

:!newfile-1.txt
*

Force Push and PR

If you set the input is_force_push_pr to true you are able to react to e.g. metadata changes within the workflow definition file. Please note that you need to add permissions for repository-projects: read. Compare the needed scope with gh pr edit

  permissions:
    contents: write
    pull-requests: write
    repository-projects: read

Sign commits

It is recommended to sign your commits. This action is able to sign commits.

First, generate a GPG key and export the GPG private key as an ASCII armored version to your clipboard:

# macOS
gpg --armor --export-secret-key jon@doe.example | pbcopy

# Ubuntu (assuming GNU base64)
gpg --armor --export-secret-key jon@doe.example -w0 | xclip

# Arch
gpg --armor --export-secret-key jon@doe.example | xclip -selection clipboard -i

# FreeBSD (assuming BSD base64)
gpg --armor --export-secret-key jon@doe.example | xclip

:warning: the gpg username and email must match the git_user_name and git_user_email parameters. Paste your clipboard as a secret named GPG_PRIVATE_KEY for example. If your key has a password, create another secret named GPG_PASSPHRASE.

# File: .github/workflows/template-sync.yml

on:
  # cronjob trigger
  schedule:
  - cron: "0 0 1 * *"
  # manual trigger
  workflow_dispatch:
jobs:
  repo-sync:
    runs-on: ubuntu-latest
    # https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs
    permissions:
      contents: write
      pull-requests: write

    steps:
      # To use this repository's private action, you must check out the repository
      - name: Checkout
        uses: actions/checkout@v4

      - name: actions-template-sync
        uses: AndreasAugustin/actions-template-sync@v2
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          source_repo_path: <owner/repo>
          git_user_name: # add the gpg username
          git_user_email: # add the gpg email
          gpg_private_key: ${{ secrets.GPG_PRIVATE_KEY }}
          # uncomment if your key has a passphrase
          # gpg_passphrase: ${{ secrets.GPG_PASSPHRASE }}

Lifecycle actions

The action has different phases which are executed in the following order

• preparation prepare and configure git related things
  • init git
  • auth related (ssh or github auth)
  • [optional] gpg setup
• prechecks run some prechecks
  • skipped if is_force_push_pr parameter is set to true
  • check if the sync branch is already existing in target repository
  • check if new changes of the source repository are already within history
• pull pull the changes from the remote repository into the action runtime
• commit commit the changes within the action runtime
• push
  • if is_force_push_pr is set to true then a force push will be executed
• pr
  • eventual create registered labels (:ninja: emojis are supported)
  • create a new PR
  • if is_force_push_pr is set to true then the PR will be created or edited
  • [optional] cleanup eventual cleanup older PRs of the action
• set github action outputs

If is_dry_run parameter is set to true then all stages modifying the github state are not run (e.g. push, cleanup and pr).

It is possible to run a subset of the mentioned lifecycle actions. preparation and github action outputs will be run every time.

:warning: Advanced feature. Use with care (possibly set is_dry_run: true configuration parameter for testing purposes)

e.g.

# File: .github/workflows/test_steps.yml

on:
  # cronjob trigger
  schedule:
  - cron: "0 0 1 * *"
  # manual trigger
  workflow_dispatch:
jobs:
  repo-sync:
    runs-on: ubuntu-latest
    # https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs
    permissions:
      contents: write
      pull-requests: write

    steps:
      # To use this repository's private action, you must check out the repository
      - name: Checkout
        uses: actions/checkout@v4

      - name: actions-template-sync first steps
        uses: AndreasAugustin/actions-template-sync@v2
        with:
          source_repo_path: <owner/repo>
          steps: "prechecks,pull"  # order matters

      - name: in between step
        run: |
          echo "I can do whatever I want"
          git status

      - name: actions-template-sync next steps
        uses: AndreasAugustin/actions-template-sync@v2
        with:
          source_repo_path: <owner/repo>
          steps: "commit,push,pr"  # order matters

Lifecycle hooks

Different lifecycle hooks are supported. You need to enable the functionality with the option is_allow_hooks and set it to true :warning: use this functionality with caution. You can use one of the available docker images to test it out. With great power comes great responsibility.

In addition, you need either a configuration file with the name templatesync.yml within the root of the target repository or you set the hooks input parameter within the action definition with a related yaml string

The following hooks are supported (please check docs/ARCHITECTURE.md for a better understanding of the lifecycles).

• prepull is executed before the code is pulled from the source repository
• precommit is executed before the code is commited
• prepush is executed before the push is executed, right after the commit
• precleanup is executed before older PRs targeting the same branch are closed
• prepr is executed before the PR is done

Remark If you need to install aditional tools just install them in an additional step upfront the action invokation. If using the docker image the underlying OS is defined by an Alpine container.

Example for the hooks input parameter

- name: Test action step
  uses: AndreasAugustin/actions-template-sync@v2
  env:
    MY_VAR: "foo"  # possible to define envrionment variables
  with:
    source_repo_path: AndreasAugustin/template.git
    upstream_branch: main
    is_dry_run: true
    is_allow_hooks: true
    hooks: >
      prepull:
        commands:
          - echo 'hi, we are within the prepull phase'
          - echo 'maybe you want to do adjustments on the local code'

Schema and example for the templatesync.yml

Remark It is possible to use environment variables within the github action definition usable within the command configuration, e.g.

- name: Test action step
  uses: AndreasAugustin/actions-template-sync@v2
  with:
    source_repo_path: AndreasAugustin/template.git
    upstream_branch: main
    is_dry_run: true
    is_allow_hooks: true

Please not the double quotes within the following prepull echo command

hooks:
  prepull:
    commands:
      - echo "hi, we are within the prepull phase ${MY_VAR}"
      - echo 'maybe you want to do adjustments on the local code'
  precommit:
    commands:
      - echo 'hi, we are within the precommit phase'
      - echo 'maybe you want to add further changes before the code is committed'
  prepush:
    commands:
      - echo 'hi, we are within the prepush phase'
      - echo 'maybe you want to add further changes and commits'
  precleanup:
    commands:
      - echo 'hi, we are within the precleanup phase'
      - echo 'maybe you want to interact with older PRs before they are closed'
  prepr:
    commands:
      - echo 'hi, we are within the prepr phase'
      - echo 'maybe you want to change the code a bit and do another push before creating the pr'

Labels creation

By default, generated PRs will be labeled with the template_sync label. If that label doesn't exist in your repository, it will be created automatically unless you specify your own existing labels. Associating a label with the generated PRs helps keeping track of them and allows for features like automatic PR cleanup.

Pull request cleanup

Depending on your way of working, you may end up with multiple pull requests related to template syncing pointing to the same branch. If you want to avoid this situation, you can instruct this action to clean up older PRs (search based on labels defined with the pr_labels config parameter).

:warning: this feature will close all pull requests with labels configured with pr_labels config parameter.

Force deletion

This feature will force delete files if those are deelted within the source repository.

:warning: it is highly related to the git_remote_pull_params config parameter and won't work with the default. You need to change the default one e.g. to git_remote_pull_params: --allow-unrelated-histories --strategy=recursive --no-edit.

GHES and custom runners

Some notes if you use GitHub Enterprise Server (GHES) and/or custom runners. The action script is based on bash. That means your runner must be able to run bash scripts. Furthermore you need to have the following command line tools installed:

• ssh
• GitHub cli
• git
• optional (dependent the features you are using)
  • git lfs if you are using the lfs functionality
  • yq if you are using the hook functionality
  • gpg if you are using the git signing functionality

Furthermore most likely you have a custom domain name. Therefore you should configure the hostname GitHub action parameter.

Remark

:whale: There is also a docker image available which has all needed tools installed. This is helpful e.g. if you are not able to use a remote action. The idea is to use the docker action

• GitHub registry
• Dockerhub registry

Troubleshooting

• refusing to allow a GitHub App to create or update workflow .github/workflows/******.yml without workflows permission

  This happens because the template repository is trying to overwrite some files inside .github/workflows/.

  Currently GITHUB_TOKEN can't be given workflow permission. You can grant our workflow with workflows permission using a PAT following the steps below:

  1. Create a PAT with these repository permissions granted: contents:write, workflows:write, metadata:read.

  2. Copy the generated token and create a new secret for your target repository.

  3. Configure the checkout action to use the token in secrets like this:

     # File: .github/workflows/template-sync.yml
     
     on:
     # cronjob trigger
     schedule:
     - cron: "0 0 1 * *"
     # manual trigger
     workflow_dispatch:
     jobs:
     repo-sync:
      runs-on: ubuntu-latest
      # https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs
      permissions:
        contents: write
        pull-requests: write
     
      steps:
        # To use this repository's private action, you must check out the repository
        - name: Checkout
          uses: actions/checkout@v4
          with:
            # submodules: true
            token: ${{ secrets.<secret_name> }}
     
        - name: actions-template-sync
          uses: AndreasAugustin/actions-template-sync@v2
          with:
            github_token: ${{ secrets.GITHUB_TOKEN }}
            source_repo_path: <owner/repo>
            upstream_branch: <target_branch> # defaults to main
            pr_labels: <label1>,<label2>[,...] # optional, no default

• pull request create failed: GraphQL: GitHub Actions is not permitted to create or approve pull requests (createPullRequest)

  Open your project Settings > Actions > General and select the checkbox Allow GitHub Actions to create and approve pull requests under the Workflow permissions section.

Release update notes

• v2
  • git lfs is no default anymore. Enable with is_git_lfs parameter.
  • infrastructure change: now using composite action instead of docker action to be more flexible to combine more actions (file system permissions).
  • local git config now instead of global git config --global in respect to be more flexible in chaining actions.
• :warning: starting with version v1 (v1.0.0) the upstream_branch variable default is not main anymore. It is now set to the remote default branch.
• starting with version v0.5.2-draft the templateversionrc file is not needed anymore. You can delete that file from the target repositories.

Debug

You must create a secret named ACTIONS_STEP_DEBUG with the value true to see the debug messages set by this command in the log. For more information, see "Enabling debug logging."

Comparison with other tools

There are other great tools available within GitHub. Here you can find a comparison.

DEV

The development environment targets are located in the Makefile

make help

:ninja: contributiong of any kind are welcome. Please checkout the contributing guidelines.

For some architectural notes please have a look at the docs

Contributors ✨

Thanks goes to these wonderful people (emoji key):

andy Augustin 📖 💻 👀 🛡️ 🤔 💬 💡 🖋 📝 🚧 🚇 📦 ⚠️	Ugo Pattacini 📖	Jose Gabrielle Rivera 💻	P.D. Rittenhouse 🤔	Daniel Boll 🐛	albertschwarzkopf 🤔	Akul Pillai 🛡️
Stefan Riembauer 🤔	Fabrizio Cacicia 🛡️ 🐛	Justin Tunis 🤔 💻 🐛	Michael Matos 🐛	Gavin Williams 🤔	Marc Siebeneicher 🤔 💻 🐛 📖	Luís Henrique A. Schünemann 🤔 📖 💻
George 💬 📖 🤔	Pedro Rivero 🤔	Eleanor Bronson 🤔	Marvin Osswald 📖	David Calvert 📖 🐛 💻 🚧 📝	Andy Airey 🐛 👀	Surya Asriadie 🐛
jellllly420 🤔 💬 📖	Shaun Tabone 💻	Kevin AUDE 🤔 💻	Jakob 👀	Kevin Deldycke 🐛 🤔 💻	Jessica Scheick 🐛	Gaspar Melsion 🤔
Ken Harding 💻 🐛	Jakob Drachmann Havtorn 🤔	Brian 📖	MuriloChianfa 📖	David Snyder 🔬	Jonathan Østrup 🤔 💻	Nat Welch 🐛 💻

This project follows the all-contributors specification. Contributions of any kind are welcome!