Unify gh actions and permissions

[drjova]

In inspire we have 2 types of packages services and libraries, we should unify the gh-actions and permissions to follow the following

• build (optional only for services)
• lint -> it should run pre-commit repo configuration
• tests
• pull-request-main
• push-main

Additionally, all libraries should use the bump2version python package to bump the version after a push to master. Some libraries still use autosemver, which should be removed!

More specifically the pull-request-main should look something like:

Services

name: Pull request main

on:
  pull_request_target:
    branches: [main]

jobs:
  lint:
    uses: ./.github/workflows/lint.yml
    with:
      ref: ${{ github.ref }}
  test:
    uses: ./.github/workflows/test.yml
    with:
      ref: ${{ github.event.pull_request.head.sha }}
    secrets: inherit

name: Push main

on:
  push:
    branches: [main]

defaults:
  run:
    shell: bash

jobs:
  lint:
    uses: ./.github/workflows/lint.yml
    with:
      ref: ${{ github.ref }}
  test:
    uses: ./.github/workflows/test.yml
    with:
      ref: ${{ github.ref }}
    secrets: inherit
  deploy:
    needs: test
    runs-on: ubuntu-latest
    steps:
      - name: send event
        uses: cern-sis/gh-workflows/.github/actions/kubernetes-project-new-images@v6.2.0
        with:
          event-type: update
          images: |
            cern-sis/inspire/backoffice@${{ needs.test.outputs.image-id }}
          token: ${{ secrets.PAT_FIRE_EVENTS_ON_CERN_SIS_KUBERNETES }}

Packages

name: Pull request main

on:
  pull_request_target:
    branches: [main]

jobs:
  lint:
    uses: ./.github/workflows/lint.yml
    with:
      ref: ${{ github.ref }}
  test:
    uses: ./.github/workflows/test.yml
    with:
      ref: ${{ github.event.pull_request.head.sha }}
    secrets: inherit

name: Push master

on:
  push:
    branches:
      - master

jobs:

  lint:
    uses: ./.github/workflows/lint.yml
    with:
      ref: ${{ github.ref }}

  tests:
    uses: ./.github/workflows/tests.yml

  bump_version:
    needs: [tests]
    uses: ./.github/workflows/bump.yml

  publish_package:
    needs: bump_version
    uses: ./.github/workflows/build-and-publish.yml

Package list

• inspire-utils
• inspire-schemas
• inspire-dojson
• inspire-json-merger
• inspire-matcher
• inspire-query-parser

Services

• inspirehep
• inspire-classifier
• workflows
• backoffice
• refextract

[PascalEgn]

Unify GH actions

• None of the GH actions should depend on the usage of secrets for the pull request triggered actions, secrets should only be needed for actions that trigger on push to master
• We should remove the run-tests.sh scripts in all repos

Packages:

Pull request master/main:

Notes:

• on pull_request --> runs on the PR repo scope, e.g. no access to secrets and changes to the gh actions on the PR can run without merging first (requires a more careful look on changes!)

Linting:

• Should be triggered from pull_request (as no access to secrets is needed)
• Just run pre-commit configuration of repo

Tests:

• Should be triggered from pull_request (as no access to secrets is needed)
• Should trigger tests running on Python 2.7 (for inspire-next) and Python 3.11 (for inspirehep) as well as the newest Python version (if this fails the pipeline should not break, as its just to test if the package would be ready for an upgrade)

Setup:

name: Pull request master

on:
  pull_request:
    branches:
      - master

jobs:
  pre_commit:
    uses: ./.github/workflows/pre-commit.yml

  python2_tests:
    needs: [pre_commit]
    uses: ./.github/workflows/test-python-2.yml

  python3_tests:
    needs: [pre_commit]
    uses: ./.github/workflows/test-python-3.yml

Push master/main:

Notes:

• Do we need to run the pre-commit again as we already did for the PR?
• We should remove the use of autosemver and only use the bump action for version bumps

Setup:

name: Push master

on:
  push:
    branches:
      - master

jobs:
  pre_commit:
    uses: ./.github/workflows/pre-commit.yml

  python2_tests:
    needs: [pre_commit]
    uses: ./.github/workflows/test-python-2.yml

  python3_tests:
    needs: [pre_commit]
    uses: ./.github/workflows/test-python-3.yml

  bump_version:
    needs: [pre_commit, python2_tests, python3_tests]
    uses: ./.github/workflows/bump-and-publish.yml
    secrets: inherit

name: Bump package version and publish to pypi

on:
  workflow_call:

jobs:
  bump:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          ref: ${{ github.head_ref }}
          token: ${{ secrets.INSPIRE_BOT_TOKEN }}
          fetch-depth: 0

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Configure git
        shell: bash
        run: |
          git config user.name github-actions
          git config user.email "38065702145+github-actions[bot]@users.noreply.github.com"

      - name: Install bump2version
        shell: bash
        run: |
          python -m pip install --upgrade pip
          pip install bump2version

      - name: Bump version
        shell: bash
        run: |
          bump2version patch --tag --verbose

      - name: Push changes
        uses: ad-m/github-push-action@v0.8.0
        with:
          branch: ${{ github.ref }}
          github_token: ${{ secrets.INSPIRE_BOT_TOKEN }}
          tags: true

  publish:
    needs: bump
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
          fetch-tags: true

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - name: Install python dependencies # TODO needs to be adjusted individually
        run: |
          python -m pip install --upgrade pip
          pip install setuptools wheel
          pip install -e .[tests]

      - name: Show python dependencies
        run: |
          python3 --version
          pip freeze

      - name: Build package
        run: |
          python setup.py sdist bdist_wheel

      - name: Publish package
        uses: pypa/gh-action-pypi-publish@v1.10.1
        with:
          user: __token__
          password: ${{ secrets.pypi_password }}

General

name: Run pre-commit

on:
  workflow_call:

jobs:
  pre-commit:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout Code
        uses: actions/checkout@v4
    - name: Setup Python
        uses: actions/setup-python@v5
        with:
       python-version: '3.11'    
    - name: Run pre-commit
        uses: pre-commit/action@v3.0.1

name: Test Python 2

on:
  workflow_call:

jobs:
  test:
    runs-on: ubuntu-20.04
    strategy:
      matrix:
        include:
          - python: python2
            pip: pip
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          python-version: ${{ matrix.python-version }}
          fetch-depth: 0

      - name: Install python dependencies # TODO needs to be adjusted individually
        run: |
          wget https://bootstrap.pypa.io/pip/2.7/get-pip.py
          ${{ matrix.python }} get-pip.py
          ${{ matrix.python }} -m ${{ matrix.pip }} install --user --upgrade pip
          ${{ matrix.python }} -m ${{ matrix.pip }} --no-cache-dir install --user setuptools wheel "urllib3==1.25.11"
          ${{ matrix.python }} -m ${{ matrix.pip }} --no-cache-dir install --user -e .[tests]

      - name: Show python dependencies
        run: |
          ${{ matrix.python }} --version
          ${{ matrix.pip }} freeze

      - name: Run tests
        run: |
            py.test tests

name: Test Python 3

on:
  workflow_call:

jobs:
  test:
    runs-on: ubuntu-latest
    continue-on-error: ${{ matrix.experimental }}
    strategy:
      fail-fast: true
      matrix:
        python-version: "3.11"
        experimental: [false]
        include:
          - python-version: "3.x"
            check-latest: true
            experimental: true

    steps:
    - name: Checkout repository
      uses: actions/checkout@v4

    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v5
      with:
        python-version: ${{ matrix.python-version }}

    - name: Install dependencies # TODO needs to be adjusted individually
      run: |
        python -m pip install --upgrade pip
        pip install -e .[tests]

    - name: Show python dependencies
      run: |
        python3 --version
        pip freeze

    - name: Run tests
      run: |
          py.test tests