derberg / manage-files-in-multiple-repositories

GitHub Action that introduces support for global workflows. Global workflows are the one you update in just one repo and they are automatically updated in other repositories.
MIT License
51 stars 18 forks source link

API limit #40

Closed ReenigneArcher closed 1 year ago

ReenigneArcher commented 2 years ago

Describe the bug

A clear and concise description of what the bug is.

Sometimes when running this action, I get rate limited. I already have set the matrix to only 2 parallel jobs. I will switch to 1, but it would be nice if the action had a retry option to try again after a certain number of seconds or something.

Warning: Failed replicating files for this repo: Error: Unable to create a PR: GraphqlResponseError: Request failed due to following response errors:
 - was submitted too quickly

How to Reproduce

Steps to reproduce the issue. Attach all resources that can help us understand the issue:

name: Global replicator

on:
  push:
    branches: [master]
    paths:
      - ".github/**"
      - "dependabot/**"
      - "gitignore/**"
  workflow_dispatch:
    inputs:
      repo_name:
        description: |
          You can specify the repository's name where workflows should be pushed manually, as long as workflow settings do not ignore the repository.
          If you do not specify the exact repository name, the workflow will try to replicate all missing changes to all repositories.
        required: false

jobs:
  replicate:
    runs-on: ubuntu-latest

    strategy:
      fail-fast: false  # false to run all, true to fail entire job if any fail
      max-parallel: 2  # jobs fail submitting PRs if run too fast
      matrix:
        include:
        - job_name: 'replicate workflows'
          patterns_to_ignore: ''
          patterns_to_include: '.github/label-actions.yml,.github/pr_release_template.md,.github/workflows/auto-create-pr.yml,.github/workflows/automerge.yml,.github/workflows/autoupdate.yml,.github/workflows/issues.yml,.github/workflows/issues-stale.yml,.github/workflows/pull-requests.yml'
          commit_message: 'ci: update global workflows'
          repos_to_ignore: ''
          topics_to_include: ''
          exclude_private: false
          exclude_forked: true
          branches: '^nightly$'
          destination: ''
        - job_name: 'replicate dependabot'
          patterns_to_ignore: ''
          patterns_to_include: 'dependabot/dependabot.yml'
          commit_message: 'ci: update dependabot'
          repos_to_ignore: ''
          topics_to_include: ''
          exclude_private: false
          exclude_forked: true
          branches: '^nightly$'
          destination: '.github'
        - job_name: 'replicate python'
          patterns_to_ignore: ''
          patterns_to_include: '.flake8,.github/workflows/python-flake8.yml'
          commit_message: 'ci: update global python'
          repos_to_ignore: ''
          topics_to_include: 'python'
          exclude_private: false
          exclude_forked: true
          branches: '^nightly$'
          destination: ''
        - job_name: 'replicate docker'
          patterns_to_ignore: ''
          patterns_to_include: '.github/ci-docker.yml'
          commit_message: 'ci: update global docker'
          repos_to_ignore: ''
          topics_to_include: 'docker'
          exclude_private: false
          exclude_forked: true
          branches: '^nightly$'
          destination: ''
        - job_name: 'replicate cpp'
          patterns_to_ignore: ''
          patterns_to_include: '.clang-format,.github/workflows_extra/cpp-clang-format-lint.yml'
          commit_message: 'ci: update global cpp'
          repos_to_ignore: ''
          topics_to_include: 'cpp'
          exclude_private: false
          exclude_forked: true
          branches: '^nightly$'
          destination: ''
        - job_name: 'custom issues'
          patterns_to_ignore: ''
          patterns_to_include: '.github/ISSUE_TEMPLATE/config.yml'
          commit_message: 'ci: update issue templates'
          repos_to_ignore: ''
          topics_to_include: 'replicator-custom-issues'
          exclude_private: false
          exclude_forked: true
          branches: '^nightly$'
          destination: ''
        - job_name: 'release notifier'
          patterns_to_ignore: ''
          patterns_to_include: '.github/workflows/release-notifier.yml'
          commit_message: 'ci: update release notifier'
          repos_to_ignore: ''
          topics_to_include: 'replicator-release-notifications'
          exclude_private: false
          exclude_forked: true
          branches: '^nightly$'
          destination: ''

    name: ${{ matrix.job_name }}

    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Replicating files
        uses: derberg/copy-files-to-other-repositories@v1
        with:
          github_token: ${{ secrets.GH_BOT_TOKEN }}
          committer_username: ${{ secrets.GH_BOT_NAME }}
          committer_email: ${{ secrets.GH_BOT_EMAIL }}
          patterns_to_ignore: ${{ matrix.patterns_to_ignore }}
          patterns_to_include: ${{ matrix.patterns_to_include }}
          commit_message: ${{ matrix.commit_message }}
          repos_to_ignore: ${{ matrix.repos_to_ignore }}
          topics_to_include: ${{ matrix.topics_to_include }}
          exclude_private: ${{ matrix.exclude_private }}
          exclude_forked: ${{ matrix.exclude_forked }}
          branches: ${{ matrix.branches }}
          destination: ${{ matrix.destination }}

Expected behavior

A clear and concise description of what you expected to happen.

Auto retry to recreate PRs if rate limited.

derberg commented 1 year ago

@ReenigneArcher sorry for the late response.

I already had issue with PR creation API in the past and already did retry there -> https://github.com/derberg/copy-files-to-other-repositories/blob/main/lib/api-calls.js#L157

So as you can see there are already 5 retires and 5sec delay before each. The problem is not solvable with more retires and larger delays. It is not a rate limit, at least I haven't found in the past the answer to that issue. Maybe it is kind of DDoS prevention or something. Dunno 🤷🏼

I'm afraid the only solution is to not do parallel execution really.

Side question is, why do you need this parallel execution? We you care so much on the speed of the action? Maybe issues you have I already solved in AsyncAPI, the project where we use this action on over 60 repos.

ReenigneArcher commented 1 year ago

Side question is, why do you need this parallel execution?

I don't necessarily need it, just speeds up the workflow a bit as I have 6 scenarios. Not a big deal at all. I would be much happier to see #24 addressed. If a rebase could be done to a specified branch instead of opening multiple PRs, I think this action would be nearly perfect. It's okay if you want to close this. Thanks for all your efforts!

derberg commented 1 year ago

yeah #24 is tricky.

I'll close this one then as GitHub is really mysterious on this error. There is no clean solution for now