search

rvojcik / gitlab-project-export

Simple python project for exporting GitLab projects with Export Project funkcionality in GitLab API. Import of exported projects is also supported
GNU General Public License v3.0
103 stars 39 forks source link

Throttle requests #11

Closed fitzg2 closed 3 years ago

fitzg2 commented 3 years ago

Trying to download many projects results in 

Export failed for project <group>/<project>
API not respond well with 429 {"message":{"error":"This endpoint has been requested too many times. Try again later."}}
fitzg2 commented 3 years ago

Ah seeing this in the code

        # If set, wait between exports
        if args.debug:
            print("Waiting between exports for %d seconds" % (wait_between_exports))
        time.sleep(wait_between_exports)

Had to set wait_between_exports in the gitlab config. Probably should be added to the sample config.yml

rvojcik commented 3 years ago

Hi @fitzg2 , yes that's true. Gitlab.com instance has several rate limits.

In config.yaml-example there is already some information about that

  # Wait Between Exports
  # How many seconds to wait between projects to export.
  # This is due rate-limiting in gitlab.com
  # https://docs.gitlab.com/ee/user/project/settings/import_export.html#rate-limits
  #
  # If you break the limit API respond with 429 for your IP address for a while.
  #
  # When you are using this in your private gitlab instance you can set it to 0
  wait_between_exports: 303