search

WeblateOrg / weblate

Web based localization tool with tight version control integration.
https://weblate.org/
GNU General Public License v3.0
4.66k stars 1.02k forks source link

GitHub: Random 429 Client Error: too many requests #6583

Closed stanislav-brabec closed 3 years ago

stanislav-brabec commented 3 years ago

I am experiencing this error e-mail randomly, often several similar mails at once:

Broken repository browser URL

This component is set up incorrectly.

The repository browser URL points to non-existing locations:

https://github.com/bbidulock/icewm/blob/icewm-1-4-BRANCH/src/aapm.cc#L154 (429 Client Error: too many requests for url: https://github.com/bbidulock/icewm/blob/icewm-1-4-BRANCH/src/aapm.cc#L154)

Looking at the log, it happened i the time, when several robots were crawling Weblate in parallel and several users were browsing, and one user did complicated searches.

It seems that GitHub introduces on decreased the limit. Weblate should reflect it and decrease frequency of source browser URL checks.

Looking into the localmessages, I see:

2021-09-21T17:13:23.338344+00:00 nuka weblate[26460]: WARNING URL check failed: HTTPError: 404 Client Error: Not Found for url: https://github.com/libyui/libyui-qt/blob/master/src/QY2DiskUsageList.cc#L189
2021-09-21T17:13:40.999848+00:00 nuka weblate[26460]: WARNING URL check failed: HTTPError: 429 Client Error: too many requests for url: https://github.com/uyuni-project/uyuni-docs/blob/manager-4.2/modules/upgrade/nav-upgrade-guide.adoc#L1
2021-09-21T17:13:41.329890+00:00 nuka weblate[26460]: WARNING URL check failed: HTTPError: 429 Client Error: too many requests for url: https://github.com/uyuni-project/uyuni/blob/master-weblate/susemanager/yast/susemanager_ask.rb#L38
2021-09-21T17:13:41.640259+00:00 nuka weblate[26460]: WARNING URL check failed: HTTPError: 429 Client Error: too many requests for url: https://github.com/uyuni-project/uyuni-docs/blob/manager-4.1/modules/installation/nav-installation-guide.adoc#L1
2021-09-21T17:13:41.900937+00:00 nuka weblate[26460]: WARNING URL check failed: HTTPError: 429 Client Error: too many requests for url: https://github.com/openSUSE/scout/blob/master/scout/autoconf.py#L14

The first error is correct, the rest is unexpected.

To Reproduce the issue

It is not easy to reproduce. It appears randomly, once per several days or weeks.

Server configuration and status

Weblate installation: openSUSE Leap 15.3 + dedicated repositories.

weblate list_versions

* Weblate: 4.7.1
 * Django: 3.2.5
 * siphashc: 1.3
 * translate-toolkit: 3.3.6
 * lxml: 4.6.3
 * Pillow: 8.3.1
 * bleach: 3.1.5
 * python-dateutil: 2.8.1
 * social-auth-core: 4.1.0
 * social-auth-app-django: 4.0.0
 * django-crispy-forms: 1.9.0
 * oauthlib: 2.0.6
 * django-compressor: 2.4
 * djangorestframework: 3.11.2
 * django-filter: 2.4.0
 * django-appconf: 1.0.4
 * user-agents: 2.1
 * filelock: 3.0.12
 * setuptools: 40.5.0
 * jellyfish: 0.7.2
 * openpyxl: 3.0.3
 * celery: 5.0.5
 * kombu: 5.1.0
 * translation-finder: 2.9
 * weblate-language-data: 2021.5
 * html2text: 2019.8.11
 * pycairo: 1.20.0
 * pygobject: 3.34.0
 * diff-match-patch: 20181111
 * requests: 2.25.1
 * django-redis: 4.11.0
 * hiredis: 1.0.1
 * sentry_sdk: 0.14.4
 * Cython: 0.29.14
 * misaka: 2.1.1
 * GitPython: 3.0.5
 * borgbackup: 1.1.15
 * pyparsing: 2.4.7
 * pyahocorasick: 1.4.1
 * python-redis-lock: 3.6.0
 * Python: 3.6.13
 * Git: 2.31.1
 * psycopg2: 2.8.5
 * chardet: 3.0.4
 * ruamel.yaml: 0.17.4
 * Redis server: 6.0.14
 * PostgreSQL server: 12.8
 * Database backends: django.db.backends.postgresql
 * Cache backends: default:RedisCache, avatar:FileBasedCache
 * Email setup: django.core.mail.backends.smtp.EmailBackend: localhost
 * OS encoding: filesystem=utf-8, default=utf-8
 * Celery: redis://localhost:6379, redis://localhost:6379, regular
 * Platform: Linux 5.3.18-59.19-default (x86_64)

Weblate deploy checks

weblate check --deploy

WARNINGS:
?: (weblate.W025.ass) Failure in loading handler for ass file format: aeidon or gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W025.ini) Failure in loading handler for ini file format: Missing iniparse library.
    HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W025.islu) Failure in loading handler for islu file format: Missing iniparse library.
    HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W025.laravel) Failure in loading handler for laravel file format: No module named 'phply'
    HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W025.php) Failure in loading handler for php file format: No module named 'phply'
    HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W025.srt) Failure in loading handler for srt file format: aeidon or gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W025.ssa) Failure in loading handler for ssa file format: aeidon or gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W025.sub) Failure in loading handler for sub file format: aeidon or gaupol package required for Subtitle support
    HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#optional-deps
?: (weblate.W033.Gerrit) Failure in loading VCS module for Gerrit: git: 'review' is not a git command. See 'git --help'.
 (1)
    HINT: https://docs.weblate.org/en/weblate-4.7.1/vcs.html
?: (weblate.W033.Mercurial) Failure in loading VCS module for Mercurial: [Errno 2] No such file or directory: 'hg': 'hg'
    HINT: https://docs.weblate.org/en/weblate-4.7.1/vcs.html
?: (weblate.W033.Subversion) Failure in loading VCS module for Subversion: git: 'svn' is not a git command. See 'git --help'.

The most similar commands are
    fsck
    mv
    show
 (1)
    HINT: https://docs.weblate.org/en/weblate-4.7.1/vcs.html

INFOS:
?: (weblate.I021) Error collection is not set up, it is highly recommended for production use
    HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/install.html#collecting-errors
?: (weblate.I028) Backups are not configured, it is highly recommended for production use
    HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/backup.html
?: (weblate.I031) New Weblate version is available, please upgrade to 4.8.1.
    HINT: https://docs.weblate.org/en/weblate-4.7.1/admin/upgrade.html

System check identified 14 issues (1 silenced).

Additional context

I am not sure, whether GitHub counts only direct accesses, or also accesses with Referrer.

Analyzing the web crawler logs, I suggest to enhance robots.txt with:

/api/
/checks/
/engage/
/languages/*/*/search/

and maybe

/widgets/
stanislav-brabec commented 3 years ago

And a minor suggestion: remote HTTP responses like 429, 500, 503, 504 and similar possibly temporary replies should not trigger immediate mails to all project/component subscribers. It makes sense to report it after several failed attempts.

github-actions[bot] commented 3 years ago

Thank you for your report; the issue you have reported has just been fixed.