pypa / bandersnatch

A PyPI mirror client according to PEP 381 http://www.python.org/dev/peps/pep-0381/

Academic Free License v3.0

455 stars 141 forks source link

"Unauthorized error" and "error when attempting to clear the cache" and "xxx package not updating. Giving up" . #1220

Closed 89ao closed 2 years ago

89ao commented 2 years ago

When I look in to bandersnatch log I see some "Unauthorized error" and "error when attempting to clear the cache" and "xxx package not updating. Giving up" . Could your please tell me why and help me solve them?

I'm using bandersnatch 6.0.0 and logs are below:

2022-09-22 20:49:21,090 bandersnatch.package: INFO Fetching metadata for package: tencentcloud-sdk-python-cynosdb (serial 15172235)
2022-09-22 20:49:21,146 bandersnatch.master: WARNING Got an error when attempting to clear the cache
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/master.py", line 130, in check_for_stale_cache
    async with self.session.request("PURGE", path):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 640, in _request
    resp.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://pypi.org/pypi/onednn-devel-cpu-iomp/json')
2022-09-22 20:49:21,151 bandersnatch.package: ERROR Stale serial for package onednn-devel-cpu-iomp - Attempt 1
2022-09-22 20:49:21,169 bandersnatch.package: INFO nbdev-cards-on-flakes no longer exists on PyPI
2022-09-22 20:49:21,170 bandersnatch.package: INFO Fetching metadata for package: tencentcloud-sdk-python-memcached (serial 15173407)
2022-09-22 20:49:21,210 bandersnatch.master: WARNING Got an error when attempting to clear the cache
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/master.py", line 130, in check_for_stale_cache
    async with self.session.request("PURGE", path):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 640, in _request
    resp.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://pypi.org/pypi/tencentcloud-sdk-python-cynosdb/json')
2022-09-22 20:49:21,211 bandersnatch.package: ERROR Stale serial for package tencentcloud-sdk-python-cynosdb - Attempt 1
2022-09-22 20:49:21,339 bandersnatch.package: INFO p-cal no longer exists on PyPI
2022-09-22 20:49:21,339 bandersnatch.package: INFO Fetching metadata for package: tgbotcalendar (serial 15163720)
2022-09-22 20:49:21,341 bandersnatch.package: INFO pygameux no longer exists on PyPI
2022-09-22 20:49:21,341 bandersnatch.package: INFO Fetching metadata for package: tgbotcallbackdata (serial 15163656)
2022-09-22 20:49:21,351 bandersnatch.master: WARNING Got an error when attempting to clear the cache
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/master.py", line 130, in check_for_stale_cache
    async with self.session.request("PURGE", path):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 640, in _request
    resp.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://pypi.org/pypi/tencentcloud-sdk-python-memcached/json')
2022-09-22 20:49:21,351 bandersnatch.package: ERROR Stale serial for package tencentcloud-sdk-python-memcached - Attempt 1
2022-09-22 20:49:21,360 bandersnatch.package: INFO pyprospect no longer exists on PyPI
2022-09-22 20:49:21,360 bandersnatch.package: INFO Fetching metadata for package: tgbotscenario (serial 15163708)
2022-09-22 20:49:21,363 bandersnatch.package: INFO onelogin-aws no longer exists on PyPI
2022-09-22 20:49:21,383 bandersnatch.package: INFO pyrejoke no longer exists on PyPI
2022-09-22 20:49:21,394 bandersnatch.package: INFO streamlit-extras-arnaudmiribel no longer exists on PyPI
2022-09-22 20:49:21,406 bandersnatch.package: INFO qpuview no longer exists on PyPI
2022-09-22 20:49:21,644 bandersnatch.package: INFO tgbotcalendar no longer exists on PyPI
2022-09-22 20:49:21,651 bandersnatch.package: INFO tgbotscenario no longer exists on PyPI
2022-09-22 20:49:21,799 bandersnatch.package: INFO tgbotcallbackdata no longer exists on PyPI
2022-09-22 20:49:22,152 bandersnatch.package: INFO Fetching metadata for package: onednn-devel-cpu-iomp (serial 15168412)
2022-09-22 20:49:22,213 bandersnatch.package: INFO Fetching metadata for package: tencentcloud-sdk-python-cynosdb (serial 15172235)
2022-09-22 20:49:22,250 bandersnatch.master: WARNING Got an error when attempting to clear the cache
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/master.py", line 130, in check_for_stale_cache
    async with self.session.request("PURGE", path):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 640, in _request
    resp.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://pypi.org/pypi/onednn-devel-cpu-iomp/json')
2022-09-22 20:49:22,251 bandersnatch.package: ERROR Stale serial for package onednn-devel-cpu-iomp - Attempt 2
2022-09-22 20:49:22,315 bandersnatch.master: WARNING Got an error when attempting to clear the cache
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/master.py", line 130, in check_for_stale_cache
    async with self.session.request("PURGE", path):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 640, in _request
    resp.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://pypi.org/pypi/tencentcloud-sdk-python-cynosdb/json')
2022-09-22 20:49:22,315 bandersnatch.package: ERROR Stale serial for package tencentcloud-sdk-python-cynosdb - Attempt 2
2022-09-22 20:49:22,352 bandersnatch.package: INFO Fetching metadata for package: tencentcloud-sdk-python-memcached (serial 15173407)
2022-09-22 20:49:22,466 bandersnatch.master: WARNING Got an error when attempting to clear the cache
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/master.py", line 130, in check_for_stale_cache
...
...
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/master.py", line 137, in check_for_stale_cache
    raise StalePage(
bandersnatch.master.StalePage: Expected PyPI serial 15168412 for request https://pypi.python.org/pypi/onednn-devel-cpu-iomp/json but got 15168374. HTTP PURGE has been issued to the request url

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/mirror.py", line 133, in package_syncer
    await package.update_metadata(self.master, attempts=3)
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/package.py", line 86, in update_metadata
    raise error_class(package_name=self.name, attempts=attempts)
bandersnatch.errors.StaleMetadata: Stale serial for onednn-devel-cpu-iomp after 3 attempts
2022-09-22 20:49:24,467 bandersnatch.package: INFO Fetching metadata for package: tencentcloud-sdk-python-memcached (serial 15173407)
2022-09-22 20:49:24,472 bandersnatch.master: WARNING Got an error when attempting to clear the cache
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/master.py", line 130, in check_for_stale_cache
    async with self.session.request("PURGE", path):
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 1138, in __aenter__
    self._resp = await self._coro
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 640, in _request
    resp.raise_for_status()
  File "/usr/local/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 401, message='Unauthorized', url=URL('https://pypi.org/pypi/tencentcloud-sdk-python-cynosdb/json')
2022-09-22 20:49:24,472 bandersnatch.package: ERROR Stale serial for package tencentcloud-sdk-python-cynosdb - Attempt 3
2022-09-22 20:49:24,472 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-22 20:49:24,472 bandersnatch.mirror: ERROR Error syncing package: tencentcloud-sdk-python-cynosdb@15172235
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/package.py", line 61, in update_metadata
    self._metadata = await master.get_package_metadata(
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/master.py", line 240, in get_package_metadata
    metadata_response = await metadata_generator.asend(None)
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/master.py", line 158, in get
    await self.check_for_stale_cache(path, required_serial, got_serial)
  File "/usr/local/lib/python3.10/site-packages/bandersnatch/master.py", line 137, in check_for_stale_cache
    raise StalePage(
bandersnatch.master.StalePage: Expected PyPI serial 15172235 for request https://pypi.python.org/pypi/tencentcloud-sdk-python-cynosdb/json but got 15172233. HTTP PURGE has been issued to the request url

cooperlees commented 2 years ago

Howdy,

What's your workers set to and how often are you syncing?

1) The 401's might be rate limiting as the URLs look valid to me. We could confirm by reading PyPI Warehouse code + asking there for confirmation there

2) "error when attempting to clear the cache: sounds like https://github.com/pypa/bandersnatch/issues/56

I don't think the PURGEs work anymore with the CDN - So the workaround could not be working.
https://github.com/pypi/warehouse/issues/4892 is still opened ... But haven't checked if this still happens or not ...

3) "xxx package not updating. Giving up" I'd love to see an example. Looking at bandersnatch code I can't find it:

[cooper@cooper-fedora-MJ0BWXDF bandersnatch]$ grep -Ri "package not updating. Giving up" *
[cooper@cooper-fedora-MJ0BWXDF bandersnatch]$

89ao commented 2 years ago

What's your workers set to and how often are you syncing?

I set 5 workers and update every 1800s（30min）,and I have 4 update servers to mirror packages. Do we have a recommond setting on this situation?

"not updating. Giving up" examples are below:

[root@VM_21_93_centos /data/home/motorao]# grep -Ri "not updating. Giving up" /yum/pip/banderlogfile.log.2022-09-23_17 
2022-09-23 00:57:30,898 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 00:57:31,536 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 00:57:31,615 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 01:27:31,025 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 01:27:31,445 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 01:27:31,612 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 01:57:31,105 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 01:57:31,480 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 01:57:31,564 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.2022-09-23 02:27:31,060 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 02:27:31,504 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 02:27:31,677 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 02:57:31,192 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 02:57:31,573 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 02:57:31,649 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 03:27:31,289 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 03:27:31,777 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 03:27:31,782 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 03:57:31,344 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 03:57:31,729 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 03:57:31,744 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 04:27:31,408 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 04:27:31,827 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 04:27:31,924 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 04:57:31,611 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 04:57:31,975 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 04:57:32,012 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 05:27:31,692 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 05:27:32,116 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 05:27:32,159 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 05:57:31,810 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 05:57:32,211 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 05:57:32,246 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 06:27:31,840 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 06:27:32,172 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 06:27:32,234 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.2022-09-23 06:57:31,491 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 06:57:31,672 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 06:57:31,776 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 07:27:32,165 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 07:27:32,524 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 07:27:32,679 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 07:57:32,262 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 07:57:32,643 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 07:57:32,710 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 08:27:32,412 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 08:27:32,742 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-cynosdb (15172235) not updating. Giving up.
2022-09-23 08:27:32,780 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 08:57:32,387 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 08:57:32,746 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 09:27:32,473 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 09:27:32,924 bandersnatch.package: ERROR Stale serial for tencentcloud-sdk-python-memcached (15173407) not updating. Giving up.
2022-09-23 09:57:32,065 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 10:27:32,167 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 10:57:32,828 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 11:27:32,944 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 11:57:32,998 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 12:27:33,055 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 12:57:33,052 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 13:27:33,209 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 13:57:32,760 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 14:27:33,483 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 14:57:33,718 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 15:27:33,840 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 15:57:33,880 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.
2022-09-23 16:27:33,897 bandersnatch.package: ERROR Stale serial for onednn-devel-cpu-iomp (15168412) not updating. Giving up.

cooperlees commented 2 years ago

1) Would all 4 mirrors come from the same Internet routable IP address? If so, that would probably be getting rate limited.

2) I'd personally recommend deleting these packages from your TODO file and manually full sync them and it should hopefully recover. I don't know how you've ended up in that siutation for two packages but you must have done a sync and maybe things were changed with PyPI? Not sure here. Here is the code: https://github.com/pypa/bandersnatch/blob/main/src/bandersnatch/package.py#L84

89ao commented 2 years ago

thank you very much ,I'll have a try on this .