pypi / warehouse

The Python Package Index
https://pypi.org
Apache License 2.0
3.6k stars 968 forks source link

PyPI serving stale content from JSON/Simple API #12214

Open chriseldredge opened 2 years ago

chriseldredge commented 2 years ago

Likely related to #11936, I am seeing stale content served for the URL https://pypi.org/pypi/dbnd-postgres/json.

The following events show in changes_since_serial today:

['dbnd-postgres', '1.0.2.2', 1663173894, 'new release', 15093808]
['dbnd-postgres', '1.0.2.2', 1663173894, 'add py2.py3 file dbnd_postgres-1.0.2.2-py2.py3-none-any.whl', 15093809]
['dbnd-postgres', '1.0.2.2', 1663173939, 'add source file dbnd-postgres-1.0.2.2.tar.gz', 15093840]

However this curl command shows the serial ID did not pick up the most recent event:

curl -sq -I https://pypi.org/pypi/dbnd-postgres/json -H 'Fastly-Debug: 1' --compressed

Output:

HTTP/2 200
access-control-allow-headers: Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
access-control-allow-methods: GET
access-control-allow-origin: *
access-control-expose-headers: X-PyPI-Last-Serial
access-control-max-age: 86400
cache-control: max-age=900, public
content-encoding: gzip
content-security-policy: base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ fastly-insights.com *.fastly-insights.com *.ethicalads.io https://api.pwnedpasswords.com https://cdn.jsdelivr.net/npm/mathjax@3.2.2/es5/sre/mathmaps/ https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.ingress.cmh1.psfhosted.org/ www.google-analytics.com *.fastly-insights.com *.ethicalads.io; script-src 'self' www.googletagmanager.com www.google-analytics.com *.fastly-insights.com *.ethicalads.io 'sha256-U3hKDidudIaxBDEzwGJApJgPEf2mWk6cfMWghrAa6i0=' https://cdn.jsdelivr.net/npm/mathjax@3.2.2/ 'sha256-1CldwzdEg2k1wTmf7s5RWVd7NMXI/7nxxjJM2C4DqII=' 'sha256-0POaN8stWYQxhzjKS+/eOfbbJ/u4YHO5ZagJvLpMypo='; style-src 'self' fonts.googleapis.com *.ethicalads.io 'sha256-2YHqZokjiizkHi1Zt+6ar0XJ0OeEy/egBnlm+MDMtrM=' 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=' 'sha256-JLEjeN9e5dGsz5475WyRaoA4eQOdNPxDIeUhclnJDCE=' 'sha256-mQyxHEuwZJqpxCw3SLmc4YOySNKXunyu2Oiz1r3/wAE=' 'sha256-OCf+kv5Asiwp++8PIevKBYSgnNLNUZvxAp4a7wMLuKA=' 'sha256-h5LOiLhk6wiJrGsG5ItM0KimwzWQH/yAcmoJDJL//bY='; worker-src *.fastly-insights.com
content-type: application/json
etag: "OQffMU7lDwd/T0iw2gQQVQ"
referrer-policy: origin-when-cross-origin
server: nginx/1.13.9
surrogate-control: max-age=86400, stale-while-revalidate=300, stale-if-error=86400
surrogate-key: all-legacy-json project-legacy-json project/dbnd-postgres
x-pypi-last-serial: 15093809
accept-ranges: bytes
date: Wed, 14 Sep 2022 20:20:20 GMT
fastly-debug-path: (D cache-iad-kiad7000060-IAD 1663186820) (F cache-iad-kcgs7200149-IAD 1663173907)
fastly-debug-ttl: (H cache-iad-kiad7000060-IAD 73486.697 86400.000 12913)
fastly-debug-digest: d7cf4a7d85536d23624f76df11d1059e7f9287eea031e6c73846429273a5acfd
x-served-by: cache-iad-kiad7000060-IAD
x-cache: HIT
x-cache-hits: 147
x-timer: S1663186820.267731,VS0,VE0
vary: Accept-Encoding
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: deny
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-permitted-cross-domain-policies: none
content-length: 118223

When I repeat this command without the --compressed flag I see the expected output:

curl -sq -I https://pypi.org/pypi/dbnd-postgres/json -H 'Fastly-Debug: 1'

Output:

HTTP/2 200
access-control-allow-headers: Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
access-control-allow-methods: GET
access-control-allow-origin: *
access-control-expose-headers: X-PyPI-Last-Serial
access-control-max-age: 86400
cache-control: max-age=900, public
content-security-policy: base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ fastly-insights.com *.fastly-insights.com *.ethicalads.io https://api.pwnedpasswords.com https://cdn.jsdelivr.net/npm/mathjax@3.2.2/es5/sre/mathmaps/ https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.ingress.cmh1.psfhosted.org/ www.google-analytics.com *.fastly-insights.com *.ethicalads.io; script-src 'self' www.googletagmanager.com www.google-analytics.com *.fastly-insights.com *.ethicalads.io 'sha256-U3hKDidudIaxBDEzwGJApJgPEf2mWk6cfMWghrAa6i0=' https://cdn.jsdelivr.net/npm/mathjax@3.2.2/ 'sha256-1CldwzdEg2k1wTmf7s5RWVd7NMXI/7nxxjJM2C4DqII=' 'sha256-0POaN8stWYQxhzjKS+/eOfbbJ/u4YHO5ZagJvLpMypo='; style-src 'self' fonts.googleapis.com *.ethicalads.io 'sha256-2YHqZokjiizkHi1Zt+6ar0XJ0OeEy/egBnlm+MDMtrM=' 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=' 'sha256-JLEjeN9e5dGsz5475WyRaoA4eQOdNPxDIeUhclnJDCE=' 'sha256-mQyxHEuwZJqpxCw3SLmc4YOySNKXunyu2Oiz1r3/wAE=' 'sha256-OCf+kv5Asiwp++8PIevKBYSgnNLNUZvxAp4a7wMLuKA=' 'sha256-h5LOiLhk6wiJrGsG5ItM0KimwzWQH/yAcmoJDJL//bY='; worker-src *.fastly-insights.com
content-type: application/json
etag: "XcI/eU0vTs6ZFLleB2JpKQ"
referrer-policy: origin-when-cross-origin
server: nginx/1.13.9
surrogate-control: max-age=86400, stale-while-revalidate=300, stale-if-error=86400
surrogate-key: all-legacy-json project-legacy-json project/dbnd-postgres
x-pypi-last-serial: 15093840
accept-ranges: bytes
date: Wed, 14 Sep 2022 20:27:40 GMT
fastly-debug-path: (D cache-iad-kiad7000106-IAD 1663187261) (F cache-iad-kiad7000060-IAD 1663173992)
fastly-debug-ttl: (H cache-iad-kiad7000106-IAD - - 13269)
fastly-debug-digest: d7cf4a7d85536d23624f76df11d1059e7f9287eea031e6c73846429273a5acfd
x-served-by: cache-iad-kiad7000106-IAD
x-cache: HIT
x-cache-hits: 1
x-timer: S1663187261.581965,VS0,VE3
vary: Accept-Encoding, Accept-Encoding
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: deny
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-permitted-cross-domain-policies: none
content-length: 553040

This indicates to me that cache invalidation did not work.

I see from recent GitHub issues this has been a recurring issue recently. It is painful since without manual intervention it won't self-resolve until the TTL of 24 hours expires.

ewdurbin commented 2 years ago

Hmmm, that does certainly appear to be a cache invalidation issue. I'm not able to determine that we have had any systemic failures of the purge task (no reported failures in the last 48 hours).

I'm wondering if the open incident impacting Fastly API layer has anything to do with it, but it makes no mention of purges.

ewdurbin commented 2 years ago

Issuing a purge by the project key project/dbnd-postgres does seem to have resolved this specific instance (bringing the compressed response in line with the uncompressed one). So purges are working as expected by key at this moment.

The only failure mechanism I can see inside of our codebase that would lead to this result is if the purge task is not being properly enqueued, though I don't see any errors in our telemetry indicating such a failure either.

chriseldredge commented 2 years ago

The manual purge resolved the problem, but another one has cropped up this morning. This seems to happen somewhat frequently when a new release adds more than one file very close together. It seems like there's a race between when the cache invalidation occurs that perhaps is preventing a subsequent overlapping invalidation from executing.

['tencentcloud-sdk-python-bmlb', '3.0.732', 1663201857, 'new release', 15098168]
['tencentcloud-sdk-python-bmlb', '3.0.732', 1663201857, 'add py2.py3 file tencentcloud_sdk_python_bmlb-3.0.732-py2.py3-none-any.whl', 15098169]
['tencentcloud-sdk-python-bmlb', '3.0.732', 1663201862, 'add source file tencentcloud-sdk-python-bmlb-3.0.732.tar.gz', 15098170]
curl -sq -I https://pypi.org/pypi/tencentcloud-sdk-python-bmlb/json -H 'Fastly-Debug: 1' --compressed
HTTP/2 200
access-control-allow-headers: Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
access-control-allow-methods: GET
access-control-allow-origin: *
access-control-expose-headers: X-PyPI-Last-Serial
access-control-max-age: 86400
cache-control: max-age=900, public
content-encoding: gzip
content-security-policy: base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ fastly-insights.com *.fastly-insights.com *.ethicalads.io https://api.pwnedpasswords.com https://cdn.jsdelivr.net/npm/mathjax@3.2.2/es5/sre/mathmaps/ https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self' https://checkout.stripe.com; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.ingress.cmh1.psfhosted.org/ www.google-analytics.com *.fastly-insights.com *.ethicalads.io; script-src 'self' www.googletagmanager.com www.google-analytics.com *.fastly-insights.com *.ethicalads.io 'sha256-U3hKDidudIaxBDEzwGJApJgPEf2mWk6cfMWghrAa6i0=' https://cdn.jsdelivr.net/npm/mathjax@3.2.2/ 'sha256-1CldwzdEg2k1wTmf7s5RWVd7NMXI/7nxxjJM2C4DqII=' 'sha256-0POaN8stWYQxhzjKS+/eOfbbJ/u4YHO5ZagJvLpMypo='; style-src 'self' fonts.googleapis.com *.ethicalads.io 'sha256-2YHqZokjiizkHi1Zt+6ar0XJ0OeEy/egBnlm+MDMtrM=' 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=' 'sha256-JLEjeN9e5dGsz5475WyRaoA4eQOdNPxDIeUhclnJDCE=' 'sha256-mQyxHEuwZJqpxCw3SLmc4YOySNKXunyu2Oiz1r3/wAE=' 'sha256-OCf+kv5Asiwp++8PIevKBYSgnNLNUZvxAp4a7wMLuKA=' 'sha256-h5LOiLhk6wiJrGsG5ItM0KimwzWQH/yAcmoJDJL//bY='; worker-src *.fastly-insights.com
content-type: application/json
etag: "XnNNuWLsw0Qy0v0Sa/p3iw"
referrer-policy: origin-when-cross-origin
server: nginx/1.13.9
surrogate-control: max-age=86400, stale-while-revalidate=300, stale-if-error=86400
surrogate-key: all-legacy-json project-legacy-json project/tencentcloud-sdk-python-bmlb
x-pypi-last-serial: 15098169
accept-ranges: bytes
date: Thu, 15 Sep 2022 14:39:45 GMT
fastly-debug-path: (D cache-iad-kjyo7100176-IAD 1663252785) (F cache-iad-kiad7000106-IAD 1663201864)
fastly-debug-ttl: (H cache-iad-kjyo7100176-IAD - - 50921)
fastly-debug-digest: 6e4df74f912d01b6d565ab286c5aa8b0ce76887517b9db6f1e2b112994b00c27
x-served-by: cache-iad-kjyo7100176-IAD
x-cache: HIT
x-cache-hits: 1
x-timer: S1663252785.243295,VS0,VE2
vary: Accept-Encoding
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: deny
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-permitted-cross-domain-policies: none
content-length: 129393
curl -sq -I https://pypi.org/pypi/tencentcloud-sdk-python-bmlb/json -H 'Fastly-Debug: 1'
HTTP/2 200
access-control-allow-headers: Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
access-control-allow-methods: GET
access-control-allow-origin: *
access-control-expose-headers: X-PyPI-Last-Serial
access-control-max-age: 86400
cache-control: max-age=900, public
content-security-policy: base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ fastly-insights.com *.fastly-insights.com *.ethicalads.io https://api.pwnedpasswords.com https://cdn.jsdelivr.net/npm/mathjax@3.2.2/es5/sre/mathmaps/ https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self' https://checkout.stripe.com; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.ingress.cmh1.psfhosted.org/ www.google-analytics.com *.fastly-insights.com *.ethicalads.io; script-src 'self' www.googletagmanager.com www.google-analytics.com *.fastly-insights.com *.ethicalads.io 'sha256-U3hKDidudIaxBDEzwGJApJgPEf2mWk6cfMWghrAa6i0=' https://cdn.jsdelivr.net/npm/mathjax@3.2.2/ 'sha256-1CldwzdEg2k1wTmf7s5RWVd7NMXI/7nxxjJM2C4DqII=' 'sha256-0POaN8stWYQxhzjKS+/eOfbbJ/u4YHO5ZagJvLpMypo='; style-src 'self' fonts.googleapis.com *.ethicalads.io 'sha256-2YHqZokjiizkHi1Zt+6ar0XJ0OeEy/egBnlm+MDMtrM=' 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=' 'sha256-JLEjeN9e5dGsz5475WyRaoA4eQOdNPxDIeUhclnJDCE=' 'sha256-mQyxHEuwZJqpxCw3SLmc4YOySNKXunyu2Oiz1r3/wAE=' 'sha256-OCf+kv5Asiwp++8PIevKBYSgnNLNUZvxAp4a7wMLuKA=' 'sha256-h5LOiLhk6wiJrGsG5ItM0KimwzWQH/yAcmoJDJL//bY='; worker-src *.fastly-insights.com
content-type: application/json
etag: "bkVDS2B7X2PsUt8Lq+m3VQ"
referrer-policy: origin-when-cross-origin
server: nginx/1.13.9
surrogate-control: max-age=86400, stale-while-revalidate=300, stale-if-error=86400
surrogate-key: all-legacy-json project-legacy-json project/tencentcloud-sdk-python-bmlb
x-pypi-last-serial: 15098170
accept-ranges: bytes
date: Thu, 15 Sep 2022 14:39:50 GMT
fastly-debug-path: (D cache-iad-kjyo7100162-IAD 1663252791) (F cache-iad-kiad7000106-IAD 1663201996)
fastly-debug-ttl: (H cache-iad-kjyo7100162-IAD - - 50795)
fastly-debug-digest: 6e4df74f912d01b6d565ab286c5aa8b0ce76887517b9db6f1e2b112994b00c27
x-served-by: cache-iad-kjyo7100162-IAD
x-cache: HIT
x-cache-hits: 1
x-timer: S1663252791.880861,VS0,VE4
vary: Accept-Encoding, Accept-Encoding
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: deny
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-permitted-cross-domain-policies: none
content-length: 640713
ewdurbin commented 2 years ago

Yeah, I wonder if our combination of soft purges and stale-while-revalidate is introducing a race. @dstufft any ideas here?

ewdurbin commented 2 years ago

This section seems relevant: https://developer.fastly.com/learning/concepts/stale/#shielding-considerations

ewdurbin commented 2 years ago

Seems we already try to protect against that edge case with this VCL: https://github.com/python/pypi-infra/blob/985133617a45bdf1ad24c217627707445a7435ff/terraform/warehouse/vcl/main.vcl#L5-L11

di commented 2 years ago

We had some discussion here, one theory is that this may be due to us issuing multiple purge requests in quick succession. We're going to leave this open for now, as we might need to reach out to Fastly support to help debug.

di commented 2 years ago

In https://github.com/pypi/warehouse/issues/12272 it was noted that this is happening with the Simple API as well, as would be expected.

sciyoshi commented 2 years ago

This also seems to be affecting the latest release of mypy (0.981) which is not currently installable via poetry.

ewdurbin commented 2 years ago

Values from the backends are matching what I'm seeing for both compressed and uncompressed responses currently (from CHI point of presence).

@sciyoshi can you share output from the following commands:

curl -sq -H 'Fastly-Debug: 1' -I https://pypi.org/pypi/mypy/json curl -sq -H 'Fastly-Debug: 1' -I --compressed https://pypi.org/pypi/mypy/json

sciyoshi commented 2 years ago

@ewdurbin sure - here's the output:

> curl -sq -H 'Fastly-Debug: 1' -I https://pypi.org/pypi/mypy/json
HTTP/2 200
access-control-allow-headers: Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
access-control-allow-methods: GET
access-control-allow-origin: *
access-control-expose-headers: X-PyPI-Last-Serial
access-control-max-age: 86400
cache-control: max-age=900, public
content-security-policy: base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ fastly-insights.com *.fastly-insights.com *.ethicalads.io https://api.pwnedpasswords.com https://cdn.jsdelivr.net/npm/mathjax@3.2.2/es5/sre/mathmaps/ https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic
.com; form-action 'self' https://checkout.stripe.com; frame-ancestors 'none'; frame-src 'none'; img-src 'self' https://warehouse-camo.ingress.cmh1.psfhosted.org/ www.google-analytics.com *.fastly-insights.com *.ethicalads.io; script-src 'self' www.googletagmanager.com www.google-analytics.com *.fastly-insights.com *.ethicalads.io 'sha256-U3hKDidudIa
xBDEzwGJApJgPEf2mWk6cfMWghrAa6i0=' https://cdn.jsdelivr.net/npm/mathjax@3.2.2/ 'sha256-1CldwzdEg2k1wTmf7s5RWVd7NMXI/7nxxjJM2C4DqII=' 'sha256-0POaN8stWYQxhzjKS+/eOfbbJ/u4YHO5ZagJvLpMypo='; style-src 'self' fonts.googleapis.com *.ethicalads.io 'sha256-2YHqZokjiizkHi1Zt+6ar0XJ0OeEy/egBnlm+MDMtrM=' 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=' '
sha256-JLEjeN9e5dGsz5475WyRaoA4eQOdNPxDIeUhclnJDCE=' 'sha256-mQyxHEuwZJqpxCw3SLmc4YOySNKXunyu2Oiz1r3/wAE=' 'sha256-OCf+kv5Asiwp++8PIevKBYSgnNLNUZvxAp4a7wMLuKA=' 'sha256-h5LOiLhk6wiJrGsG5ItM0KimwzWQH/yAcmoJDJL//bY='; worker-src *.fastly-insights.com
content-type: application/json
etag: "aAQN6E+jltnU36zCGEdw6Q"
referrer-policy: origin-when-cross-origin
server: nginx/1.13.9
surrogate-control: max-age=86400, stale-while-revalidate=300, stale-if-error=86400
surrogate-key: all-legacy-json project-legacy-json project/mypy
x-pypi-last-serial: 15218304
accept-ranges: bytes
date: Tue, 27 Sep 2022 15:55:03 GMT
fastly-debug-path: (D cache-yul12832-YUL 1664294103) (F cache-yul12830-YUL 1664270144) (D cache-iad-kiad7000173-IAD 1664270144) (F cache-iad-kcgs7200155-IAD 1664217287)
fastly-debug-ttl: (H cache-yul12832-YUL - - 23959) (H cache-iad-kiad7000173-IAD 33542.955 86400.000 52857)
fastly-debug-digest: 4bbbfa91b65930e47ecdf33fd12d0f32abb49ea94de23346281ae11d8d7a7fb0
x-served-by: cache-iad-kiad7000173-IAD, cache-yul12832-YUL
x-cache: HIT, HIT
x-cache-hits: 493, 1
x-timer: S1664294103.018667,VS0,VE1
vary: Accept-Encoding, Accept-Encoding
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: deny
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-permitted-cross-domain-policies: none
content-length: 475053
> curl -sq -H 'Fastly-Debug: 1' -I --compressed https://pypi.org/pypi/mypy/json
HTTP/2 200
access-control-allow-headers: Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
access-control-allow-methods: GET
access-control-allow-origin: *
access-control-expose-headers: X-PyPI-Last-Serial
access-control-max-age: 86400
cache-control: max-age=900, public
content-encoding: gzip
content-security-policy: base-uri 'self'; block-all-mixed-content; connect-src 'self' https://api.github.com/repos/ fastly-insights.com *.fastly-insights.com *.ethicalads.io https://api.pwnedpasswords.com https://2p66nmmycsj3.statuspage.io; default-src 'none'; font-src 'self' fonts.gstatic.com; form-action 'self'; frame-ancestors 'none'; frame-src '
none'; img-src 'self' https://warehouse-camo.ingress.cmh1.psfhosted.org/ www.google-analytics.com *.fastly-insights.com *.ethicalads.io; script-src 'self' www.googletagmanager.com www.google-analytics.com *.fastly-insights.com *.ethicalads.io 'sha256-U3hKDidudIaxBDEzwGJApJgPEf2mWk6cfMWghrAa6i0='; style-src 'self' fonts.googleapis.com *.ethicalads.io
 'sha256-2YHqZokjiizkHi1Zt+6ar0XJ0OeEy/egBnlm+MDMtrM=' 'sha256-47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU='; worker-src *.fastly-insights.com
content-type: application/json
etag: "wVyF0KeDmFzsUN0xxzrIfg"
referrer-policy: origin-when-cross-origin
server: nginx/1.13.9
surrogate-control: max-age=86400, stale-while-revalidate=300, stale-if-error=86400
surrogate-key:
x-pypi-last-serial: 14483175
accept-ranges: bytes
date: Tue, 27 Sep 2022 15:55:08 GMT
fastly-debug-path: (D cache-yul12830-YUL 1664294109) (F cache-yul12830-YUL 1664271983) (D cache-iad-kiad7000138-IAD 1664271983) (F cache-iad-kcgs7200155-IAD 1659108366) (D cache-lcy19263-LCY 1659108366) (F cache-lcy19263-LCY 1659108339) (D cache-iad-kiad7000173-IAD 1659108339) (F cache-iad-kcgs7200120-IAD 1658588617)
fastly-debug-ttl: (H cache-yul12830-YUL 64273.801 86400.000 22126) (H cache-iad-kiad7000138-IAD 20723.341 86400.000 65677) (H cache-lcy19263-LCY 86372.920 86400.000 27) (H cache-iad-kiad7000173-IAD 85107.298 86400.000 1293)
fastly-debug-digest: 4bbbfa91b65930e47ecdf33fd12d0f32abb49ea94de23346281ae11d8d7a7fb0
x-served-by: cache-iad-kiad7000173-IAD, cache-lcy19263-LCY, cache-iad-kiad7000138-IAD, cache-yul12830-YUL
x-cache: HIT, HIT, HIT, HIT
x-cache-hits: 2, 1, 62, 36
x-timer: S1664294109.907644,VS0,VE0
vary: Accept-Encoding
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-frame-options: deny
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-permitted-cross-domain-policies: none
content-length: 93198

FWIW the uncompressed version does include 0.981 for me but the --compressed one does not.

di commented 2 years ago

Another instance here: https://github.com/piwheels/packages/issues/326

ewdurbin commented 2 years ago

OK, I'll go ahead and get a support request opened with Fastly.

ewdurbin commented 2 years ago

Ticket filed with Fastly, ID 535394

di commented 2 years ago

Another report here: https://github.com/pypi/warehouse/issues/12290

Lumute commented 2 years ago

Thank you guys for looking into this, in the meantime, is there anything we can do as a workaround on the user side to force a cache refresh of our project?

ewdurbin commented 2 years ago

Just realizing that the Origin Cache is no longer configured to purge the right service for test.pypi.org after https://github.com/pypi/infra/issues/95 🤦🏼

We broke up the legacy hand curated Fastly service into terraformed services that are in line with production, but did not update the configuration for the service to purge. Since the old service still exists... no failure was evident.

I'm updating the backend config for test.pypi.org now and will issue a purge all for test.pypi.org

ewdurbin commented 2 years ago

test.pypi.org configuration is repaired and a purge all for that service has been issued.

Lumute commented 2 years ago

Thanks you! my project is now refreshed and working fine... ready for production, yay...

chriseldredge commented 2 years ago

I have observed this issue about 1000 times in total between 2020-10-05 through 2020-10-12. I've put some deduplicated samples in a gist here: https://gist.github.com/chriseldredge/954bf985bd798b70ee4e9adf01bb4428.

The issue seems not to affect all CDN nodes uniformly. I am mirroring metadata from many regions around the world and see that sometimes the cache is stale only in one or two regions, and more rarely the cache gets stale in most or all regions.

It also appears to disproportionately affect certain packages, notably ccxt. This may be due to the cost of recomputing the metadata for packages that have many versions or it could be because of how frequently updated are published.

I'm curious if the support ticket with Fastly has found anything worth sharing?

ewdurbin commented 2 years ago

@chriseldredge thank you for the additional information. I'll update our ticket with Fastly, they were not able to provide any insight so far.

chriseldredge commented 2 years ago

Following up again. I am continuing to observe this problem frequently. Here are the instances I've found over the last week. The data indicates that the problem sometimes only happens in some regions around the world which supports the hypothesis that this is caused by some race in cache invalidation.

2022-11-14T08:48:48 eu-west-3   cobo-custody
2022-11-14T08:52:58 ap-south-1  tf-nightly
2022-11-14T09:29:27 ap-northeast-1  st-textnav
2022-11-14T10:10:12 ap-northeast-1  gy-multiprocessing
2022-11-14T11:13:57 eu-west-2   c2cciutils
2022-11-14T11:32:40 eu-west-3   clonalevolution
2022-11-14T12:14:44 eu-north-1  metaphor-connectors
2022-11-14T12:24:18 ap-northeast-1  bigdl-orca-spark2
2022-11-14T12:39:08 ap-south-1  bigdl-chronos-spark3
2022-11-14T12:39:12 ap-northeast-1  bigdl-spark3
2022-11-14T12:39:16 eu-west-3   flowmachine
2022-11-14T12:39:17 eu-west-3   bigdl-nano
2022-11-14T12:40:51 ap-southeast-2  wwpdb-apps-ann-tasks-v2
2022-11-14T12:40:58 us-east-2   wwpdb-apps-ann-tasks-v2
2022-11-14T12:41:00 eu-south-1  wwpdb-apps-ann-tasks-v2
2022-11-14T12:41:05 us-west-2   wwpdb-apps-ann-tasks-v2
2022-11-14T12:41:07 ap-south-1  wwpdb-apps-ann-tasks-v2
2022-11-14T12:41:10 ap-northeast-1  wwpdb-apps-ann-tasks-v2
2022-11-14T12:41:11 eu-west-1   wwpdb-apps-ann-tasks-v2
2022-11-14T12:41:11 us-east-1   wwpdb-apps-ann-tasks-v2
2022-11-14T12:41:15 eu-west-3   wwpdb-apps-ann-tasks-v2
2022-11-14T12:41:20 eu-west-2   wwpdb-apps-ann-tasks-v2
2022-11-14T12:41:25 eu-central-1    wwpdb-apps-ann-tasks-v2
2022-11-14T12:41:32 ap-southeast-1  wwpdb-apps-ann-tasks-v2
2022-11-14T12:41:32 eu-north-1  wwpdb-apps-ann-tasks-v2
2022-11-14T12:45:14 eu-west-3   pyowletapi
2022-11-14T12:56:03 ap-northeast-1  bigdl-orca
2022-11-14T12:56:50 ap-southeast-2  bigdl-spark2
2022-11-14T13:11:20 eu-north-1  bigdl-spark3
2022-11-14T13:28:50 ap-south-1  hubspot-api-client
2022-11-14T13:30:13 eu-north-1  itk-numerics
2022-11-14T13:30:16 ap-southeast-1  itk-segmentation
2022-11-14T13:32:03 eu-west-2   trialzen-dash-pivottable
2022-11-14T13:42:49 eu-west-3   mlcvzoo-base
2022-11-14T13:43:06 eu-north-1  vfx-fs-util
2022-11-14T13:45:05 eu-north-1  dash-spinners
2022-11-14T13:48:39 us-west-2   zebrazoom
2022-11-14T13:59:42 us-east-1   checkov
2022-11-14T13:59:45 eu-west-3   checkov
2022-11-14T13:59:52 eu-west-2   checkov
2022-11-14T13:59:53 eu-west-2   duckdb
2022-11-14T13:59:55 eu-central-1    checkov
2022-11-14T14:00:00 eu-north-1  checkov
2022-11-14T14:00:02 ap-southeast-1  checkov
2022-11-14T14:00:11 us-east-2   checkov
2022-11-14T14:00:18 ap-southeast-2  checkov
2022-11-14T14:00:29 eu-south-1  checkov
2022-11-14T14:00:33 ap-south-1  checkov
2022-11-14T14:00:34 us-west-2   checkov
2022-11-14T14:00:36 eu-west-1   checkov
2022-11-14T14:00:39 ap-northeast-1  checkov
2022-11-14T14:00:40 ap-northeast-1  robotframework-dictkeyallpaths
2022-11-14T14:03:59 eu-north-1  xkcd-python
2022-11-14T14:50:23 eu-west-3   bettertui
2022-11-14T16:14:16 ap-southeast-2  ai-flow-nightly
2022-11-14T16:15:00 eu-north-1  akida-models
2022-11-14T16:16:01 eu-north-1  prestool
2022-11-14T16:43:50 ap-southeast-1  dissect-cstruct
2022-11-14T17:03:41 ap-southeast-1  dynamax
2022-11-14T17:23:20 eu-west-3   kangas
2022-11-14T19:01:11 ap-southeast-2  resotoworker
2022-11-14T20:00:11 us-east-1   opentdf
2022-11-14T20:00:15 us-west-2   opentdf
2022-11-14T20:00:16 ap-southeast-1  opentdf
2022-11-14T20:00:20 eu-west-1   opentdf
2022-11-14T20:00:21 eu-north-1  opentdf
2022-11-14T20:00:29 eu-west-2   opentdf
2022-11-14T20:00:36 eu-south-1  opentdf
2022-11-14T20:00:37 eu-central-1    opentdf
2022-11-14T20:00:43 ap-southeast-2  opentdf
2022-11-14T20:01:00 ap-south-1  opentdf
2022-11-14T20:01:02 ap-northeast-1  opentdf
2022-11-14T20:01:09 eu-west-3   opentdf
2022-11-14T20:01:10 us-east-2   opentdf
2022-11-14T20:07:41 ap-southeast-2  assemblyline-v4-service
2022-11-14T20:07:42 ap-southeast-2  assemblyline
2022-11-14T20:34:57 ap-southeast-1  spanishconjugator
2022-11-14T20:35:05 eu-west-1   spanishconjugator
2022-11-14T20:35:05 us-west-2   spanishconjugator
2022-11-14T20:35:09 eu-north-1  spanishconjugator
2022-11-14T20:35:15 eu-south-1  spanishconjugator
2022-11-14T20:35:17 eu-west-2   spanishconjugator
2022-11-14T20:35:21 eu-central-1    spanishconjugator
2022-11-14T20:35:30 ap-southeast-2  spanishconjugator
2022-11-14T20:35:45 ap-south-1  spanishconjugator
2022-11-14T20:35:47 ap-northeast-1  spanishconjugator
2022-11-14T20:35:50 us-east-2   spanishconjugator
2022-11-14T20:35:54 eu-west-3   spanishconjugator
2022-11-14T20:37:14 us-east-1   spanishconjugator
2022-11-14T20:54:59 us-west-2   mypy-boto3-managedblockchain
2022-11-14T20:58:39 ap-northeast-1  pyeasyweb3
2022-11-14T21:42:24 ap-southeast-1  neuralhydrology
2022-11-14T22:28:58 ap-southeast-1  lala-tools
2022-11-14T22:28:59 ap-northeast-1  lala-tools
2022-11-14T22:43:30 eu-central-1    proman-versioning
2022-11-14T22:47:15 eu-north-1  classixclustering
2022-11-14T23:48:44 eu-west-3   lusid-sdk-preview
2022-11-14T23:48:52 us-west-2   lusid-sdk-preview
2022-11-14T23:48:52 eu-west-1   lusid-sdk-preview
2022-11-14T23:48:54 eu-north-1  lusid-sdk-preview
2022-11-14T23:48:54 eu-south-1  lusid-sdk-preview
2022-11-14T23:48:54 eu-west-2   lusid-sdk-preview
2022-11-14T23:49:06 eu-central-1    lusid-sdk-preview
2022-11-14T23:49:11 ap-southeast-2  lusid-sdk-preview
2022-11-14T23:49:20 ap-south-1  lusid-sdk-preview
2022-11-14T23:49:22 us-east-2   lusid-sdk-preview
2022-11-14T23:49:24 ap-southeast-1  lusid-sdk-preview
2022-11-14T23:49:25 ap-northeast-1  lusid-sdk-preview
2022-11-14T23:49:36 us-east-1   lusid-sdk-preview
2022-11-15T00:27:06 ap-southeast-1  constructs
2022-11-15T00:29:11 ap-northeast-1  aws-cdk-asset-node-proxy-agent-v5
2022-11-15T00:32:53 ap-southeast-2  pyrampl
2022-11-15T00:34:07 ap-southeast-2  cdk8s
2022-11-15T00:34:13 ap-south-1  cdk8s
2022-11-15T00:34:14 ap-northeast-1  cdk8s
2022-11-15T00:34:17 ap-south-1  cdk-aurora-globaldatabase
2022-11-15T00:34:37 eu-north-1  apache-airflow-providers-airbyte
2022-11-15T00:34:37 eu-north-1  apache-airflow-providers-alibaba
2022-11-15T00:34:37 eu-north-1  apache-airflow-providers-amazon
2022-11-15T00:34:37 eu-north-1  apache-airflow-providers-apache-beam
2022-11-15T00:34:37 eu-north-1  apache-airflow-providers-apache-cassandra
2022-11-15T00:34:37 eu-north-1  apache-airflow-providers-apache-drill
2022-11-15T00:34:37 eu-north-1  apache-airflow-providers-apache-druid
2022-11-15T00:34:37 eu-north-1  apache-airflow-providers-apache-hdfs
2022-11-15T00:34:37 eu-north-1  apache-airflow-providers-apache-hive
2022-11-15T00:34:46 eu-south-1  cdktf-tf-module-stack
2022-11-15T00:34:56 ap-southeast-2  cdktf-tf-module-stack
2022-11-15T00:35:31 eu-west-3   pepperize-cdk-organizations
2022-11-15T00:35:35 eu-north-1  aws-sam-cli
2022-11-15T00:35:36 eu-north-1  pepperize-cdk-organizations
2022-11-15T00:35:39 eu-south-1  aws-sam-cli
2022-11-15T00:35:40 eu-south-1  pepperize-cdk-organizations
2022-11-15T00:38:26 eu-west-3   cdk-spot-one
2022-11-15T02:18:00 eu-south-1  lmfaooofiji123
2022-11-15T02:18:10 ap-southeast-2  lmfaooofiji123
2022-11-15T02:18:10 ap-southeast-1  lmfaooofiji123
2022-11-15T02:18:10 us-east-1   lmfaooofiji123
2022-11-15T02:18:12 eu-central-1    lmfaooofiji123
2022-11-15T02:18:18 ap-south-1  lmfaooofiji123
2022-11-15T02:18:23 ap-northeast-1  lmfaooofiji123
2022-11-15T02:18:33 us-east-2   lmfaooofiji123
2022-11-15T02:18:44 eu-west-3   lmfaooofiji123
2022-11-15T02:18:50 eu-north-1  lmfaooofiji123
2022-11-15T02:18:52 eu-west-1   lmfaooofiji123
2022-11-15T02:18:56 eu-west-2   lmfaooofiji123
2022-11-15T02:18:59 us-west-2   lmfaooofiji123
2022-11-15T03:21:16 eu-west-3   odoo14-addon-cooperator
2022-11-15T03:32:12 eu-west-3   tencentcloud-sdk-python-batch
2022-11-15T03:32:22 eu-north-1  tencentcloud-sdk-python-batch
2022-11-15T03:32:24 us-east-1   tencentcloud-sdk-python-batch
2022-11-15T03:32:25 us-east-2   tencentcloud-sdk-python-batch
2022-11-15T03:32:26 us-west-2   tencentcloud-sdk-python-batch
2022-11-15T03:32:27 eu-west-1   tencentcloud-sdk-python-batch
2022-11-15T03:32:28 eu-west-2   tencentcloud-sdk-python-batch
2022-11-15T03:32:33 eu-south-1  tencentcloud-sdk-python-batch
2022-11-15T03:32:38 ap-southeast-1  tencentcloud-sdk-python-batch
2022-11-15T03:32:41 ap-south-1  tencentcloud-sdk-python-batch
2022-11-15T03:32:43 ap-southeast-2  tencentcloud-sdk-python-batch
2022-11-15T03:32:45 eu-central-1    tencentcloud-sdk-python-batch
2022-11-15T03:32:55 ap-northeast-1  tencentcloud-sdk-python-batch
2022-11-15T09:09:08 eu-south-1  autogluon-common
2022-11-15T09:09:17 eu-west-3   autogluon-common
2022-11-15T09:09:17 ap-south-1  autogluon-common
2022-11-15T09:09:19 us-west-2   autogluon-common
2022-11-15T09:09:25 eu-west-1   autogluon-common
2022-11-15T09:09:30 ap-southeast-2  autogluon-common
2022-11-15T09:09:30 eu-central-1    autogluon-common
2022-11-15T09:09:30 eu-north-1  autogluon-common
2022-11-15T09:09:34 ap-northeast-1  autogluon-common
2022-11-15T09:09:35 ap-northeast-1  autogluon-text
2022-11-15T09:09:40 eu-west-2   autogluon-common
2022-11-15T09:09:42 us-east-2   autogluon-common
2022-11-15T09:10:04 ap-southeast-1  autogluon-common
2022-11-15T09:11:35 us-east-1   autogluon-common
2022-11-15T09:13:29 eu-north-1  wltp-jrshift
2022-11-15T09:32:05 eu-west-3   arvan-cli
2022-11-15T09:42:51 ap-southeast-1  clip-as-service
2022-11-15T09:51:14 ap-northeast-1  tf-nightly
2022-11-15T09:52:14 ap-northeast-1  tf-nightly-gpu
2022-11-15T10:05:10 ap-northeast-1  new-test-new-new
2022-11-15T10:20:35 ap-southeast-1  tf-nightly-gpu
2022-11-15T10:24:32 ap-southeast-1  tf-nightly-cpu-aws
2022-11-15T10:57:33 eu-west-3   envd
2022-11-15T11:07:36 eu-north-1  tf-nightly
2022-11-15T11:21:15 ap-south-1  pepperize-cdk-ssm-parameters-cross-region
2022-11-15T11:23:03 ap-southeast-1  c2cciutils
2022-11-15T11:29:25 eu-north-1  tid-pec2-model
2022-11-15T11:30:01 ap-southeast-1  scrunch
2022-11-15T11:40:18 eu-west-3   samna
2022-11-15T12:16:15 ap-southeast-2  chellow
2022-11-15T12:24:00 eu-north-1  bigdl-friesian
2022-11-15T12:24:01 eu-west-1   bigdl-chronos
2022-11-15T12:24:39 ap-southeast-1  bigdl-friesian-spark2
2022-11-15T12:24:47 eu-south-1  kadap-test
2022-11-15T12:55:32 ap-south-1  bigdl-dllib-spark2
2022-11-15T12:55:33 eu-south-1  bigdl-dllib-spark2
2022-11-15T12:55:45 eu-north-1  bigdl-friesian-spark2
2022-11-15T13:25:15 ap-south-1  gdata-vaas
2022-11-15T13:25:31 eu-north-1  nbdev-pandas
2022-11-15T13:31:59 eu-west-2   tensorflow-intel
2022-11-15T13:56:59 ap-southeast-1  prophecy-build-tool-test
2022-11-15T14:04:18 eu-west-3   pyroomacoustics
2022-11-15T14:11:12 eu-north-1  ultracart-rest-sdk
2022-11-15T14:25:18 ap-northeast-1  easy-migration-tools
2022-11-15T14:26:43 ap-south-1  eea-geolocation
2022-11-15T14:36:06 eu-west-3   girder-hashsum-download
2022-11-15T15:13:22 ap-south-1  module-qlr
2022-11-15T15:19:51 ap-southeast-2  movepy
2022-11-15T15:19:55 ap-northeast-1  movepy
2022-11-15T15:27:44 eu-west-1   launchpy
2022-11-15T15:27:48 eu-west-3   launchpy
2022-11-15T15:28:45 eu-west-2   launchpy
2022-11-15T15:28:55 ap-southeast-2  dbnomics-fetcher-ops
2022-11-15T15:29:42 ap-south-1  launchpy
2022-11-15T15:29:50 ap-southeast-1  sukiyou-sql
2022-11-15T15:29:51 ap-southeast-1  launchpy
2022-11-15T15:41:45 ap-northeast-1  cybrid-api-organization-python
2022-11-15T15:41:45 ap-northeast-1  data-flow-diagram
2022-11-15T15:42:18 eu-south-1  cybrid-api-id-python
2022-11-15T15:42:38 eu-west-2   sfkit
2022-11-15T15:42:38 eu-west-1   sfkit
2022-11-15T15:42:40 ap-south-1  sfkit
2022-11-15T15:47:45 eu-north-1  dansplotcore
2022-11-15T15:48:30 us-east-1   urllib3
2022-11-15T15:48:33 ap-south-1  urllib3
2022-11-15T15:48:34 eu-west-1   urllib3
2022-11-15T15:48:35 eu-west-2   urllib3
2022-11-15T15:48:36 eu-west-3   urllib3
2022-11-15T15:48:41 ap-northeast-1  urllib3
2022-11-15T15:48:42 eu-north-1  urllib3
2022-11-15T15:48:45 us-west-2   urllib3
2022-11-15T15:48:48 ap-southeast-1  urllib3
2022-11-15T15:48:52 ap-southeast-2  urllib3
2022-11-15T15:48:57 eu-central-1    urllib3
2022-11-15T15:49:08 us-east-2   urllib3
2022-11-15T15:49:16 eu-south-1  urllib3
2022-11-15T16:10:20 ap-south-1  ciecplib
2022-11-15T16:10:20 ap-south-1  ask-robot
2022-11-15T16:10:27 eu-west-2   sfkit
2022-11-15T16:12:28 eu-west-1   pydrumscore
2022-11-15T16:14:25 eu-west-2   jnunes-commons
2022-11-15T16:20:41 ap-southeast-2  jnunes-commons
2022-11-15T16:20:42 eu-central-1    jnunes-commons
2022-11-15T16:21:15 ap-south-1  csle-collector
2022-11-15T16:33:23 eu-north-1  kedaofx
2022-11-15T16:50:21 ap-southeast-1  libertydreamer
2022-11-15T16:52:04 ap-south-1  assemblyline
2022-11-15T16:52:05 ap-south-1  assemblyline-core
2022-11-15T16:52:15 eu-north-1  neosctl
2022-11-15T17:33:31 eu-south-1  spanishconjugator
2022-11-15T17:33:42 us-east-1   spanishconjugator
2022-11-15T17:33:42 ap-south-1  spanishconjugator
2022-11-15T17:33:47 eu-west-1   spanishconjugator
2022-11-15T17:33:49 eu-west-2   spanishconjugator
2022-11-15T17:33:50 eu-west-3   spanishconjugator
2022-11-15T17:33:53 eu-north-1  spanishconjugator
2022-11-15T17:33:54 ap-northeast-1  spanishconjugator
2022-11-15T17:34:00 ap-southeast-1  spanishconjugator
2022-11-15T17:34:00 us-west-2   spanishconjugator
2022-11-15T17:34:07 eu-central-1    spanishconjugator
2022-11-15T17:34:09 ap-southeast-2  spanishconjugator
2022-11-15T17:34:14 us-east-2   spanishconjugator
2022-11-15T18:06:43 ap-southeast-1  pynews
2022-11-15T18:40:26 eu-north-1  neon-skill-alerts
2022-11-15T18:58:53 eu-south-1  assemblyline-core
2022-11-15T20:23:16 eu-south-1  aioshelly
2022-11-15T21:39:03 eu-west-3   gs-apps
2022-11-15T21:39:08 ap-southeast-1  gs-engine
2022-11-15T21:43:01 eu-west-3   q3dfit
2022-11-15T23:16:26 eu-north-1  tencentcloud-sdk-python-ba
2022-11-15T23:39:07 ap-southeast-1  gs-include
2022-11-15T23:41:05 ap-southeast-1  silabs-mltk
2022-11-16T00:23:49 eu-north-1  cdk-skylight
2022-11-16T00:25:04 eu-west-2   constructs
2022-11-16T00:28:47 ap-southeast-1  fabric-am-handlers
2022-11-16T00:29:59 eu-west-3   assisted-service-client
2022-11-16T00:30:04 eu-west-1   tencentcloud-sdk-python-smh
2022-11-16T00:31:48 ap-southeast-1  cdk8s
2022-11-16T00:33:46 ap-southeast-1  cdktf-tf-module-stack
2022-11-16T00:37:19 ap-southeast-2  cdk8s-image
2022-11-16T00:40:07 us-west-2   cdk-certbot-dns-route53
2022-11-16T00:40:09 ap-south-1  cdk-certbot-dns-route53
2022-11-16T00:40:17 ap-southeast-2  cdk-certbot-dns-route53
2022-11-16T00:40:18 eu-central-1    cdk-certbot-dns-route53
2022-11-16T00:40:39 us-east-1   cdk-certbot-dns-route53
2022-11-16T00:40:41 eu-north-1  cdk-certbot-dns-route53
2022-11-16T00:40:42 ap-southeast-1  cdk-certbot-dns-route53
2022-11-16T00:40:52 eu-west-3   cdk-certbot-dns-route53
2022-11-16T00:40:53 us-east-2   cdk-certbot-dns-route53
2022-11-16T00:40:56 ap-northeast-1  cdk-certbot-dns-route53
2022-11-16T00:40:58 eu-west-2   cdk-certbot-dns-route53
2022-11-16T00:40:59 eu-west-1   cdk-certbot-dns-route53
2022-11-16T00:41:02 eu-south-1  cdk-certbot-dns-route53
2022-11-16T00:46:15 ap-southeast-2  tencentcloud-sdk-python-tcaplusdb
2022-11-16T00:50:06 ap-south-1  tencentcloud-sdk-python-tdid
2022-11-16T00:51:13 ap-southeast-2  assisted-service-client
2022-11-16T00:55:01 ap-south-1  tencentcloud-sdk-python-thpc
2022-11-16T00:56:15 ap-southeast-2  tencentcloud-sdk-python-tmt
2022-11-16T00:58:03 ap-south-1  tencentcloud-sdk-python-yunjing
2022-11-16T06:36:45 eu-west-3   big-thing-py
2022-11-16T06:58:07 eu-north-1  ros-cdk-fnf
2022-11-16T07:29:21 ap-southeast-1  cdktf-cdktf-provider-gitlab
2022-11-16T07:30:22 eu-west-3   cdktf-cdktf-provider-newrelic
2022-11-16T07:35:03 ap-northeast-1  cdktf-cdktf-provider-spotinst
2022-11-16T07:35:17 ap-southeast-1  ros-cdk-maxcompute
2022-11-16T07:45:13 eu-west-3   acceldata-airflow-sdk
2022-11-16T08:40:50 ap-southeast-1  oasys1-esrf-extensions
2022-11-16T08:40:50 eu-west-3   oasys1-esrf-extensions
2022-11-16T08:44:28 ap-northeast-1  crypt-dir
2022-11-16T09:01:36 us-east-1   dask-cuda
2022-11-16T09:01:37 eu-west-2   dask-cuda
2022-11-16T09:01:40 ap-southeast-1  dask-cuda
2022-11-16T09:01:42 eu-west-3   dask-cuda
2022-11-16T09:01:43 eu-south-1  dask-cuda
2022-11-16T09:01:50 eu-west-1   dask-cuda
2022-11-16T09:02:00 us-west-2   dask-cuda
2022-11-16T09:02:01 ap-southeast-2  dask-cuda
2022-11-16T09:02:08 eu-central-1    dask-cuda
2022-11-16T09:02:10 eu-north-1  dask-cuda
2022-11-16T09:02:22 ap-northeast-1  dask-cuda
2022-11-16T09:02:24 ap-south-1  dask-cuda
2022-11-16T09:02:33 us-east-2   dask-cuda
2022-11-16T09:09:20 ap-northeast-1  autogluon-features
2022-11-16T09:09:29 us-east-2   dask-cuda
2022-11-16T09:09:31 us-east-1   dask-cuda
2022-11-16T09:09:32 eu-west-2   dask-cuda
2022-11-16T09:09:38 ap-southeast-1  dask-cuda
2022-11-16T09:09:42 eu-south-1  dask-cuda
2022-11-16T09:09:42 eu-west-3   dask-cuda
2022-11-16T09:09:46 eu-west-1   dask-cuda
2022-11-16T09:09:47 eu-west-1   autogluon-timeseries
2022-11-16T09:09:57 us-west-2   dask-cuda
2022-11-16T09:09:59 ap-southeast-2  dask-cuda
2022-11-16T09:10:06 eu-central-1    dask-cuda
2022-11-16T09:10:07 eu-north-1  dask-cuda
2022-11-16T09:10:20 ap-northeast-1  dask-cuda
2022-11-16T09:10:20 ap-south-1  dask-cuda
2022-11-16T09:10:42 eu-west-3   general-calculator-zsd
2022-11-16T09:32:49 us-west-2   spanishconjugator
2022-11-16T09:32:50 ap-southeast-2  spanishconjugator
2022-11-16T09:32:54 eu-north-1  spanishconjugator
2022-11-16T09:33:00 eu-central-1    spanishconjugator
2022-11-16T09:33:08 ap-northeast-1  spanishconjugator
2022-11-16T09:33:10 ap-south-1  spanishconjugator
2022-11-16T09:33:14 us-east-2   spanishconjugator
2022-11-16T09:33:21 eu-west-2   spanishconjugator
2022-11-16T09:33:22 us-east-1   spanishconjugator
2022-11-16T09:33:26 ap-southeast-1  spanishconjugator
2022-11-16T09:33:30 eu-south-1  spanishconjugator
2022-11-16T09:33:32 eu-west-3   spanishconjugator
2022-11-16T09:33:35 eu-west-1   spanishconjugator
2022-11-16T09:42:08 ap-south-1  basepage-qhms
2022-11-16T09:56:57 ap-northeast-1  tf-nightly
2022-11-16T09:57:00 ap-south-1  tf-nightly
2022-11-16T10:22:14 eu-west-1   pyagrum-nightly
2022-11-16T10:22:28 us-west-2   pyagrum-nightly
2022-11-16T10:22:29 ap-southeast-2  pyagrum-nightly
2022-11-16T10:22:33 eu-north-1  pyagrum-nightly
2022-11-16T10:22:42 eu-central-1    pyagrum-nightly
2022-11-16T10:22:47 ap-northeast-1  pyagrum-nightly
2022-11-16T10:22:47 ap-south-1  pyagrum-nightly
2022-11-16T10:22:50 us-east-2   pyagrum-nightly
2022-11-16T10:22:59 us-east-1   pyagrum-nightly
2022-11-16T10:23:04 eu-west-2   pyagrum-nightly
2022-11-16T10:23:05 ap-southeast-1  pyagrum-nightly
2022-11-16T10:23:05 eu-south-1  pyagrum-nightly
2022-11-16T10:23:10 eu-west-3   pyagrum-nightly
2022-11-16T10:28:43 ap-northeast-1  ldrb
2022-11-16T10:43:01 eu-west-3   simpleworkspace
2022-11-16T10:54:56 eu-west-3   cat-win
2022-11-16T10:55:29 eu-central-1    cat-win
2022-11-16T10:55:31 ap-south-1  tf-nightly
2022-11-16T10:55:36 ap-northeast-1  cat-win
2022-11-16T10:55:36 ap-northeast-1  rcclab-gui
2022-11-16T10:55:36 ap-northeast-1  persistent
2022-11-16T10:55:45 ap-southeast-1  tf-nightly
2022-11-16T10:55:49 eu-west-2   persistent
2022-11-16T10:55:49 eu-west-2   tf-nightly
2022-11-16T10:55:49 eu-west-2   rcclab-gui
2022-11-16T10:55:52 eu-south-1  rcclab-gui
2022-11-16T10:55:52 eu-south-1  persistent
2022-11-16T10:55:52 eu-south-1  tf-nightly
2022-11-16T10:55:53 eu-south-1  cat-win
2022-11-16T10:55:55 eu-west-3   persistent
2022-11-16T10:55:55 eu-west-3   tf-nightly
2022-11-16T10:56:18 eu-north-1  cat-win
2022-11-16T10:56:18 eu-north-1  rcclab-gui
2022-11-16T10:56:18 eu-north-1  persistent
2022-11-16T10:56:18 eu-north-1  tf-nightly
2022-11-16T10:56:36 ap-northeast-1  sqlalchemy-iris
2022-11-16T10:56:46 ap-southeast-1  sqlalchemy-iris
2022-11-16T10:56:56 eu-west-3   sqlalchemy-iris
2022-11-16T11:42:38 eu-west-3   dials-data
2022-11-16T11:42:45 eu-west-1   dials-data
2022-11-16T11:42:52 eu-north-1  dials-data
2022-11-16T11:42:58 ap-southeast-2  dials-data
2022-11-16T11:43:00 us-west-2   dials-data
2022-11-16T11:43:07 ap-south-1  dials-data
2022-11-16T11:43:07 us-east-2   dials-data
2022-11-16T11:43:08 eu-central-1    dials-data
2022-11-16T11:43:16 ap-northeast-1  dials-data
2022-11-16T11:43:22 ap-southeast-1  dials-data
2022-11-16T11:43:24 us-east-1   dials-data
2022-11-16T11:43:27 eu-west-2   dials-data
2022-11-16T11:43:31 eu-south-1  dials-data
2022-11-16T11:51:34 eu-west-3   localstack-ext
2022-11-16T12:23:48 ap-south-1  flywheel-sdk
2022-11-16T12:24:02 ap-southeast-1  flywheel-sdk
2022-11-16T12:24:20 eu-west-3   flywheel-sdk
2022-11-16T12:24:38 ap-southeast-2  flywheel-sdk
2022-11-16T12:25:29 eu-south-1  skoufas-dbf-reader
2022-11-16T12:25:29 eu-south-1  apify-client
2022-11-16T12:26:17 eu-west-2   apify-client
2022-11-16T12:26:17 eu-west-2   esdn-sensing
2022-11-16T12:26:47 eu-north-1  esdn-sensing
2022-11-16T12:26:48 eu-north-1  authelio-sdk
2022-11-16T12:26:48 eu-north-1  cloudspot-erp-api
2022-11-16T12:26:48 eu-north-1  helper-utils
2022-11-16T12:26:57 ap-southeast-2  esdn-sensing
2022-11-16T12:26:58 ap-southeast-2  helper-utils
2022-11-16T12:26:58 ap-southeast-2  pepperize-cdk-route53-health-check
2022-11-16T12:26:58 eu-central-1    esdn-sensing
2022-11-16T12:26:59 eu-central-1    helper-utils
2022-11-16T12:26:59 eu-central-1    pepperize-cdk-route53-health-check
2022-11-16T12:27:05 eu-west-2   helper-utils
2022-11-16T12:27:08 ap-south-1  esdn-sensing
2022-11-16T12:27:09 ap-south-1  helper-utils
2022-11-16T12:27:09 ap-south-1  pepperize-cdk-route53-health-check
2022-11-16T12:27:20 ap-southeast-1  authelio-sdk
2022-11-16T12:27:20 ap-southeast-1  cloudspot-erp-api
2022-11-16T12:27:21 ap-southeast-1  helper-utils
2022-11-16T12:27:31 eu-west-3   cloudspot-erp-api
2022-11-16T12:27:32 eu-west-3   helper-utils
2022-11-16T12:27:32 eu-west-3   authelio-sdk
2022-11-16T12:28:00 ap-southeast-1  certora-cli-alpha-master
2022-11-16T12:28:00 ap-southeast-1  lzkyyds
2022-11-16T12:28:04 ap-northeast-1  flywheel-sdk
2022-11-16T12:28:06 eu-west-2   certora-cli-alpha-master
2022-11-16T12:28:18 ap-northeast-1  helper-utils
2022-11-16T12:28:19 eu-west-3   lzkyyds
2022-11-16T12:28:19 eu-west-3   certora-cli-alpha-master
2022-11-16T12:28:19 eu-west-3   remotemanager
2022-11-16T12:28:28 eu-west-1   remotemanager
2022-11-16T12:28:37 ap-southeast-2  remotemanager
2022-11-16T12:28:46 ap-south-1  remotemanager
2022-11-16T12:29:45 eu-central-1    odoo13-addon-stock-picking-invoice-link
2022-11-16T12:32:17 eu-west-3   netgen-mesher
2022-11-16T12:49:11 eu-west-3   ksos
2022-11-16T13:03:30 ap-south-1  bridgecrew
2022-11-16T13:03:30 ap-south-1  odoo-addon-helpdesk-mgmt
2022-11-16T13:10:05 eu-west-3   trix
2022-11-16T13:19:28 eu-central-1    nornir-maze
2022-11-16T13:24:35 ap-southeast-1  jina
2022-11-16T13:25:36 ap-southeast-1  spu
2022-11-16T13:25:37 ap-northeast-1  spu
2022-11-16T13:30:39 eu-west-2   mlfoundry
2022-11-16T13:30:54 eu-west-3   aicli
2022-11-16T13:32:31 ap-southeast-1  nbdev-apl
2022-11-16T14:01:21 ap-southeast-1  bigdl-friesian
2022-11-16T14:01:29 eu-south-1  bigdl-chronos
2022-11-16T14:01:44 eu-west-3   bigdl-dllib-spark2
2022-11-16T14:02:02 ap-south-1  bigdl-chronos-spark2
2022-11-16T14:02:02 ap-southeast-2  bigdl-chronos-spark2
2022-11-16T14:02:29 eu-south-1  renovosolutions-aws-cdk-one-time-event
2022-11-16T14:03:41 eu-west-3   tensorflow-intel
2022-11-16T14:15:52 ap-south-1  bigdl-dllib-spark3
2022-11-16T14:24:35 eu-west-3   survlime
2022-11-16T14:25:08 ap-southeast-1  spu
2022-11-16T14:39:02 ap-southeast-1  transfer-lmir
2022-11-16T15:16:47 us-east-1   ccxt
2022-11-16T15:16:54 ap-northeast-1  ccxt
2022-11-16T15:16:55 eu-south-1  ccxt
2022-11-16T15:17:09 eu-west-3   ccxt
2022-11-16T15:17:12 us-east-2   ccxt
2022-11-16T15:17:17 eu-west-1   ccxt
2022-11-16T15:17:19 ap-south-1  ccxt
2022-11-16T15:17:21 eu-north-1  ccxt
2022-11-16T15:17:22 ap-southeast-2  ccxt
2022-11-16T15:17:36 eu-central-1    ccxt
2022-11-16T15:17:39 us-west-2   ccxt
2022-11-16T15:17:46 ap-southeast-1  ccxt
2022-11-16T15:17:53 eu-west-2   ccxt
2022-11-16T15:23:51 ap-northeast-1  vkit-nightly
2022-11-16T15:30:19 ap-southeast-2  assemblyline-service-client
2022-11-16T15:30:19 ap-southeast-2  assemblyline-service-server
2022-11-16T15:39:13 ap-southeast-2  ccxt
2022-11-16T15:39:26 eu-central-1    ccxt
2022-11-16T15:39:32 us-west-2   ccxt
2022-11-16T15:39:37 ap-southeast-1  ccxt
2022-11-16T15:39:38 us-east-1   ccxt
2022-11-16T15:39:44 eu-west-2   ccxt
2022-11-16T15:39:44 ap-northeast-1  ccxt
2022-11-16T15:39:51 eu-south-1  ccxt
2022-11-16T15:39:57 eu-west-3   ccxt
2022-11-16T15:39:58 eu-west-3   spine-aws-common
2022-11-16T15:40:00 us-east-2   ccxt
2022-11-16T15:40:07 eu-west-1   ccxt
2022-11-16T15:40:08 eu-north-1  ccxt
2022-11-16T15:40:09 ap-south-1  ccxt
2022-11-16T15:45:35 ap-southeast-1  batchx
2022-11-16T15:45:35 ap-southeast-1  qontract-reconcile
2022-11-16T15:45:36 ap-southeast-1  shapelets-platform
2022-11-16T15:45:41 eu-west-2   qontract-reconcile
2022-11-16T15:45:42 ap-northeast-1  shapelets-platform
2022-11-16T15:45:42 ap-northeast-1  qontract-reconcile
2022-11-16T15:45:42 ap-northeast-1  batchx
2022-11-16T15:45:44 eu-south-1  shapelets-platform
2022-11-16T15:45:44 eu-south-1  batchx
2022-11-16T15:45:44 eu-south-1  qontract-reconcile
2022-11-16T15:45:55 eu-west-3   shapelets-platform
2022-11-16T15:45:56 eu-west-3   batchx
2022-11-16T15:45:56 eu-west-3   qontract-reconcile
2022-11-16T16:38:09 ap-southeast-1  pyssl-package
2022-11-16T16:39:38 ap-south-1  ipv64
2022-11-16T16:40:01 eu-central-1    pulumi-azure-native
2022-11-16T16:49:57 eu-central-1    enstools
2022-11-16T17:07:59 ap-southeast-1  pybd-gui
2022-11-16T17:08:20 eu-west-3   assemblyline-service-server
2022-11-16T17:08:29 ap-south-1  assemblyline
2022-11-16T17:08:31 eu-west-1   assemblyline-core
2022-11-16T17:08:32 eu-north-1  assemblyline-core
2022-11-16T17:09:01 ap-southeast-1  pypierrat
2022-11-16T17:10:29 eu-north-1  flywheel-sdk
2022-11-16T17:19:05 eu-west-2   sqreen
2022-11-16T17:31:59 eu-west-2   spanishconjugator
2022-11-16T17:32:05 us-east-2   spanishconjugator
2022-11-16T17:32:07 eu-west-3   spanishconjugator
2022-11-16T17:32:14 ap-south-1  spanishconjugator
2022-11-16T17:32:20 eu-west-1   spanishconjugator
2022-11-16T17:32:22 eu-north-1  spanishconjugator
2022-11-16T17:32:26 ap-southeast-2  spanishconjugator
2022-11-16T17:32:39 eu-central-1    spanishconjugator
2022-11-16T17:32:44 us-west-2   spanishconjugator
2022-11-16T17:32:47 ap-southeast-1  spanishconjugator
2022-11-16T17:32:48 us-east-1   spanishconjugator
2022-11-16T17:32:54 ap-northeast-1  spanishconjugator
2022-11-16T17:32:54 eu-south-1  spanishconjugator
2022-11-16T17:41:42 ap-southeast-1  aicli
2022-11-16T17:50:20 ap-southeast-2  sshreader
2022-11-16T17:58:42 ap-northeast-1  slackbot-helper
2022-11-16T18:15:19 eu-south-1  actions-paket-test-vol1234
2022-11-16T18:30:43 eu-west-3   assemblyline-ui
2022-11-16T18:30:44 eu-west-3   assemblyline-v4-service
2022-11-16T18:30:48 ap-southeast-2  assemblyline
2022-11-16T18:30:52 eu-north-1  assemblyline-core
2022-11-16T18:32:17 ap-southeast-1  pybanyan-openapi-client
2022-11-16T20:21:57 eu-west-3   homeassistant
2022-11-16T20:53:08 eu-south-1  mypy-boto3-personalize-events
2022-11-16T20:53:08 eu-south-1  mypy-boto3-proton
2022-11-16T20:53:08 eu-south-1  mypy-boto3-billingconductor
2022-11-16T21:36:57 ap-southeast-1  graphscope
2022-11-16T21:37:33 eu-north-1  gs-include
2022-11-16T21:53:26 eu-north-1  returnn
2022-11-16T22:27:18 ap-northeast-1  weconnect-cupra-one-edition
2022-11-16T23:11:27 ap-south-1  paasta-tools
2022-11-16T23:11:27 eu-west-3   paasta-tools
2022-11-16T23:11:39 ap-southeast-2  paasta-tools
2022-11-16T23:11:42 eu-west-2   paasta-tools
2022-11-16T23:11:51 eu-north-1  paasta-tools
2022-11-16T23:11:55 us-east-2   paasta-tools
2022-11-16T23:11:58 ap-northeast-1  paasta-tools
2022-11-16T23:12:01 eu-west-1   paasta-tools
2022-11-16T23:12:05 us-west-2   paasta-tools
2022-11-16T23:12:09 eu-south-1  paasta-tools
2022-11-16T23:12:10 us-east-1   paasta-tools
2022-11-16T23:12:13 eu-central-1    paasta-tools
2022-11-16T23:12:18 ap-southeast-1  paasta-tools
2022-11-16T23:12:26 eu-west-3   paasta-tools
2022-11-17T00:16:11 ap-southeast-2  cdk-fargate-run-task
2022-11-17T00:18:42 eu-south-1  tpds-application
2022-11-17T00:23:02 eu-west-3   cdk-lambda-bash
2022-11-17T00:30:57 ap-south-1  gs-engine
2022-11-17T00:30:58 eu-west-3   gs-engine
2022-11-17T00:33:02 ap-southeast-2  cluie
2022-11-17T00:40:51 eu-west-3   cdk-aurora-globaldatabase
2022-11-17T01:53:10 ap-southeast-1  python-rmap
2022-11-17T02:37:38 us-east-1   pyagrum-nightly
2022-11-17T02:37:43 us-west-2   pyagrum-nightly
2022-11-17T02:37:45 eu-central-1    pyagrum-nightly
2022-11-17T02:37:46 eu-south-1  pyagrum-nightly
2022-11-17T02:37:51 ap-southeast-1  pyagrum-nightly
2022-11-17T02:37:54 eu-west-3   pyagrum-nightly
2022-11-17T02:38:00 us-east-2   pyagrum-nightly
2022-11-17T02:38:06 ap-south-1  pyagrum-nightly
2022-11-17T02:38:15 ap-southeast-2  pyagrum-nightly
2022-11-17T02:38:16 eu-west-2   pyagrum-nightly
2022-11-17T02:38:26 eu-north-1  pyagrum-nightly
2022-11-17T02:38:30 ap-northeast-1  pyagrum-nightly
2022-11-17T02:38:34 eu-west-1   pyagrum-nightly
2022-11-17T03:07:40 eu-west-3   marqo
2022-11-17T04:52:06 ap-south-1  aesencriptionss
2022-11-17T05:00:28 ap-northeast-1  pulumi-rke
2022-11-17T05:00:45 ap-southeast-1  vcap-utils
2022-11-17T09:05:37 ap-southeast-2  csle-collector
2022-11-17T09:05:55 ap-southeast-1  keras-nightly
2022-11-17T09:31:52 eu-west-2   spanishconjugator
2022-11-17T09:32:01 eu-south-1  spanishconjugator
2022-11-17T09:32:06 us-west-2   spanishconjugator
2022-11-17T09:32:06 ap-south-1  spanishconjugator
2022-11-17T09:32:07 eu-central-1    spanishconjugator
2022-11-17T09:32:08 eu-west-1   spanishconjugator
2022-11-17T09:32:12 eu-west-3   spanishconjugator
2022-11-17T09:32:20 us-east-2   spanishconjugator
2022-11-17T09:32:20 us-east-1   spanishconjugator
2022-11-17T09:32:24 ap-southeast-2  spanishconjugator
2022-11-17T09:32:35 ap-northeast-1  spanishconjugator
2022-11-17T09:32:43 ap-southeast-1  spanishconjugator
2022-11-17T09:32:45 eu-north-1  spanishconjugator
2022-11-17T09:44:08 eu-west-3   python-rmap
2022-11-17T09:47:56 eu-south-1  tf-nightly-gpu
2022-11-17T09:48:37 ap-southeast-1  tf-nightly
2022-11-17T09:58:15 ap-southeast-2  tf-nightly
2022-11-17T10:04:14 ap-southeast-2  tf-nightly
2022-11-17T10:51:57 ap-southeast-2  alibabacloud-dcdn20180115
2022-11-17T10:59:00 ap-northeast-1  shapelets-platform
2022-11-17T11:08:00 ap-southeast-1  odoo10-addon-storage-image-product
2022-11-17T11:24:53 ap-southeast-1  community-layout
2022-11-17T11:35:40 ap-southeast-2  jina
2022-11-17T11:40:37 ap-southeast-2  mlpack
2022-11-17T11:54:40 ap-southeast-1  odoo-addon-account-payment-order
2022-11-17T12:06:57 eu-south-1  bigdl-serving
2022-11-17T12:07:58 eu-south-1  fbgemm-gpu-nightly
2022-11-17T12:19:29 ap-southeast-1  huaweicloudsdkantiddos
2022-11-17T12:22:30 eu-north-1  huaweicloudsdkcore
2022-11-17T12:24:28 ap-northeast-1  bigdl-orca-spark2
2022-11-17T12:24:30 eu-north-1  bigdl-friesian-spark2
2022-11-17T12:24:36 ap-southeast-1  bigdl-chronos-spark2
2022-11-17T12:28:28 ap-southeast-1  evdspy
2022-11-17T12:39:18 ap-northeast-1  bigdl-chronos-spark3
2022-11-17T12:46:42 eu-south-1  gt4sd
2022-11-17T12:46:43 us-west-2   gt4sd
2022-11-17T12:46:44 eu-central-1    gt4sd
2022-11-17T12:46:54 eu-west-1   gt4sd
2022-11-17T12:47:00 eu-west-3   gt4sd
2022-11-17T12:47:00 us-east-1   gt4sd
2022-11-17T12:47:08 ap-southeast-2  gt4sd
2022-11-17T12:47:13 ap-northeast-1  gt4sd
2022-11-17T12:47:17 ap-southeast-1  gt4sd
2022-11-17T12:47:18 eu-north-1  gt4sd
2022-11-17T12:47:30 us-east-2   gt4sd
2022-11-17T12:47:32 ap-south-1  gt4sd
2022-11-17T12:47:35 eu-west-2   gt4sd
2022-11-17T12:53:06 ap-southeast-2  shapelets-platform
2022-11-17T12:55:34 ap-south-1  bigdl-chronos
2022-11-17T13:11:05 ap-southeast-1  bigdl-orca-spark3
2022-11-17T13:11:10 eu-north-1  bigdl-friesian-spark3
2022-11-17T13:12:22 ap-south-1  maisaedu-poormans-dms
2022-11-17T13:12:22 ap-south-1  b4udata
2022-11-17T13:24:42 eu-west-3   nbdev-pytorch
2022-11-17T13:33:40 eu-west-3   pycounts-abthuy
2022-11-17T13:42:35 eu-west-3   zope-hookable
2022-11-17T13:59:23 eu-west-1   crtmlib
2022-11-17T13:59:26 eu-west-3   naveeeen
2022-11-17T13:59:26 eu-west-3   crtmlib
2022-11-17T14:00:08 eu-central-1    tf-nightly-cpu-aws
2022-11-17T14:00:08 eu-west-2   naveeeen
2022-11-17T14:00:08 eu-central-1    naveeeen
2022-11-17T14:00:08 eu-west-2   pypierrat
2022-11-17T14:00:08 eu-central-1    crtmlib
2022-11-17T14:00:08 eu-west-2   crtmlib
2022-11-17T14:00:09 eu-west-2   excalibrate
2022-11-17T14:00:38 ap-southeast-2  tf-nightly-cpu-aws
2022-11-17T14:00:38 ap-southeast-2  pypierrat
2022-11-17T14:00:42 ap-northeast-1  tf-nightly-cpu-aws
2022-11-17T14:00:45 ap-southeast-1  tf-nightly-cpu-aws
2022-11-17T14:00:45 ap-southeast-1  pypierrat
2022-11-17T14:00:49 eu-north-1  tf-nightly-cpu-aws
2022-11-17T14:00:49 eu-north-1  pypierrat
2022-11-17T14:00:59 ap-south-1  tf-nightly-cpu-aws
2022-11-17T14:00:59 ap-south-1  pypierrat
2022-11-17T14:02:08 eu-west-2   forecast-clarify
2022-11-17T14:02:20 eu-west-1   bettertui
2022-11-17T14:02:21 eu-west-1   forecast-clarify
2022-11-17T14:02:24 eu-west-3   forecast-clarify
2022-11-17T14:03:07 eu-west-2   biomaj-download
2022-11-17T14:03:07 eu-west-2   renovosolutions-aws-cdk-route53targets
2022-11-17T14:03:12 us-west-2   freelanceapi
2022-11-17T14:04:04 eu-central-1    freelanceapi
2022-11-17T14:04:04 eu-central-1    renovosolutions-aws-ses-smtp-credentials
2022-11-17T14:04:05 eu-central-1    randomlib
2022-11-17T14:04:07 eu-west-2   freelanceapi
2022-11-17T14:04:07 eu-west-2   renovosolutions-aws-ses-smtp-credentials
2022-11-17T14:04:07 eu-west-2   plone-outputfilters
2022-11-17T14:04:07 eu-west-2   randomlib
2022-11-17T14:04:20 eu-west-1   freelanceapi
2022-11-17T14:04:20 eu-west-1   renovosolutions-aws-ses-smtp-credentials
2022-11-17T14:04:21 eu-west-1   randomlib
2022-11-17T14:04:25 eu-west-3   randomlib
2022-11-17T14:05:05 eu-central-1    lusid-notifications-sdk-preview
2022-11-17T14:05:05 eu-central-1    renovosolutions-aws-cdk-aws-sso
2022-11-17T14:05:05 eu-central-1    renovosolutions-aws-cdk-renovo-s3-bucket
2022-11-17T14:05:23 eu-west-3   django-hawk
2022-11-17T14:05:24 eu-west-3   naveeeen
2022-11-17T14:05:41 ap-northeast-1  randomlib
2022-11-17T14:06:02 ap-south-1  randomlib
2022-11-17T14:06:04 ap-south-1  django-hawk
2022-11-17T14:06:06 eu-west-2   renovosolutions-aws-cdk-renovo-s3-bucket
2022-11-17T14:06:19 eu-west-1   naveeeen
2022-11-17T14:06:39 ap-northeast-1  renovosolutions-aws-cdk-aws-sso
2022-11-17T14:07:04 eu-central-1    django-hawk
2022-11-17T14:07:05 eu-central-1    naveeeen
2022-11-17T14:07:05 eu-central-1    renovosolutions-aws-cdk-managed-instance-role
2022-11-17T14:07:07 eu-west-2   camcops-server
2022-11-17T14:07:18 eu-west-1   renovosolutions-aws-cdk-managed-instance-role
2022-11-17T14:07:19 eu-west-1   chanfig
2022-11-17T14:07:39 ap-northeast-1  django-hawk
2022-11-17T14:07:40 ap-southeast-2  django-hawk
2022-11-17T14:07:51 eu-north-1  renovosolutions-aws-cdk-aspects-security-group
2022-11-17T14:07:51 ap-southeast-1  renovosolutions-aws-cdk-aspects-security-group
2022-11-17T14:07:56 ap-south-1  hpgcseismic
2022-11-17T14:19:07 eu-south-1  kfc-lib
2022-11-17T14:19:11 eu-west-1   mcstasscript
2022-11-17T14:31:08 eu-west-1   spanishconjugator
2022-11-17T14:31:10 us-east-1   spanishconjugator
2022-11-17T14:31:11 eu-west-3   spanishconjugator
2022-11-17T14:31:21 ap-southeast-2  spanishconjugator
2022-11-17T14:31:30 ap-southeast-1  spanishconjugator
2022-11-17T14:31:30 ap-northeast-1  spanishconjugator
2022-11-17T14:31:33 eu-north-1  spanishconjugator
2022-11-17T14:31:37 us-east-2   spanishconjugator
2022-11-17T14:31:44 ap-south-1  spanishconjugator
2022-11-17T14:31:50 eu-central-1    spanishconjugator
2022-11-17T14:31:56 eu-west-2   spanishconjugator
2022-11-17T14:31:57 us-west-2   spanishconjugator
2022-11-17T14:32:00 eu-south-1  spanishconjugator
2022-11-17T14:32:07 eu-west-1   spanishconjugator
2022-11-17T14:32:12 eu-west-3   spanishconjugator
2022-11-17T14:32:22 ap-southeast-2  spanishconjugator
2022-11-17T14:32:30 ap-southeast-1  spanishconjugator
2022-11-17T14:32:30 ap-northeast-1  spanishconjugator
2022-11-17T14:39:28 ap-southeast-1  gc-google-services-api
2022-11-17T15:00:09 ap-southeast-2  jina-now
2022-11-17T15:00:10 ap-southeast-2  redis-timeseries-manager
2022-11-17T15:00:48 eu-south-1  maisaedu-poormans-dms
2022-11-17T15:02:57 eu-west-3   ppnet
2022-11-17T15:22:24 ap-south-1  accesscontrol
2022-11-17T15:22:46 eu-west-1   zodbpickle
2022-11-17T15:22:47 eu-west-1   accesscontrol
2022-11-17T15:30:51 ap-southeast-2  pepperize-cdk-vpc
2022-11-17T15:31:25 ap-northeast-1  pepperize-cdk-vpc
2022-11-17T15:31:25 ap-northeast-1  atlantiscore
2022-11-17T15:33:28 ap-south-1  thesilentpypi
2022-11-17T15:33:28 ap-south-1  tonic
2022-11-17T15:34:14 ap-south-1  nlpbaselines
2022-11-17T15:35:40 eu-west-3   hpgcseismic
2022-11-17T15:35:45 eu-west-1   apibara
2022-11-17T15:35:45 eu-west-1   hpgcseismic
2022-11-17T15:35:51 eu-west-3   apibara
2022-11-17T15:36:36 eu-west-1   qontract-reconcile
2022-11-17T15:37:09 ap-northeast-1  scikit-build-core
2022-11-17T15:37:10 ap-northeast-1  sinabs
2022-11-17T15:37:11 eu-north-1  scikit-build-core
2022-11-17T15:37:11 eu-north-1  sinabs
2022-11-17T15:37:15 ap-south-1  scikit-build-core
2022-11-17T15:37:15 ap-south-1  sinabs
2022-11-17T16:07:38 ap-southeast-2  biofilm
2022-11-17T16:10:26 eu-west-3   tensorflow-text-nightly
2022-11-17T16:13:26 eu-west-3   assemblyline-v4-service
2022-11-17T17:04:53 us-west-2   smuthi
2022-11-17T17:33:31 eu-central-1    spanishconjugator
2022-11-17T17:33:36 eu-west-2   spanishconjugator
2022-11-17T17:33:37 eu-south-1  spanishconjugator
2022-11-17T17:33:42 us-west-2   spanishconjugator
2022-11-17T17:33:49 us-east-1   spanishconjugator
2022-11-17T17:33:50 eu-west-3   spanishconjugator
2022-11-17T17:33:51 eu-west-1   spanishconjugator
2022-11-17T17:33:59 ap-southeast-2  spanishconjugator
2022-11-17T17:34:01 us-east-2   spanishconjugator
2022-11-17T17:34:11 eu-north-1  spanishconjugator
2022-11-17T17:34:12 ap-northeast-1  spanishconjugator
2022-11-17T17:34:16 ap-southeast-1  spanishconjugator
2022-11-17T17:34:21 ap-south-1  spanishconjugator
2022-11-17T17:48:14 ap-south-1  qontract-reconcile
2022-11-17T17:59:46 ap-southeast-2  jina-now
2022-11-17T18:19:14 ap-south-1  stringsort
2022-11-17T19:20:53 ap-southeast-2  lzkyyds
2022-11-17T19:23:01 ap-southeast-1  dynascii
2022-11-17T19:37:46 ap-southeast-2  qt5-levelmeter
2022-11-17T20:28:24 ap-northeast-1  acc-provision
2022-11-17T21:39:00 ap-southeast-2  gs-include
2022-11-17T21:39:00 eu-west-2   gs-include
2022-11-17T21:43:50 us-west-2   coiled
2022-11-17T21:43:52 ap-southeast-1  coiled
2022-11-17T21:43:55 us-east-1   coiled
2022-11-17T21:43:56 eu-central-1    coiled
2022-11-17T21:43:56 ap-southeast-2  coiled
2022-11-17T21:43:57 eu-west-2   coiled
2022-11-17T21:44:14 eu-north-1  coiled
2022-11-17T21:44:25 eu-west-1   coiled
2022-11-17T21:44:25 eu-south-1  coiled
2022-11-17T21:44:41 eu-west-3   coiled
2022-11-17T21:44:46 ap-south-1  coiled
2022-11-17T21:45:13 us-east-2   coiled
2022-11-17T21:45:53 ap-northeast-1  coiled
2022-11-17T23:43:03 ap-northeast-1  moto
2022-11-18T00:14:45 ap-south-1  tencentcloud-sdk-python-afc
2022-11-18T00:17:47 ap-southeast-2  aim-ui
2022-11-18T00:25:46 ap-northeast-1  cdk-pipelines-github
2022-11-18T00:30:45 ap-southeast-2  tencentcloud-sdk-python-es
2022-11-18T00:35:43 eu-west-3   tencentcloud-sdk-python-habo
2022-11-18T00:36:43 ap-southeast-2  raindancers-network-raindancers-cdk
2022-11-18T00:46:30 ap-south-1  tencentcloud-sdk-python-pts
2022-11-18T00:51:37 eu-west-3   tencentcloud-sdk-python-taf
2022-11-18T00:52:36 eu-west-3   tencentcloud-sdk-python-tcbr
2022-11-18T02:44:02 eu-west-2   pyagrum-nightly
2022-11-18T02:44:28 eu-west-1   pyagrum-nightly
2022-11-18T02:44:30 eu-south-1  pyagrum-nightly
2022-11-18T02:44:32 eu-north-1  pyagrum-nightly
2022-11-18T02:44:32 us-east-1   pyagrum-nightly
2022-11-18T02:44:37 ap-northeast-1  pyagrum-nightly
2022-11-18T02:44:38 eu-central-1    pyagrum-nightly
2022-11-18T02:44:39 us-east-2   pyagrum-nightly
2022-11-18T02:44:41 ap-south-1  pyagrum-nightly
2022-11-18T02:44:42 ap-southeast-1  pyagrum-nightly
2022-11-18T02:44:49 ap-southeast-2  pyagrum-nightly
2022-11-18T02:44:51 us-west-2   pyagrum-nightly
2022-11-18T02:44:52 eu-west-3   pyagrum-nightly
2022-11-18T03:21:25 ap-southeast-1  kclvm
2022-11-18T05:31:36 ap-southeast-1  aws-cdk-aws-cloudtrail
2022-11-18T05:31:46 ap-south-1  aws-cdk-aws-codegurureviewer
2022-11-18T05:32:50 ap-south-1  aws-cdk-aws-emr
2022-11-18T05:32:54 ap-southeast-2  aws-cdk-aws-events-targets
2022-11-18T05:33:37 eu-central-1    aws-cdk-aws-iotsitewise
2022-11-18T05:33:41 ap-southeast-1  aws-cdk-aws-iotwireless
2022-11-18T05:33:47 ap-southeast-2  aws-cdk-aws-kinesis
2022-11-18T05:33:49 ap-south-1  aws-cdk-aws-kinesisanalytics
2022-11-18T05:34:54 ap-southeast-2  aws-cdk-aws-rds
2022-11-18T05:35:37 ap-southeast-1  aws-cdk-aws-ses
2022-11-18T05:35:49 ap-south-1  aws-cdk-aws-sso
2022-11-18T05:37:18 ap-northeast-1  aws-cdk-aws-autoscalingplans
2022-11-18T05:37:52 eu-north-1  aws-cdk-aws-codestarconnections
2022-11-18T05:37:52 eu-north-1  aws-cdk-aws-codestarnotifications
2022-11-18T05:38:03 ap-south-1  aws-cdk-aws-customerprofiles
2022-11-18T05:38:03 ap-south-1  aws-cdk-aws-databrew
2022-11-18T05:38:26 ap-northeast-1  aws-cdk-aws-ecr-assets
2022-11-18T05:38:26 ap-northeast-1  aws-cdk-aws-ecs
2022-11-18T05:38:30 eu-north-1  aws-cdk-aws-ecs-patterns
2022-11-18T05:38:31 eu-north-1  aws-cdk-aws-efs
2022-11-18T05:38:41 ap-south-1  aws-cdk-aws-elasticloadbalancingv2-actions
2022-11-18T05:38:41 ap-south-1  aws-cdk-aws-elasticloadbalancingv2-targets
2022-11-18T05:39:35 eu-north-1  aws-cdk-aws-kinesisanalytics
2022-11-18T05:39:36 eu-north-1  aws-cdk-aws-kinesisanalytics-flink
2022-11-18T05:39:52 ap-south-1  aws-cdk-aws-lambda-python
2022-11-18T05:39:52 ap-south-1  aws-cdk-aws-lex
2022-11-18T05:40:24 eu-south-1  aws-cdk-aws-opensearchservice
2022-11-18T05:40:45 ap-south-1  aws-cdk-aws-rolesanywhere
2022-11-18T05:40:45 ap-south-1  aws-cdk-aws-route53
2022-11-18T05:41:32 eu-west-1   aws-cdk-aws-transfer
2022-11-18T05:44:10 ap-northeast-1  odoo14-addon-purchase-cancel-reason
2022-11-18T08:06:23 eu-south-1  appstartup
2022-11-18T08:06:29 eu-north-1  appstartup
2022-11-18T08:06:30 eu-west-1   appstartup
2022-11-18T08:06:34 ap-south-1  appstartup
2022-11-18T08:06:35 eu-central-1    appstartup
2022-11-18T08:06:40 ap-southeast-1  appstartup
2022-11-18T08:06:50 eu-west-3   appstartup
2022-11-18T08:06:50 us-east-1   appstartup
2022-11-18T08:06:58 us-east-2   appstartup
2022-11-18T08:07:00 us-west-2   appstartup
2022-11-18T08:07:03 ap-northeast-1  appstartup
2022-11-18T08:07:05 ap-southeast-2  appstartup
2022-11-18T08:07:09 eu-west-2   appstartup
2022-11-18T09:05:08 ap-south-1  drf-rbac-plus
2022-11-18T09:22:18 eu-west-3   csle-collector
2022-11-18T09:45:25 ap-northeast-1  tff-nightly
2022-11-18T09:52:15 ap-southeast-2  tf-nightly-gpu
2022-11-18T10:07:38 ap-south-1  diffoscope
2022-11-18T10:30:28 ap-south-1  tf-nightly-cpu
2022-11-18T10:43:04 eu-west-2   apache-airflow-providers-influxdb
2022-11-18T10:43:04 eu-west-2   apache-airflow-providers-microsoft-azure
2022-11-18T10:43:04 eu-west-2   apache-airflow-providers-microsoft-psrp
2022-11-18T10:43:27 eu-west-1   apache-airflow-providers-presto
2022-11-18T10:43:28 eu-west-1   apache-airflow-providers-segment
2022-11-18T10:43:28 eu-west-1   apache-airflow-providers-slack
2022-11-18T10:44:04 eu-west-2   apache-airflow-providers-tabular
2022-11-18T10:44:05 eu-west-2   apache-airflow-providers-zendesk
2022-11-18T11:00:56 ap-northeast-1  djangorestframework-stubs
2022-11-18T12:10:28 ap-northeast-1  snapml
2022-11-18T12:24:13 ap-southeast-2  bigdl-chronos
2022-11-18T12:24:13 ap-southeast-2  bigdl
2022-11-18T12:28:07 eu-west-3   ezdxf
2022-11-18T12:40:06 ap-southeast-2  bigdl-friesian-spark3
2022-11-18T12:40:17 eu-south-1  bigdl-spark3
2022-11-18T12:40:22 ap-south-1  bigdl-nano
2022-11-18T12:56:13 ap-south-1  bigdl-dllib
2022-11-18T12:56:27 eu-north-1  bigdl-chronos
2022-11-18T12:56:28 ap-southeast-1  bigdl-chronos
2022-11-18T12:57:00 ap-southeast-2  bigdl-spark2
2022-11-18T13:11:01 ap-northeast-1  bigdl-spark3
2022-11-18T13:11:22 eu-north-1  pydantic-aiohttp
2022-11-18T13:52:35 ap-southeast-2  csle-collector
2022-11-18T14:13:33 ap-south-1  shyft-time-series
2022-11-18T14:15:09 eu-central-1    nanoindentation
2022-11-18T14:15:30 eu-west-3   nanoindentation
2022-11-18T14:15:30 ap-northeast-1  nanoindentation
2022-11-18T14:15:31 us-east-2   nanoindentation
2022-11-18T14:15:32 ap-south-1  nanoindentation
2022-11-18T14:15:36 eu-south-1  nanoindentation
2022-11-18T14:15:43 us-west-2   nanoindentation
2022-11-18T14:15:47 eu-west-2   nanoindentation
2022-11-18T14:15:49 ap-southeast-1  nanoindentation
2022-11-18T14:15:54 eu-north-1  nanoindentation
2022-11-18T14:15:56 us-east-1   nanoindentation
2022-11-18T14:15:56 eu-west-1   nanoindentation
2022-11-18T14:16:23 ap-southeast-2  nanoindentation
2022-11-18T14:18:52 eu-north-1  yadraw
2022-11-18T15:42:08 ap-southeast-1  bytetracker
2022-11-18T15:55:58 eu-south-1  libprobe
2022-11-18T16:14:51 ap-southeast-1  odoo13-addon-mail-activity-team
2022-11-18T16:29:17 eu-central-1    ccxt
2022-11-18T16:29:21 us-east-2   ccxt
2022-11-18T16:29:23 ap-southeast-2  ccxt
2022-11-18T16:29:24 ap-south-1  ccxt
2022-11-18T16:29:35 ap-northeast-1  ccxt
2022-11-18T16:29:40 ap-southeast-1  ccxt
2022-11-18T16:29:41 eu-south-1  ccxt
2022-11-18T16:29:43 eu-west-3   ccxt
2022-11-18T16:29:48 us-west-2   ccxt
2022-11-18T16:29:48 eu-west-2   ccxt
2022-11-18T16:29:52 eu-north-1  ccxt
2022-11-18T16:30:02 us-east-1   ccxt
2022-11-18T16:30:05 eu-west-1   ccxt
2022-11-18T17:32:57 ap-southeast-2  spanishconjugator
2022-11-18T17:33:03 ap-south-1  spanishconjugator
2022-11-18T17:33:07 ap-northeast-1  spanishconjugator
2022-11-18T17:33:09 eu-south-1  spanishconjugator
2022-11-18T17:33:13 ap-southeast-1  spanishconjugator
2022-11-18T17:33:20 us-west-2   spanishconjugator
2022-11-18T17:33:20 eu-west-3   spanishconjugator
2022-11-18T17:33:24 eu-north-1  spanishconjugator
2022-11-18T17:33:25 eu-west-2   spanishconjugator
2022-11-18T17:33:32 us-east-1   spanishconjugator
2022-11-18T17:33:38 eu-west-1   spanishconjugator
2022-11-18T17:33:47 us-east-2   spanishconjugator
2022-11-18T17:33:51 eu-central-1    spanishconjugator
2022-11-18T18:04:51 ap-south-1  cirq-web
2022-11-18T18:41:54 eu-west-3   large-image-converter
2022-11-18T18:42:09 eu-south-1  volttron
2022-11-18T18:42:13 eu-west-2   large-image-source-deepzoom
2022-11-18T18:42:52 eu-west-3   large-image-source-ometiff
2022-11-18T18:48:30 ap-south-1  acryl-datahub
2022-11-18T18:48:31 us-east-1   acryl-datahub
2022-11-18T18:48:38 ap-southeast-1  acryl-datahub
2022-11-18T18:48:46 ap-northeast-1  acryl-datahub
2022-11-18T18:48:46 ap-northeast-1  audiolm-pytorch
2022-11-18T18:48:48 eu-west-3   acryl-datahub
2022-11-18T18:48:51 eu-north-1  acryl-datahub
2022-11-18T18:48:52 us-west-2   acryl-datahub
2022-11-18T18:49:01 eu-south-1  acryl-datahub
2022-11-18T18:49:04 us-east-2   acryl-datahub
2022-11-18T18:49:05 eu-west-1   acryl-datahub
2022-11-18T18:49:10 eu-west-2   acryl-datahub
2022-11-18T18:49:20 eu-central-1    acryl-datahub
2022-11-18T18:49:24 ap-southeast-2  acryl-datahub
2022-11-18T20:04:18 eu-west-3   py-bdd-context
2022-11-18T20:07:01 ap-southeast-1  koya-python-airtable
2022-11-18T20:07:02 ap-south-1  koya-python-airtable
2022-11-18T20:11:06 ap-southeast-2  aim-ui
2022-11-18T20:12:16 eu-west-3   koya-python-airtable
2022-11-18T21:31:52 us-west-2   hat-event
2022-11-18T21:32:44 eu-west-3   mypy-boto3-appflow
2022-11-18T21:54:13 eu-central-1    mypy-boto3-stepfunctions
2022-11-18T21:54:31 ap-southeast-2  mypy-boto3-transfer
2022-11-18T21:56:13 ap-south-1  gs-coordinator
2022-11-18T21:56:17 ap-southeast-1  gs-engine
2022-11-18T22:12:27 ap-southeast-2  dagster-azure
2022-11-18T22:36:50 ap-northeast-1  mypy-boto3-dynamodbstreams
2022-11-18T22:36:51 ap-northeast-1  mypy-boto3-cloudfront
2022-11-18T22:36:58 eu-west-1   mypy-boto3-auditmanager
2022-11-18T22:37:04 ap-southeast-1  mypy-boto3-chime-sdk-voice
2022-11-18T22:48:12 ap-southeast-2  mypy-boto3-stepfunctions
2022-11-18T23:45:32 eu-west-1   mypy-boto3-servicecatalog
2022-11-18T23:55:43 ap-southeast-2  ypy-websocket
2022-11-19T00:32:02 ap-southeast-1  finance-hermes
2022-11-19T00:34:40 eu-west-3   cdk-ecs-service-extensions
2022-11-19T00:47:20 ap-southeast-2  mypy-boto3-glue
2022-11-19T02:02:09 ap-south-1  nonebot-plugin-zyk-novelai
2022-11-19T02:02:12 us-east-2   nonebot-plugin-zyk-novelai
2022-11-19T02:02:12 us-west-2   nonebot-plugin-zyk-novelai
2022-11-19T02:02:13 ap-northeast-1  nonebot-plugin-zyk-novelai
2022-11-19T02:02:18 ap-southeast-1  nonebot-plugin-zyk-novelai
2022-11-19T02:02:27 eu-west-2   nonebot-plugin-zyk-novelai
2022-11-19T02:02:29 eu-central-1    nonebot-plugin-zyk-novelai
2022-11-19T02:02:31 us-east-1   nonebot-plugin-zyk-novelai
2022-11-19T02:02:39 eu-west-1   nonebot-plugin-zyk-novelai
2022-11-19T02:02:46 ap-southeast-2  nonebot-plugin-zyk-novelai
2022-11-19T02:02:56 eu-south-1  nonebot-plugin-zyk-novelai
2022-11-19T02:03:01 eu-north-1  nonebot-plugin-zyk-novelai
2022-11-19T02:03:04 eu-west-3   nonebot-plugin-zyk-novelai
2022-11-19T02:49:22 eu-west-1   pyagrum-nightly
2022-11-19T02:49:28 ap-southeast-2  pyagrum-nightly
2022-11-19T02:49:40 eu-south-1  pyagrum-nightly
2022-11-19T02:49:46 eu-north-1  pyagrum-nightly
2022-11-19T02:49:47 ap-south-1  pyagrum-nightly
2022-11-19T02:49:50 eu-west-3   pyagrum-nightly
2022-11-19T02:49:54 ap-northeast-1  pyagrum-nightly
2022-11-19T02:49:54 us-east-2   pyagrum-nightly
2022-11-19T02:49:56 us-west-2   pyagrum-nightly
2022-11-19T02:49:58 ap-southeast-1  pyagrum-nightly
2022-11-19T02:50:08 us-east-1   pyagrum-nightly
2022-11-19T02:50:10 eu-central-1    pyagrum-nightly
2022-11-19T02:50:11 eu-west-2   pyagrum-nightly
2022-11-19T02:50:22 eu-west-1   pyagrum-nightly
2022-11-19T02:50:53 ap-northeast-1  pyagrum-nightly
2022-11-19T02:50:58 ap-southeast-1  pyagrum-nightly
2022-11-19T02:51:10 eu-central-1    pyagrum-nightly
2022-11-19T03:10:05 eu-west-2   pollination-daylight-factor
2022-11-19T03:10:14 eu-west-1   pollination-daylight-factor
2022-11-19T03:10:16 ap-southeast-2  pollination-daylight-factor
2022-11-19T03:10:32 eu-south-1  pollination-daylight-factor
2022-11-19T03:10:36 eu-north-1  pollination-daylight-factor
2022-11-19T03:10:39 ap-south-1  pollination-daylight-factor
2022-11-19T03:10:41 ap-northeast-1  pollination-daylight-factor
2022-11-19T03:10:42 eu-west-3   pollination-daylight-factor
2022-11-19T03:10:44 us-east-2   pollination-daylight-factor
2022-11-19T03:10:46 us-west-2   pollination-daylight-factor
2022-11-19T03:10:48 ap-southeast-1  pollination-daylight-factor
2022-11-19T03:10:55 us-east-1   pollination-daylight-factor
2022-11-19T03:11:02 eu-central-1    pollination-daylight-factor
2022-11-19T03:34:38 eu-west-3   mypy-boto3-sagemaker
2022-11-19T04:58:31 us-west-2   odoo14-addons-oca-management-system
2022-11-19T05:14:02 ap-southeast-1  odoo12-addon-partner-coc
2022-11-19T05:30:24 ap-southeast-2  odoo-addon-purchase-order-approved
2022-11-19T09:09:24 eu-south-1  autogluon-timeseries
2022-11-19T11:21:18 ap-south-1  lino-react
2022-11-19T11:21:22 eu-west-1   lino-react
2022-11-19T11:21:26 eu-south-1  lino-react
2022-11-19T11:21:30 ap-southeast-1  lino-react
2022-11-19T11:21:35 us-east-1   lino-react
2022-11-19T11:21:41 ap-northeast-1  lino-react
2022-11-19T11:21:45 eu-north-1  lino-react
2022-11-19T11:21:48 us-east-2   lino-react
2022-11-19T11:21:53 eu-west-3   lino-react
2022-11-19T11:22:06 eu-west-2   lino-react
2022-11-19T11:22:08 ap-southeast-2  lino-react
2022-11-19T11:22:09 us-west-2   lino-react
2022-11-19T11:22:11 eu-central-1    lino-react
2022-11-19T13:05:32 eu-west-2   pyvicare
2022-11-19T13:05:33 eu-central-1    pyvicare
2022-11-19T13:05:35 ap-south-1  pyvicare
2022-11-19T13:05:40 us-east-1   pyvicare
2022-11-19T13:05:41 eu-south-1  pyvicare
2022-11-19T13:05:44 ap-southeast-1  pyvicare
2022-11-19T13:05:48 eu-west-1   pyvicare
2022-11-19T13:05:55 ap-northeast-1  pyvicare
2022-11-19T13:06:10 eu-north-1  pyvicare
2022-11-19T13:06:11 us-east-2   pyvicare
2022-11-19T13:06:15 eu-west-3   pyvicare
2022-11-19T13:06:26 ap-southeast-2  pyvicare
2022-11-19T13:06:30 us-west-2   pyvicare
2022-11-19T14:05:47 eu-north-1  bigdl-dllib-spark2
2022-11-19T14:06:04 ap-southeast-2  bigdl-friesian-spark2
2022-11-19T14:06:09 ap-south-1  bigdl-chronos-spark2
2022-11-19T14:06:18 ap-southeast-1  bigdl-spark2
2022-11-19T17:30:18 eu-west-1   spanishconjugator
2022-11-19T17:30:22 us-east-2   spanishconjugator
2022-11-19T17:30:27 us-east-1   spanishconjugator
2022-11-19T17:30:29 eu-north-1  spanishconjugator
2022-11-19T17:30:38 eu-west-3   spanishconjugator
2022-11-19T17:30:43 eu-south-1  spanishconjugator
2022-11-19T17:30:43 ap-south-1  spanishconjugator
2022-11-19T17:30:45 ap-southeast-1  spanishconjugator
2022-11-19T17:30:47 ap-southeast-2  spanishconjugator
2022-11-19T17:30:55 eu-central-1    spanishconjugator
2022-11-19T17:30:59 ap-northeast-1  spanishconjugator
2022-11-19T17:31:00 us-west-2   spanishconjugator
2022-11-19T17:31:03 eu-west-2   spanishconjugator
2022-11-19T17:31:17 eu-west-1   spanishconjugator
2022-11-19T17:31:28 eu-north-1  spanishconjugator
2022-11-19T17:31:41 eu-south-1  spanishconjugator
2022-11-19T17:31:43 ap-south-1  spanishconjugator
2022-11-19T17:31:45 ap-southeast-1  spanishconjugator
2022-11-19T17:31:47 ap-southeast-2  spanishconjugator
2022-11-19T17:32:38 eu-west-3   spanishconjugator
2022-11-19T18:02:30 ap-southeast-1  kcli
2022-11-19T18:02:31 ap-south-1  kcli
2022-11-19T22:13:59 eu-north-1  my-math-by
2022-11-20T00:14:33 eu-central-1    cdk8s
2022-11-20T00:14:37 us-west-2   cdk8s
2022-11-20T00:14:46 ap-northeast-1  cdk8s
2022-11-20T00:14:55 eu-south-1  cdk8s
2022-11-20T00:14:56 eu-west-2   cdk8s
2022-11-20T00:14:57 us-east-1   cdk8s
2022-11-20T00:14:59 eu-west-1   cdk8s
2022-11-20T00:15:04 ap-south-1  cdk8s
2022-11-20T00:15:07 ap-southeast-1  cdk8s
2022-11-20T00:15:11 ap-southeast-2  cdk8s
2022-11-20T00:15:15 eu-west-3   cdk8s
2022-11-20T00:15:15 us-east-2   cdk8s
2022-11-20T00:15:21 eu-north-1  cdk8s
2022-11-20T00:26:08 ap-southeast-2  constructs
2022-11-20T02:47:20 eu-north-1  pyagrum-nightly
2022-11-20T02:47:21 eu-west-3   pyagrum-nightly
2022-11-20T06:36:13 eu-north-1  fake-bpy-module-latest
2022-11-20T06:36:22 ap-southeast-1  fake-bpy-module-latest
2022-11-20T06:36:23 eu-central-1    fake-bpy-module-latest
2022-11-20T06:36:25 ap-south-1  fake-bpy-module-latest
2022-11-20T06:36:35 us-east-1   fake-bpy-module-latest
2022-11-20T06:36:42 eu-south-1  fake-bpy-module-latest
2022-11-20T06:36:45 ap-southeast-2  fake-bpy-module-latest
2022-11-20T06:36:46 ap-northeast-1  fake-bpy-module-latest
2022-11-20T06:36:46 eu-west-2   fake-bpy-module-latest
2022-11-20T06:36:49 eu-west-1   fake-bpy-module-latest
2022-11-20T06:37:00 eu-west-3   fake-bpy-module-latest
2022-11-20T06:37:03 us-west-2   fake-bpy-module-latest
2022-11-20T06:37:05 us-east-2   fake-bpy-module-latest
2022-11-20T12:24:48 eu-west-2   bigdl-spark2
2022-11-20T12:39:17 eu-central-1    bigdl-nano
2022-11-20T12:45:12 eu-north-1  season
2022-11-20T12:55:09 eu-central-1    bigdl-friesian
2022-11-20T12:55:40 ap-northeast-1  bigdl-friesian-spark2
2022-11-20T12:55:41 ap-south-1  bigdl-chronos-spark2
2022-11-20T12:55:42 ap-southeast-1  bigdl-chronos-spark2
2022-11-20T12:55:46 eu-west-3   bigdl-spark2
2022-11-20T16:16:48 eu-north-1  os-android-app-automation
2022-11-20T16:55:35 eu-north-1  tf-nightly-intel
2022-11-20T18:54:27 us-west-2   moto
2022-11-20T19:29:23 us-east-2   lusid-sdk-preview
2022-11-20T19:29:27 eu-west-2   lusid-sdk-preview
2022-11-20T19:29:33 eu-west-3   lusid-sdk-preview
2022-11-20T19:29:41 ap-northeast-1  lusid-sdk-preview
2022-11-20T19:29:41 ap-southeast-2  lusid-sdk-preview
2022-11-20T19:29:42 eu-north-1  lusid-sdk-preview
2022-11-20T19:29:44 eu-central-1    lusid-sdk-preview
2022-11-20T19:29:46 ap-southeast-1  lusid-sdk-preview
2022-11-20T19:29:50 ap-south-1  lusid-sdk-preview
2022-11-20T19:29:54 us-east-1   lusid-sdk-preview
2022-11-20T19:30:10 us-west-2   lusid-sdk-preview
2022-11-20T19:30:14 eu-south-1  lusid-sdk-preview
2022-11-20T19:30:16 eu-west-1   lusid-sdk-preview
2022-11-20T23:45:31 us-west-2   randseal
2022-11-21T00:25:56 eu-north-1  bambuvizitor
2022-11-21T04:41:00 ap-southeast-2  pulp-rpm-client
2022-11-21T04:41:02 us-east-2   pulp-rpm-client
2022-11-21T04:41:04 eu-west-2   pulp-rpm-client
2022-11-21T04:41:07 ap-south-1  pulp-rpm-client
2022-11-21T04:41:09 us-east-1   pulp-rpm-client
2022-11-21T04:41:18 eu-west-3   pulp-rpm-client
2022-11-21T04:41:19 eu-central-1    pulp-rpm-client
2022-11-21T04:41:22 eu-north-1  pulp-rpm-client
2022-11-21T04:41:29 ap-northeast-1  pulp-rpm-client
2022-11-21T04:41:39 eu-south-1  pulp-rpm-client
2022-11-21T04:41:39 us-west-2   pulp-rpm-client
2022-11-21T04:41:47 ap-southeast-1  pulp-rpm-client
2022-11-21T04:41:56 eu-west-1   pulp-rpm-client
2022-11-21T06:19:42 eu-west-3   flowclient
2022-11-21T07:27:58 ap-south-1  sortoffigs
2022-11-21T08:33:13 us-east-1   checkov
2022-11-21T08:33:19 us-west-2   checkov
2022-11-21T08:33:28 ap-south-1  checkov
2022-11-21T08:33:33 eu-west-1   checkov
2022-11-21T08:33:40 eu-west-2   checkov
2022-11-21T08:33:45 us-east-2   checkov
2022-11-21T08:33:46 ap-northeast-1  checkov
2022-11-21T08:33:53 eu-west-3   checkov
2022-11-21T08:33:57 eu-central-1    checkov
2022-11-21T08:34:00 eu-north-1  checkov
2022-11-21T08:34:01 ap-southeast-2  checkov
2022-11-21T08:34:03 ap-southeast-1  checkov
2022-11-21T08:34:04 eu-south-1  checkov
2022-11-21T08:37:59 ap-southeast-2  mozartpy
2022-11-21T09:09:21 eu-west-1   autogluon-features
2022-11-21T09:09:25 eu-west-2   autogluon-features
2022-11-21T09:09:26 us-east-2   autogluon-features
2022-11-21T09:09:32 ap-northeast-1  autogluon-features
2022-11-21T09:09:39 eu-west-3   autogluon-features
2022-11-21T09:09:43 eu-central-1    autogluon-features
2022-11-21T09:09:47 ap-southeast-2  autogluon-features
2022-11-21T09:09:48 eu-north-1  autogluon-features
2022-11-21T09:09:50 ap-southeast-1  autogluon-features
2022-11-21T09:09:51 eu-south-1  autogluon-features
2022-11-21T09:09:54 eu-south-1  autogluon
2022-11-21T09:09:56 us-east-1   autogluon-features
2022-11-21T09:10:06 us-west-2   autogluon-features
2022-11-21T09:10:14 ap-south-1  autogluon-features
2022-11-21T09:22:09 ap-south-1  gpmm
2022-11-21T09:53:29 eu-north-1  tf-nightly-gpu
2022-11-21T09:53:32 ap-southeast-1  shubhlipi
2022-11-21T09:53:57 ap-south-1  tf-nightly
2022-11-21T09:55:54 ap-south-1  duckdb
2022-11-21T11:33:49 ap-southeast-2  mlpack
2022-11-21T13:33:54 eu-west-3   dve-lumipy-preview
2022-11-21T13:40:12 us-east-2   renovosolutions-aws-cdk-one-time-event
2022-11-21T13:40:14 ap-south-1  renovosolutions-aws-cdk-one-time-event
2022-11-21T13:40:31 us-west-2   renovosolutions-aws-cdk-one-time-event
2022-11-21T13:40:33 eu-west-1   renovosolutions-aws-cdk-one-time-event
2022-11-21T13:40:36 ap-northeast-1  renovosolutions-aws-cdk-one-time-event
2022-11-21T13:40:43 ap-southeast-1  renovosolutions-aws-cdk-one-time-event
2022-11-21T13:40:45 eu-west-2   renovosolutions-aws-cdk-one-time-event
2022-11-21T13:40:50 eu-west-3   renovosolutions-aws-cdk-one-time-event
2022-11-21T13:40:55 eu-central-1    renovosolutions-aws-cdk-one-time-event
2022-11-21T13:40:56 us-east-1   renovosolutions-aws-cdk-one-time-event
2022-11-21T13:40:56 ap-southeast-2  renovosolutions-aws-cdk-one-time-event
2022-11-21T13:40:59 eu-south-1  renovosolutions-aws-cdk-one-time-event
2022-11-21T13:40:59 eu-north-1  renovosolutions-aws-cdk-one-time-event
2022-11-21T14:23:26 ap-southeast-1  kcli
2022-11-21T14:32:50 ap-south-1  bigdl-dllib
2022-11-21T14:33:22 ap-southeast-1  bigdl-dllib-spark2
2022-11-21T14:33:33 ap-southeast-2  bigdl-orca-spark2
2022-11-21T14:33:37 eu-south-1  bigdl-friesian-spark2
2022-11-21T14:33:41 eu-north-1  bigdl-spark2
2022-11-21T14:40:33 eu-south-1  qmenta-core
2022-11-21T14:47:14 ap-southeast-1  bigdl-chronos-spark3
2022-11-21T14:55:22 eu-west-3   pypackexer
2022-11-21T15:06:18 ap-southeast-2  example3-bolid-selenium-test
di commented 2 years ago

I'm very curious if pursuing something like https://github.com/pypi/warehouse/issues/12582 for purge events would help here.

aptalca commented 1 year ago

I'm confused about this. We have builders all over the world and pip seems to always fail to get the latest rapidfuzz version's sdist 12.3.4, whereas curling https://pypi.org/pypi/rapidfuzz/json always returns the sdist for that version correctly. So far I'm not seeing any intermittent issue. It seems to be that pip always fails and curl always works (on the same machine, too).

Does pip use a different endpoint for version checks?

EDIT: I put the specific details of my issue in #12662

neersighted commented 1 year ago

pip does not use the JSON API, it uses the PEP 503 endpoints only. Inspecting the index page HTML (or JSON with PEP 691's content-type set) will inform you what pip is seeing.

ewdurbin commented 1 year ago

@chriseldredge we have a potential avenue based on the value of Surrogate-Key stored on the compressed vs uncompressed responses. Is there any way the tooling you are using could issue their requests with Fastly-Debug: 1 and capture the value of Surrogate-Key when inconsistencies are found?

chriseldredge commented 1 year ago

I can probably do this in a test instance. I'll look into it and report back soon.

chriseldredge commented 1 year ago

Expected package cybrid-api-organization-python refresh serial to be greater or equal to 16116263 but was 16116261

Surrogate-Key: [all-legacy-json project-legacy-json project/cybrid-api-organization-python]

ewdurbin commented 1 year ago

Thanks @chriseldredge. That sort of invalidates what we were hoping to see. :/

TechCiel commented 1 year ago

I've run into this issue while maintaining a full mirror of pypi using bandersnatch. The stale JSONs are giving smaller serial than the serial obtained from XML-RPC, causing bandersnatch sending HTTP PURGE to the CDN edge, and fails 401.

After a bunch of network debugging, the 4 edge servers (151.101.{0,64,128,192}.223) seems behaving the same: stale version for Accept-Encoding: gzip but latest version without that.

I've tried using invalid Accept-Encoding headers and random query strings but didn't help. Below are some curl results with security headers omitted.

curl -I https://pypi.org/pypi/cmgdb/json -H 'Fastly-Debug: 1'

HTTP/2 200 
cache-control: max-age=900, public
content-type: application/json
etag: "53RmOOxx++rb2CZgzpkDXQ"
server: nginx/1.13.9
surrogate-control: max-age=86400, stale-while-revalidate=300, stale-if-error=86400
surrogate-key: all-legacy-json project-legacy-json project/cmgdb
x-pypi-last-serial: 16256766
accept-ranges: bytes
date: Fri, 30 Dec 2022 13:22:36 GMT
fastly-debug-path: (D cache-sna10727-LGB 1672406556) (F cache-lax10651-LGB 1672398633) (D cache-iad-kcgs7200128-IAD 1672398633) (F cache-iad-kcgs7200128-IAD 1672394731)
fastly-debug-ttl: (H cache-sna10727-LGB - - 7923) (H cache-iad-kcgs7200128-IAD 82497.757 86400.000 3902)
fastly-debug-digest: a6754b428860434906fa82e5ad81d09a506faedc2b3109420a6b72ec030f706c
x-served-by: cache-iad-kcgs7200128-IAD, cache-sna10727-LGB
x-cache: HIT, HIT
x-cache-hits: 3, 1
x-timer: S1672406556.298915,VS0,VE15
vary: Accept-Encoding, Accept-Encoding
content-length: 22701

curl -I https://pypi.org/pypi/cmgdb/json -H 'Fastly-Debug: 1' -H 'Accept-Encoding: gzip'

HTTP/2 200 
cache-control: max-age=900, public
content-encoding: gzip
content-type: application/json
etag: "N2IkiK0qTqHArbCsOIKeTw"
server: nginx/1.13.9
surrogate-control: max-age=86400, stale-while-revalidate=300, stale-if-error=86400
surrogate-key: all-legacy-json project-legacy-json project/cmgdb
x-pypi-last-serial: 16256765
accept-ranges: bytes
date: Fri, 30 Dec 2022 13:24:42 GMT
fastly-debug-path: (D cache-sna10739-LGB 1672406682) (F cache-lax10651-LGB 1672398173) (D cache-iad-kcgs7200128-IAD 1672398173) (F cache-iad-kiad7000082-IAD 1672394727)
fastly-debug-ttl: (H cache-sna10739-LGB - - 8509) (H cache-iad-kcgs7200128-IAD 82953.474 86400.000 3447)
fastly-debug-digest: a6754b428860434906fa82e5ad81d09a506faedc2b3109420a6b72ec030f706c
x-served-by: cache-iad-kcgs7200128-IAD, cache-sna10739-LGB
x-cache: HIT, HIT
x-cache-hits: 84, 1
x-timer: S1672406682.199174,VS0,VE1
vary: Accept-Encoding
content-length: 4060
chriseldredge commented 1 year ago

I'm curious if this issue is being prioritized vs other operational issues, bug fixes and feature work. If there is no active investigation, could the team consider providing a work-around for when we encounter this problem?

ewdurbin commented 1 year ago

We've had an active ticket with Fastly open since Dec 13 to try to get to the bottom of it, it did stall out over the US holiday period in the last two weeks of December but has picked back up as of yesterday.

We're working with Fastly support to determine how to resolve the issue and seem to have a couple possible solutions in the works that are being validated by their engineering departments now.

TechCiel commented 1 year ago

Currently inconsistent: autogluon-core nimblephysics neoscore

Implemented a work around to request an uncompressed version, worked for now.

ewdurbin commented 1 year ago

As of a few moments ago, we are now live with an attempted work-around from #12771. We hope this will mitigate the issue while we work on a better solution with Fastly support.

ewdurbin commented 1 year ago

Fastly support has confirmed we are hitting the race condition that #12771 attempts to mitigate.

We're going to look at other mitigations/solutions, but I'm curious how cache freshness has looked since we shipped that change on Friday. @chriseldredge ?

chriseldredge commented 1 year ago

It does look much improved. Since January 1st, I count 530 instances of stale page contents across all regions but starting from Jan 7 there have only been two:

2023-01-07 17:32:18.945 spanishconjugator
2023-01-08 16:39:56.839 classy-imaginary
ewdurbin commented 1 year ago

OK! Glad to know there's improvement! But we'll keep working towards full resolution.

ewdurbin commented 1 year ago

@chriseldredge curious what the latest stats are? we'll have a PR soon that should solve this once and for all, but want to get a baseline for comparison after.

BinaryVixen899 commented 1 year ago

👀

chriseldredge commented 1 year ago

Much quieter than December but still seeing some occasional hiccups:

2023-01-11 01:49:18.461 mpf-mc
2023-01-11 00:26:23.710 mppsolar
2023-01-15 08:08:04.839 nonebot-plugin-setu-customization
2023-01-16 12:35:42.794 checkov

The first three were observed each in only one region (not all the same region), but for whatever reason checkov was stale in all my regions around the same time.

TechCiel commented 1 year ago

Implemented a work around to request an uncompressed version, worked for now.

I run bandersnatch with 5 min interval, the log shows it retries uncompressed version many times:

   1368 2023-01-10
   1449 2023-01-11
   1333 2023-01-12
   1450 2023-01-13
   1105 2023-01-14
    571 2023-01-15
    867 2023-01-16
   1501 2023-01-17
   1679 2023-01-18
   1954 2023-01-19
   2061 2023-01-20

It shows ERROR: Stale serial for package * - Attempt * much less with the retry, but it's still a lot.

    414 2023-01-10
    538 2023-01-11
    503 2023-01-12
    578 2023-01-13
    484 2023-01-14
    276 2023-01-15
    376 2023-01-16
    574 2023-01-17
    788 2023-01-18
   1068 2023-01-19
   1827 2023-01-20

Only few packages kept failing after 3 attempts showing Giving up., but the problem then self contained

2023-01-13 01:22:56,520 ERROR: Stale serial for flask-servicefusion (16406942) not updating. Giving up. (package.py:82)
2023-01-13 01:24:11,008 ERROR: Stale serial for flask-servicefusion (16406942) not updating. Giving up. (package.py:82)
2023-01-13 01:24:35,785 ERROR: Stale serial for flask-servicefusion (16406942) not updating. Giving up. (package.py:82)
2023-01-13 01:25:05,824 ERROR: Stale serial for flask-servicefusion (16406942) not updating. Giving up. (package.py:82)
2023-01-14 03:50:08,342 ERROR: Stale serial for solara (16421131) not updating. Giving up. (package.py:82)
2023-01-14 03:50:08,471 ERROR: Stale serial for solara-assets (16421127) not updating. Giving up. (package.py:82)
2023-01-18 03:31:00,871 ERROR: Stale serial for astroemperor (16462480) not updating. Giving up. (package.py:82)

However, those 5 packages have been failing starting from Jan 19 23:00 UTC, lasted 8 hours, and have no sign to recover.

dagster (16493743)
dvc (16493729)
pears (16493746)
nanohub-remote (16493726)
spackle-python (16493742)

Have someone identified this issue or working on it?

Date & time are in UTC+8.

chriseldredge commented 1 year ago

Still seeing stale package pages in the last week:

2023-01-19 23:10:30 nanohub-remote
2023-01-19 23:10:55 dvc
2023-01-19 23:13:41 qiskit-terra
2023-01-19 23:15:29 dagster
2023-01-19 23:15:29 spackle-python
2023-01-19 23:16:06 pears
2023-01-20 15:07:22 jina-now
2023-01-21 14:31:40 spanishconjugator
2023-01-23 20:07:55 django-cfdi
2023-01-24 07:11:04 checkov
di commented 1 year ago

I don't think we've actually enabled the fix in https://github.com/pypi/warehouse/pull/12860 due to issues on Fastly's end.

TechCiel commented 1 year ago

The last "downtime" on Jan 20 lasted about 24 hours to wrap itself.

2023-01-20 07:09:01,726 ERROR: Stale serial for dagster (16493743) not updating. Giving up. (package.py:82)
... hundreds of lines omitted
2023-01-21 07:04:28,337 ERROR: Stale serial for nanohub-remote (16493726) not updating. Giving up. (package.py:82)

On Jan 24, django-cfdi ran into problem, but recovered after around ~10 minutes.

2023-01-24 03:59:35,786 ERROR: Stale serial for django-cfdi (16537564) not updating. Giving up. (package.py:82)
... 6 lines omitted
2023-01-24 04:10:55,236 ERROR: Stale serial for django-cfdi (16537564) not updating. Giving up. (package.py:82)

Right now, odoo14-addon-base-tier-validation has been failing for over 2 hours.

2023-01-26 17:43:17,336 ERROR: Stale serial for odoo14-addon-base-tier-validation (16573408) not updating. Giving up. (package.py:82)
... 32 lines omitted
ewdurbin commented 1 year ago

Confirming that the changes we hoped to support via #12860 have not been enabled. We are unfortunately awaiting further information from Fastly.

TechCiel commented 1 year ago

jarvis-ui has been failing since 2023-01-31 03:55 UTC. The problem does not seem to resolve automatically this time after 24 hours.

TechCiel commented 1 year ago

The problem of package jarvis-ui persists, causing many PyPI mirrors using pypa/bandersnatch being stale entirely, by keeping the reached serial back.

Could anyone from pypa confirm, or even manually mitigate this very incident?

ewdurbin commented 1 year ago

I'm not seeing any stale responses for jarvis-ui, but have issued a purge nonetheless...

We've got an update from Fastly and another potential avenue to resolve. Hoping we can get something in place next week to test the latest solution.

TechCiel commented 1 year ago

I'm not seeing any stale responses for jarvis-ui, but have issued a purge nonetheless...

I've asked some other PyPI mirrors to confirm we all repeatedly encounter the error below recently till now. The purge did NOT solve this.

bandersnatch.master.StalePage: Expected PyPI serial 16626043 for request https://pypi.org/pypi/jarvis-ui/json but got 16626041.

The expected serial to be 16626043 means the XMLRPC changelog said so, while the HTTP pypi.org is providing 16626041 for days. It's really a hard inconsistency. Could this be some other issue, like in XMLRPC?

As the mirroring is still blocked, @ewdurbin , if you may pardon me disturbing you.

ewdurbin commented 1 year ago

Hmmmm, this seems to be a separate issue:

warehouse=> select id, name, version, action, submitted_date from journals where name='jarvis-ui' order by id desc limit 10;
    id    |   name    | version |     action     |       submitted_date       
----------+-----------+---------+----------------+----------------------------
 16626043 | jarvis-ui | 0.3.3   | remove release | 2023-01-30 23:30:11.147139
 16626042 | jarvis-ui | 0.3.8   | remove release | 2023-01-30 23:30:10.755916
 16626041 | jarvis-ui | 0.4.0   | remove release | 2023-01-30 23:30:10.732833
 16626040 | jarvis-ui | 0.3.5   | remove release | 2023-01-30 23:30:10.69155
 16626039 | jarvis-ui | 0.3.1   | remove release | 2023-01-30 23:30:10.640855
 16626038 | jarvis-ui | 0.3.2   | remove release | 2023-01-30 23:30:10.541308
 16626037 | jarvis-ui | 0.2.5   | remove release | 2023-01-30 23:30:10.50915
 16626036 | jarvis-ui | 0.3.0   | remove release | 2023-01-30 23:30:10.527536
 16626035 | jarvis-ui | 0.4.8   | remove release | 2023-01-30 23:30:10.537667
 16626034 | jarvis-ui | 0.4.3   | remove release | 2023-01-30 23:30:10.518038
(10 rows)

warehouse=> select last_serial from projects where name='jarvis-ui';
 last_serial 
-------------
    16626041
(1 row)

Note that the last_serial column on the project record is not actually the latest serial. Seems the trigger that calls maintain_project_last_serial() for the journals table failed to update the project. I'll file a separate issue for that.

It has been corrected by calling a no-op update on the row for now.

warehouse=> begin;
BEGIN
warehouse=*> update journals set name=name where id=16626043;
UPDATE 1
warehouse=*> select * from journals where name='jarvis-ui' order by id desc limit 10;
    id    |   name    | version |     action     |       submitted_date       | submitted_by  |             submitted_from              
----------+-----------+---------+----------------+----------------------------+---------------+-----------------------------------------
 16626043 | jarvis-ui | 0.3.3   | remove release | 2023-01-30 23:30:11.147139 | thevickypedia | 2600:1702:1dc0:7820:6937:a99d:417e:de82
 16626042 | jarvis-ui | 0.3.8   | remove release | 2023-01-30 23:30:10.755916 | thevickypedia | 2600:1702:1dc0:7820:6937:a99d:417e:de82
 16626041 | jarvis-ui | 0.4.0   | remove release | 2023-01-30 23:30:10.732833 | thevickypedia | 2600:1702:1dc0:7820:6937:a99d:417e:de82
 16626040 | jarvis-ui | 0.3.5   | remove release | 2023-01-30 23:30:10.69155  | thevickypedia | 2600:1702:1dc0:7820:6937:a99d:417e:de82
 16626039 | jarvis-ui | 0.3.1   | remove release | 2023-01-30 23:30:10.640855 | thevickypedia | 2600:1702:1dc0:7820:6937:a99d:417e:de82
 16626038 | jarvis-ui | 0.3.2   | remove release | 2023-01-30 23:30:10.541308 | thevickypedia | 2600:1702:1dc0:7820:6937:a99d:417e:de82
 16626037 | jarvis-ui | 0.2.5   | remove release | 2023-01-30 23:30:10.50915  | thevickypedia | 2600:1702:1dc0:7820:6937:a99d:417e:de82
 16626036 | jarvis-ui | 0.3.0   | remove release | 2023-01-30 23:30:10.527536 | thevickypedia | 2600:1702:1dc0:7820:6937:a99d:417e:de82
 16626035 | jarvis-ui | 0.4.8   | remove release | 2023-01-30 23:30:10.537667 | thevickypedia | 2600:1702:1dc0:7820:6937:a99d:417e:de82
 16626034 | jarvis-ui | 0.4.3   | remove release | 2023-01-30 23:30:10.518038 | thevickypedia | 2600:1702:1dc0:7820:6937:a99d:417e:de82
(10 rows)

warehouse=*> select last_serial from projects where name='jarvis-ui';
 last_serial 
-------------
    16626043
(1 row)

warehouse=*> commit;
COMMIT