Closed rfparedes closed 3 days ago
@rfparedes ,
Thanks for letting us know. I cannot reproduce the issue and working with the Infrastructure SIG to find out the root of the issue.
This is now resolved. Since the *-latest
files are symlinks that point to the latest release, I believe that someone requested the file (symlink) while it wasn't finished syncing, causing it to cache a partial file with the wrong content-length
, making everything go haywire. I went ahead and checked the Fastly VCL configuration and saw nothing wrong with it. I verified that the downloads worked fine when bypassing Fasly.
Purging the chained service (that handles the caching) seems to have resolved the issue (and verified the content-length
was incorrect).
root@cody-rpm ~ [1]# curl -I https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-latest.x86_64.qcow2
HTTP/2 200
cache-control: public, max-age=31536000, immutable
content-type: application/octet-stream
last-modified: Wed, 20 Nov 2024 19:32:08 GMT
etag: "673e3938-1d440000"
accept-ranges: bytes
server: nginx/1.22.1
age: 16
via: 1.1 varnish, 1.1 varnish, 1.1 varnish, 1.1 varnish
date: Sun, 24 Nov 2024 17:56:22 GMT
x-served-by: cache-bur-kbur8200052-BUR, cache-bur-kbur8200055-BUR
x-cache: HIT, MISS, MISS, MISS
x-cache-hits: 1, 0, 0, 0
x-timer: S1732470982.447900,VS0,VE328
content-length: 490995712 <--- WRONG
root@cody-rpm ~# curl -I https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-latest.x86_64.qcow2
HTTP/2 200
cache-control: public, max-age=31536000, immutable
content-type: application/octet-stream
last-modified: Mon, 05 Aug 2024 07:54:29 GMT
etag: "66b08535-23450000"
accept-ranges: bytes
server: nginx/1.22.1
age: 443014
via: 1.1 varnish, 1.1 varnish, 1.1 varnish, 1.1 varnish
date: Sun, 24 Nov 2024 17:32:14 GMT
x-served-by: cache-bur-kbur8200112-BUR, cache-bur-kbur8200151-BUR
x-cache: HIT, HIT, MISS, MISS
x-cache-hits: 89, 0, 0, 0
x-timer: S1732469535.885754,VS0,VE17
content-length: 591724544 <--- RIGHT
Linux-9-GenericCloud-latest.x86_64.qcow2
AlmaLinux-9-GenericC 100% [=================================================================================================================>] 468.25M 113.49MB/s
[Files: 1 Bytes: 468.25M [105.84MB/s] Redirects: 0 Todo: 0 Errors: 0 ]
root@cody-rpm ~# openssl sha256 AlmaLinux-9-GenericCloud-latest.x86_64.qcow2
SHA2-256(AlmaLinux-9-GenericCloud-latest.x86_64.qcow2)= abddf01589d46c841f718cec239392924a03b34c4fe84929af5d543c50e37e37
Thanks for the report! Let me know if you run into any further issues.
@codyro Thanks Cody. I can confirm also, all is good now.
I have automated process that downloads: https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-latest.x86_64.qcow2
This is failing to download in entirety. I've tried from multiple locations and servers.
The https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-9.5-20241120.x86_64.qcow2 downloads fine.