search

rpm-software-management / librepo

A library providing C and Python (libcURL like) API for downloading packages and linux repository metadata in rpm-md format
http://rpm-software-management.github.io/librepo/
GNU Lesser General Public License v2.1
74 stars 91 forks source link

Incorrect content-length when downloading with content compression (gzip) #253

Closed joschi closed 2 years ago

joschi commented 2 years ago

I've been running into a weird issue since March 11, 2022 when using DNF (or YUM) on RHEL/UBI 8.5 and trying to install software from the OpenResty RPM repository: https://openresty.org/en/linux-packages.html#rhel

When setting up the OpenResty RHEL repository and trying to install OpenResty, dnf (and yum) fail to download the packages because the expected content length doesn't match the content length the server reports:

[MIRROR] openresty-1.19.9.1-1.el8.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 1161260 but expected size is: 1161264

When downloading the RPM files manually with cURL and installing them (rpm -ivh [...]), they have the correct file size and the installation works without an error.

Full output for reproducing the issue (Docker with UBI 8.5) ```text # docker run -it --rm registry.access.redhat.com/ubi8/ubi:8.5 [root@e52901de9fd9 /]# cat /etc/redhat-release Red Hat Enterprise Linux release 8.5 (Ootpa) [root@e52901de9fd9 /]# curl -sSf https://openresty.org/package/rhel/openresty.repo -o /etc/yum.repos.d/openresty.repo [root@e52901de9fd9 /]# dnf install -y openresty Updating Subscription Management repositories. Unable to read consumer identity This system is not registered with an entitlement server. You can use subscription-manager to register. Official OpenResty Open Source Repository for RHEL 86 kB/s | 67 kB 00:00 Red Hat Universal Base Image 8 (RPMs) - BaseOS 357 kB/s | 796 kB 00:02 Red Hat Universal Base Image 8 (RPMs) - AppStream 3.7 MB/s | 2.7 MB 00:00 Red Hat Universal Base Image 8 (RPMs) - CodeReady Builder 33 kB/s | 16 kB 00:00 Dependencies resolved. ========================================================================================================================================================================================== Package Architecture Version Repository Size ========================================================================================================================================================================================== Installing: openresty x86_64 1.19.9.1-1.el8 openresty 1.1 M Installing dependencies: openresty-openssl111 x86_64 1.1.1l-1.el8 openresty 1.6 M openresty-pcre x86_64 8.44-1.el8 openresty 169 k openresty-zlib x86_64 1.2.11-3.el8 openresty 59 k Transaction Summary ========================================================================================================================================================================================== Install 4 Packages Total download size: 2.9 M Installed size: 8.1 M Downloading Packages: [MIRROR] openresty-1.19.9.1-1.el8.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 1161260 but expected size is: 1161264 [MIRROR] openresty-openssl111-1.1.1l-1.el8.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 1654940 but expected size is: 1654948 [MIRROR] openresty-1.19.9.1-1.el8.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 1161260 but expected size is: 1161264 [MIRROR] openresty-openssl111-1.1.1l-1.el8.x86_64.rpm: Interrupted by header callback: Server reports Content-Length: 1654940 but expected size is: 1654948 [FAILED] openresty-openssl111-1.1.1l-1.el8.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success (2-3/4): openresty-1.19.9.1-1.el8.x86_64.rpm 56% [=========================================- ] 703 kB/s | 1.7 MB 00:01 ETA The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. Error: Error downloading packages: Cannot download openresty-openssl111-1.1.1l-1.el8.x86_64.rpm: All mirrors were tried ```
Output in `/var/log/dnf.librepo.log` ```text 2022-03-14T08:35:21+0000 INFO Librepo version: 1.14.0 with CURL_GLOBAL_ACK_EINTR support (libcurl/7.61.1 OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.6 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.2.0) libssh/0.9.4/openssl/zlib nghttp2/1.33.0) 2022-03-14T08:35:22+0000 INFO Librepo version: 1.14.0 with CURL_GLOBAL_ACK_EINTR support (libcurl/7.61.1 OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.6 libidn2/2.2.0 libpsl/0.20.2 (+libidn2/2.2.0) libssh/0.9.4/openssl/zlib nghttp2/1.33.0) 2022-03-14T08:35:22+0000 INFO Downloading: https://openresty.org/package/rhel/8/x86_64/openresty-1.19.9.1-1.el8.x86_64.rpm 2022-03-14T08:35:22+0000 INFO Downloading: https://openresty.org/package/rhel/8/x86_64/openresty-openssl111-1.1.1l-1.el8.x86_64.rpm 2022-03-14T08:35:22+0000 INFO Downloading: https://openresty.org/package/rhel/8/x86_64/openresty-pcre-8.44-1.el8.x86_64.rpm 2022-03-14T08:35:22+0000 INFO Error during transfer: Interrupted by header callback: Server reports Content-Length: 1161260 but expected size is: 1161264 2022-03-14T08:35:22+0000 INFO Downloading: https://openresty.org/package/rhel/8/x86_64/openresty-1.19.9.1-1.el8.x86_64.rpm 2022-03-14T08:35:22+0000 INFO Error during transfer: Interrupted by header callback: Server reports Content-Length: 1654940 but expected size is: 1654948 2022-03-14T08:35:22+0000 INFO Downloading: https://openresty.org/package/rhel/8/x86_64/openresty-openssl111-1.1.1l-1.el8.x86_64.rpm 2022-03-14T08:35:22+0000 INFO Error during transfer: Downloading successful, but checksum doesn't match. Calculated: 0665dea098398c99d17d497fc2d02bda0d9b6a37c6ea7ff7ba5bbf413eeb4ec7(sha256) Expected: ab9afddc4f4b0a803774dfe92594d2bab72e6ef29eb002e137be45915f612685(sha256) 2022-03-14T08:35:22+0000 INFO Downloading: https://openresty.org/package/rhel/8/x86_64/openresty-pcre-8.44-1.el8.x86_64.rpm 2022-03-14T08:35:22+0000 INFO Error during transfer: Interrupted by header callback: Server reports Content-Length: 1161260 but expected size is: 1161264 2022-03-14T08:35:22+0000 INFO Error while downloading: Cannot download openresty-1.19.9.1-1.el8.x86_64.rpm: All mirrors were tried ```

I think this could be an issue with compressed (gzip) file transfers since at least the content length is corresponding to the values in the error message (1161264 bytes vs. 1161260 bytes).

Compressed download with cURL ```text [root@5a7f07ff3765 /]# curl --compressed -s -v 'https://openresty.org/package/rhel/8/x86_64/openresty-1.19.9.1-1.el8.x86_64.rpm' * Trying 3.125.51.27... * TCP_NODELAY set * Trying 2a05:d014:808:3600:d7c8:720c:b584:cdeb... * TCP_NODELAY set * Immediate connect fail for 2a05:d014:808:3600:d7c8:720c:b584:cdeb: Cannot assign requested address * Trying 2a05:d01c:66b:e800:560c:c0ea:4e5:db36... * TCP_NODELAY set * Immediate connect fail for 2a05:d01c:66b:e800:560c:c0ea:4e5:db36: Cannot assign requested address * Connected to openresty.org (3.125.51.27) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, [no content] (0): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=openresty.org * start date: Feb 12 00:00:00 2022 GMT * expire date: May 13 23:59:59 2022 GMT * subjectAltName: host "openresty.org" matched cert's "openresty.org" * issuer: C=AT; O=ZeroSSL; CN=ZeroSSL RSA Domain Secure Site CA * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * TLSv1.3 (OUT), TLS app data, [no content] (0): * TLSv1.3 (OUT), TLS app data, [no content] (0): * TLSv1.3 (OUT), TLS app data, [no content] (0): * Using Stream ID: 1 (easy handle 0x56385f954690) * TLSv1.3 (OUT), TLS app data, [no content] (0): > GET /package/rhel/8/x86_64/openresty-1.19.9.1-1.el8.x86_64.rpm HTTP/2 > Host: openresty.org > User-Agent: curl/7.61.1 > Accept: */* > Accept-Encoding: deflate, gzip, br > * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.3 (IN), TLS app data, [no content] (0): * Connection state changed (MAX_CONCURRENT_STREAMS == 128)! * TLSv1.3 (OUT), TLS app data, [no content] (0): * TLSv1.3 (IN), TLS app data, [no content] (0): < HTTP/2 200 < date: Mon, 14 Mar 2022 08:56:37 GMT < content-type: application/x-redhat-package-manager < content-length: 1161264 < set-cookie: userid=0000260000043f2c4a280ddf; Expires=Tue, 14-Mar-23 08:56:37 GMT; Max-Age=31536000; Domain=openresty.org < last-modified: Fri, 06 Aug 2021 21:31:38 GMT < etag: "610daa3a-11b830" < expires: Fri, 11 Mar 2022 17:23:05 GMT < cache-control: max-age=3600 < server: OpenResty Edge < req-id: 0000260000043f2c4a280ddf < edge-cache-age: 232412 < cache-status: HIT < accept-ranges: bytes < * Failed writing body (0 != 16011) * TLSv1.3 (OUT), TLS app data, [no content] (0): * stopped the pause stream! * Connection #0 to host openresty.org left intact ```
Uncompressed download with cURL ```text [root@5a7f07ff3765 /]# curl -s -v 'https://openresty.org/package/rhel/8/x86_64/openresty-1.19.9.1-1.el8.x86_64.rpm' * Trying 3.125.51.27... * TCP_NODELAY set * Connected to openresty.org (3.125.51.27) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Certificate (11): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, CERT verify (15): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Finished (20): * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.3 (OUT), TLS handshake, [no content] (0): * TLSv1.3 (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 * ALPN, server accepted to use h2 * Server certificate: * subject: CN=openresty.org * start date: Feb 12 00:00:00 2022 GMT * expire date: May 13 23:59:59 2022 GMT * subjectAltName: host "openresty.org" matched cert's "openresty.org" * issuer: C=AT; O=ZeroSSL; CN=ZeroSSL RSA Domain Secure Site CA * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * TLSv1.3 (OUT), TLS app data, [no content] (0): * TLSv1.3 (OUT), TLS app data, [no content] (0): * TLSv1.3 (OUT), TLS app data, [no content] (0): * Using Stream ID: 1 (easy handle 0x56049fef6690) * TLSv1.3 (OUT), TLS app data, [no content] (0): > GET /package/rhel/8/x86_64/openresty-1.19.9.1-1.el8.x86_64.rpm HTTP/2 > Host: openresty.org > User-Agent: curl/7.61.1 > Accept: */* > * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.3 (IN), TLS handshake, [no content] (0): * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): * TLSv1.3 (IN), TLS app data, [no content] (0): * Connection state changed (MAX_CONCURRENT_STREAMS == 128)! * TLSv1.3 (OUT), TLS app data, [no content] (0): * TLSv1.3 (IN), TLS app data, [no content] (0): < HTTP/2 200 < date: Mon, 14 Mar 2022 08:56:42 GMT < content-type: application/x-redhat-package-manager < content-length: 1161260 < set-cookie: userid=0000260000043f2c4a500de5; Expires=Tue, 14-Mar-23 08:56:42 GMT; Max-Age=31536000; Domain=openresty.org < last-modified: Fri, 06 Aug 2021 21:38:11 GMT < etag: "610dabc3-11b82c" < expires: Thu, 10 Mar 2022 18:38:46 GMT < cache-control: max-age=3600 < server: OpenResty Edge < req-id: 0000260000043f2c4a500de5 < edge-cache-age: 314275 < cache-status: HIT < accept-ranges: bytes < * Failed writing body (0 != 16011) * TLSv1.3 (OUT), TLS app data, [no content] (0): * stopped the pause stream! * Connection #0 to host openresty.org left intact ```

Versions:

[root@5a7f07ff3765 /]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.5 (Ootpa)
[root@5a7f07ff3765 /]# rpm -qa 'dnf|librepo'
librepo-1.14.0-2.el8.x86_64
python3-librepo-1.14.0-2.el8.x86_64
dnf-data-4.7.0-4.el8.noarch
dnf-4.7.0-4.el8.noarch
dnf-plugin-subscription-manager-1.28.21-3.el8.x86_64

Related issue: https://github.com/openresty/openresty-packaging/issues/75

lukash commented 2 years ago

Hello, if the remote RPMs have different content length than the one recorded in the repository metadata, it's an issue with inconsistent remote repository metadata which should be resolved on the repository maintainer side. Unlikely to be a librepo bug.

joschi commented 2 years ago

@lukash Thanks for your response.

I think you're completely correct. The published meta data in https://openresty.org/package/rhel/8/x86_64/repodata/a3e9d19182cb25ab5ea81af211cc7b389820c784f9e97a7855533363291d4e3c-primary.xml.gz doesn't seem to match the actual RPM packages.

[root@8fefeb52baae /]# curl -O 'https://openresty.org/package/rhel/8/x86_64/openresty-1.19.9.1-1.el8.x86_64.rpm'
[root@8fefeb52baae /]# sha256sum openresty-1.19.9.1-1.el8.x86_64.rpm
e06fab89b440c074a49e11fe5146d4c68ee31ea84768f5a13422c5a71cb3234b  openresty-1.19.9.1-1.el8.x86_64.rpm
[root@8fefeb52baae /]# ls -l openresty-1.19.9.1-1.el8.x86_64.rpm
-rw-r--r-- 1 root root 1161260 Mar 14 14:43 openresty-1.19.9.1-1.el8.x86_64.rpm
<package type="rpm">
  <name>openresty</name>
  <arch>x86_64</arch>
  <version epoch="0" ver="1.19.9.1" rel="1.el8"/>
  <checksum type="sha256" pkgid="YES">c96544be81f92e092fd3eeb6348191c3414c6efcdffcd83401d50b4f62346b73</checksum>
  <summary>OpenResty, scalable web platform by extending NGINX with Lua</summary>
  <description>[...]</description>
  <packager>Yichun Zhang (agentzh) &lt;yichun@openresty.com&gt;</packager>
  <url>https://openresty.org/</url>
  <time file="1628285498" build="1628285497"/>
  <size package="1161264" installed="3619941" archive="3644808"/>
  <location href="openresty-1.19.9.1-1.el8.x86_64.rpm"/>
  <format>
    <rpm:license>BSD</rpm:license>
    <rpm:vendor></rpm:vendor>
    <rpm:group>System Environment/Daemons</rpm:group>
    <rpm:buildhost>centos8-pkg</rpm:buildhost>
    <rpm:sourcerpm>openresty-1.19.9.1-1.el8.src.rpm</rpm:sourcerpm>
    <rpm:header-range start="4504" end="29864"/>
    <rpm:provides>
      <rpm:entry name="openresty" flags="EQ" epoch="0" ver="1.19.9.1" rel="1.el8"/>
      <rpm:entry name="openresty(x86-64)" flags="EQ" epoch="0" ver="1.19.9.1" rel="1.el8"/>
    </rpm:provides>
    <rpm:requires>
      <rpm:entry name="/bin/sh" pre="1"/>
      <rpm:entry name="/bin/sh"/>
      <rpm:entry name="openresty-openssl111" flags="GE" epoch="0" ver="1.1.1h" rel="1"/>
      <rpm:entry name="openresty-pcre" flags="GE" epoch="0" ver="8.44" rel="1"/>
      <rpm:entry name="openresty-zlib" flags="GE" epoch="0" ver="1.2.11" rel="3"/>
      <rpm:entry name="systemd"/>
    </rpm:requires>
    <file>/usr/bin/openresty</file>
    <file>/usr/local/openresty/bin/openresty</file>
    <file>/usr/local/openresty/luajit/bin/luajit</file>
    <file>/usr/local/openresty/luajit/bin/luajit-2.1.0-beta3</file>
    <file>/usr/local/openresty/nginx/sbin/nginx</file>
  </format>
</package>