rpm-software-management / librepo

A library providing C and Python (libcURL like) API for downloading packages and linux repository metadata in rpm-md format
http://rpm-software-management.github.io/librepo/
GNU Lesser General Public License v2.1
75 stars 91 forks source link

Add support for rpm2extents transcoder #222

Open malmond77 opened 3 years ago

malmond77 commented 3 years ago

Two related parts:

  1. If LIBREPO_TRANSCODE_RPMS environment is set to a program (with parameters) then downloads are piped through it.
  2. Transcoded RPMS by definition will not have the same bits on disk as downloaded. This is inherent. The transcoder is tasked with measuring the bits that enter stdin and storing a copy of the digest(s) seen in the footer. librepo can then use these stored digests instead if the environment variable is set.

This is part of changes described in https://fedoraproject.org/wiki/Changes/RPMCoW

malmond77 commented 3 years ago

The second commit fixes an issue that only really surfaced when using public mirrors. In my previous testing (on CentOS) I didn't hit it because our internal mirrors were generally more consistent.

I'm aiming to get this code into Fedora 34 shortly, either as part of an updated tagged release (1.12.2?) or if not, just as a patch we carry in the rpm src. As part of the Fedora 34 change: this code path is intended to be optional. As far as I can tell, if the environment variable isn't set, then nothing should be different.

Conan-Kudo commented 3 years ago

@malmond77 This looks good to me, but I am not comfortable merging this until the RPM PR is merged: https://github.com/rpm-software-management/rpm/pull/1470

DemiMarie commented 2 years ago

@Conan-Kudo This is not a change I am comfortable with from a security perspective, as per past discussions. The verification needs to happen before transcoding, which means either buffering the entire package on disk or changing the metadata format.

DemiMarie commented 2 years ago

To elaborate: the idea of rpm2extents is fine, but it needs to be a separate entry in the metadata XML, with its own digest.