support verifying reproducible builds

[bmwiedemann]

Is your feature request related to a problem? Please describe. a lot of our Factory packages already allow for reproducible builds. It would be good if it was possible to have a 2nd OBS instance linked the the main OBS to notice newly built binaries and then try to reproduce them.

Describe the solution you'd like Interface-wise it could be a line in prj meta

<link project="openSUSE.org:openSUSE:Factory" rebuild="verify" buildcompare="true" />

That would then react to finished binary builds on the main OBS by using the same %_buildhost , %{disturl} and %{release} values. It should also inherit all prjconf and build-dependencies (if available).

If such a verification build shows diffs to the original (either in build-compare or in bit-wise comparison (after rpm --delsign)), it should mark it somehow - e.g. as failed. OTOH similar results show succeeded.

Describe alternatives you've considered I can continue to do occasional verification builds outside of OBS.

Additional context https://rb.zq1.de/compare.factory-20230830/build-compare-differed-builds-nachbau.txt shows the list of unverifiable packages from August 2023. It should be possible to generate a similar list from the OBS verification results.

[adrianschroeter]

This xml element is for defining source aggregation and would lead to breakages for all interconnected instances.

IMHO your need sounds more like a BuildFlag.

[bmwiedemann]

BuildFlag would be fine as well. But we also need access to all sources+binaries from the linked project.

[JanZerebecki]

It would need to use the existing _buildenv and access binaries from an archive of old build results like https://github.com/bmwiedemann/opensusearchive . (Requiring it to build in lock step with the original OBS instance is not a good approach.) Otherwise it will now and then build with a different input than what it tries to reproduce.

While it would be useful on its own, to be usable for observed binary transparency we need it:

• to produce signed attestations of builds (in-toto has some format and tooling for this, which we might need to extend with support for embedded signatures)
• to be easy to install and keep running an instance of it for people who otherwise are not involved with OpenSUSE

So my current idea is instead of implementing it in OBS to see if we can fit support for this into https://github.com/in-toto/rebuilderd using obs-build. Then trigger it and connect it back into OBS requests by a review bot or into git reviews by a CI producing build status.

[adrianschroeter]

using _buildenv won't work for any rolling distributions like Factory or SlowRoll though

regarding attestations, I suppose you have SLSA attestations in mind (where wrapped into in-toto in the past). SLSA attestations could be used for rebuilding packages, but we have not enabled this for anything beside SLE due to the amount of needed resouces to save all build dependency binaries forever basically.

[JanZerebecki]

I have not yet verified if SLSA attestations are sufficient for binary transparency, could be, they certainly seemed at one point to be designed for that, but things upstream also changed a lot.

Yes, we'll need to save every build (and better also the log, and better also those that are not used as build dependencies themselves). The space requirements are not a problem in practice, considering disk prices, at least with the Tumbleweed archive Bernhard keeps. Is there some support for something like this in OBS? Is that somehow only enabled when generating SLSA attestations?

I think the current Tumbleweed archive script theoretically misses a build if the same package is finished building again before the previous result was archived.