search

rpm-software-management / rpm

The RPM package manager
http://rpm.org
Other
508 stars 367 forks source link

--build-in-place multiple flaws #3131

Open pmatilai opened 5 months ago

pmatilai commented 5 months ago

The %mkbuilddir changes in rpm 4.20 is shaking up the tree and strange fruit are falling out, one of them being --build-in-place option from 0265acbde87036df08da16507a8c4c9a9e6c3c64. There's clearly a call for this functionality, but it's not well defined and is at odds with how rpm likes to do things, so there are all sorts of gaps and flaws in there. There are probably several tickets worth of issues in here, this is just a collection of random remarks while still fresh in memory:

DaanDeMeyer commented 5 months ago

I have run into a few of these getting --build-in-place to work for systemd, let me post my workarounds here as extra information:

the only way --build-in-place really makes sense is doing vpath builds, but we don't natively support that in rpm (yet)

I implicitly assume that I can set vpath_builddir which is luckily supported everywhere via distribution macros

%patch should be disabled with --build-in-place because it's not a pristine source we can return to source/patch meaning is questionable in the first place, because it's not what we build there

I added a BuildSourcesEphemeral= option in mkosi which mounts a writable overlay on top of the build sources that is thrown away after the build. This way the spec can patch and change all it wants without those changes persisting after the build.

build-in-place always requires spec to be aware of the action, should it instead be an option to %setup instead of cli? or maybe it should just skip %prep entirely, that seems more appropriate for the cause

For the systemd spec we had to add %version_override, %release_override and %upstream macros. The first two allow us to build rpms that are guaranteed to be newer than previous ones. We set the version override to the upstream version in git and set the release override to the timestamp of the git commit (or current unix timestamp if the tree is dirty). The upstream macro allows us to accomodate build system changes or other customizations that are only desired for --build-in-place upstream builds.

pmatilai commented 5 months ago

Yeah I ran into some of these while looking at the usages. It doesn't seem exactly easy-to-use feature just now :laughing: Can you point me to the spec file the systemd mkosi setup uses? If it's in the repo, it's not exactly obvious to an outsider. Edit: oh, that's a git submodule...

DaanDeMeyer commented 5 months ago

Yes we just use the fedora spec as is and include it as a git submodule so we can pin each commit in the systemd repository to a specific commit of the spec sources.

pmatilai commented 5 months ago

The more I look/think into this, it seems that --build-in-place should entirely disable %prep and all Source/Patch processing, because .. that's what's it all about. Or, it should take a copy of the original source directory to preserve semi-normal functionality of rpm.

DaanDeMeyer commented 5 months ago

The more I look/think into this, it seems that --build-in-place should entirely disable %prep and all Source/Patch processing, because .. that's what's it all about. Or, it should take a copy of the original source directory to preserve semi-normal functionality of rpm.

So I haven't really settled on this myself yet. In the systemd case, we have two types of patches: backports from upstream and downstream patches. The backports from upstream are of course useless in the --build-in-place scenario, but the downstream patches are useful to keep sometimes. As an example, systemd has a downstream patch to patch the systemd upstream PAM snippet to match the Fedora guidelines. That's one patch that is not disabled when the %upstream macro is defined.

A copy of the original source directory could be rather slow depending on the project unfortunately, and one of the core benefits of --build-in-place is being able to do fast rebuilds.

pmatilai commented 5 months ago

This is all very valuable feedback, thanks!

Yeah your systemd usage is a rather evolved case, with lots of external (from rpm POV) infra to support it that's not (at least currently) available to rpmbuild alone. Like the writable overlay which makes patching not an issue, but without such a thing its basically down to copy or disable (patches), because otherwise it doesn't work / is not sensible.

This probably needs to be switchable somehow. Maybe there could be a macro/script hook you need to define to enable %prep processing with build-in-place, and the hook would be responsible for copying/mounting/whatever (or I guess in your case, just "nop" because its handled elsewhere) magic needed to make patching a meaningful operation. It could also be switching to a new temporary branch in git, where you can apply all the patches you want and then restore at end.

mlschroe commented 5 months ago

Just as data point: SUSE's build tool (i.e. like mock) has both a --rpm-build-in-place and a --rpm-build-in-place-noprep option.

Conan-Kudo commented 5 months ago

I think there'd be interest in having this wired up in mock if it was easier to use...

pmatilai commented 2 months ago

I'd forgotten we have this ticket... updated the description to reflect the current state.

Commit e0925ad6e3185030b004343f73a5917fdc562d6c clarifies this all a lot by simply defining that %prep is skipped on build-in-place, ie --build-in-place effectively implies --noprep. And yes some users will need to modify their specs, there's no way to make sense out of this without breaking something.

DaanDeMeyer commented 2 months ago

the only way --build-in-place really makes sense is doing vpath builds, but we don't natively support that in rpm (yet)

I actually figured out that you can mount a build directory onto a source directory with overlayfs to do vpath builds for autotools and makefile based stuff as well. A gross hack, but it does work. So it doesn't necessarily need to be a vpath based build system.