split base/layered versionlocked package problem (krb5-workstation)

[cgwalters]

In wstree right now krb5-libs is a dependency of various things in the base OS. krb5-workstation is an app that uses it - however, it's version locked.

This introduces a problem with package layering, since if the compose server picks up a newer version from package mirror A, but the client only has an older version from mirror B, upgrades break. (Or if it's not installed, it can't be).

There are obviously a few solutions to this...likely krb5-workstation should be more flexible, and only introduce a hard requirement on the -libs package version if it needs it. Second, we may need to consider a way to retrieve rpm-md data from the same place as the ostree commit, or vice versa. Basically having unified metadata/data management.

--

EDIT (2019-09-08): rojig is a much bigger hammer attempting to solve this problem.

[jlebon]

Second, we may need to consider a way to retrieve rpm-md data from the same place as the ostree commit, or vice versa. Basically having unified metadata/data management.

Isn't this completely up to the content provider, though? Doesn't seem like something we can do much about in rpm-ostree.

[cgwalters]

I think if we provided the infrastructure to link to the rpm-md repos from the ostree commit, infra providers could use that. We basically have all the data already from the treecompose side. The infra provider would need to retain multiple versions of the rpm-md repo data though.

That said I think the general solution to this is to relax the strict version requirements.

[cgwalters]

🏈 Punted on this

[cgwalters]

Hit this again today trying to layer virt-manager; through a depchain virt-manager-common → python-libxml2 (not in the tree) → libxml2:

error: Could not depsolve transaction; 1 problem detected:
0. package python-libxml2-2.9.3-4.fc25.x86_64 requires libxml2 = 2.9.3-4.fc25, but none of the providers can be installed

And at the time:

$ rpm -q libxml2
libxml2-2.9.4-2.fc25.x86_64

[dustymabe]

doesn't this problem go away if we can replace content in the base layer? https://github.com/projectatomic/rpm-ostree/issues/485

[cgwalters]

I think we should definitely allow it, but doing that invalidates the concept that the tree is tested and run as a unit. I'm not sure I'd want it to just happen by default.

Particularly in this case, it turned out the problem is I was being served a very out of date (rpm-md) mirror.

[dustymabe]

I think we should definitely allow it, but doing that invalidates the concept that the tree is tested and run as a unit. I'm not sure I'd want it to just happen by default.

right, but don't we invalidate that when we allow package layering at all? I know technically we are just "adding to" rather than "changing" but adding things breaks things sometimes too.

Maybe for base package overrides we default to spitting out a message about the fact that this transaction will override a base package and thus the integrity of the tree may suffer and then allow the user to (y/n). Also provide a cli flag for forcing this.

[cgwalters]

Another good example of this is emacs-filesystem (pulled into base) and emacs.

[kallisti5]

Seeing this here as well with vim... this is a painful bug.

[miabbott]

Like @kallisti5 above, I ran into the problem with vim-common. (Going to paste my session here for searchability later)

$ sudo rpm-ostree status                                                                                                                                                                     
[sudo] password for miabbott:
State: idle                                         
Deployments:                                                                                             
● atomicws:fedora/x86_64/workstation                                                                     
                   Version: 26.201 (2017-10-06 17:57:57)                                                                                                                                                           
                BaseCommit: c458ed1025d135b88f611eb169c515fd478d8751bcd742cb05284a929ab07a56
           LayeredPackages: chrome-gnome-shell ffmpeg-libs krb5-workstation libvirt-client sshpass tmux vagrant-libvirt vim virt-install virt-manager

  atomicws:fedora/x86_64/workstation                                                                     
                   Version: 26.187 (2017-10-02 09:58:23)
                BaseCommit: 5637588b7331a5b49f4c417e4f5eca706f067d9b47911d5849fe9981ec30c3c9
           LayeredPackages: chrome-gnome-shell ffmpeg-libs krb5-workstation libvirt-client tmux vagrant-libvirt vim virt-manager                                                                                   
[/var/home/miabbott *]$ sudo rpm-ostree upgrade     

Updating metadata for 'updates': [==========================] 100%
rpm-md repo 'updates'; generated: 2017-10-09 14:42:40                                                                                                                                                              

Updating metadata for 'fedora': [==========================] 100%
rpm-md repo 'fedora'; generated: 2017-07-05 20:31:38                                                                                                                                                               

Updating metadata for 'rpmfusion-nonfree': [==================================] 100%
rpm-md repo 'rpmfusion-nonfree'; generated: 2017-07-10 12:08:07                                                                                                                                                    

Updating metadata for 'region51-chrome-gnome-shell': [============================================] 100%
rpm-md repo 'region51-chrome-gnome-shell'; generated: 2017-10-10 07:03:04                                                                                                                                          

Updating metadata for 'walters-syslinux-stub-i686': [========================================] 100%
rpm-md repo 'walters-syslinux-stub-i686'; generated: 2017-10-10 09:05:48                                                                                                                                           

Updating metadata for 'walters-syslinux-stub': [==================================== 100%
rpm-md repo 'walters-syslinux-stub'; generated: 2017-10-10 09:05:46                                                                                                                                                

Updating metadata for 'rpmfusion-nonfree-updates': [=========================================] 100%
rpm-md repo 'rpmfusion-nonfree-updates'; generated: 2017-10-06 17:41:43                                                                                                                                            

Updating metadata for 'rpmfusion-free': [===============================] 100%
rpm-md repo 'rpmfusion-free'; generated: 2017-07-10 12:06:27                                                                                                                                                       

Updating metadata for 'rpmfusion-free-updates': [======================================] 100%

Importing metadata [======================] 100%
Resolving dependencies... done                      
Will download: 2 packages (196.2 kB)

  Downloading from updates: [========================] 100%

Importing: [=======================] 100%
Applying 134 overlays... error: Checking out vim-common-2:8.0.1171-1.fc26.x86_64: Hardlinking c0/679f8a0eefe6ddb91b60afe7ff885aa8ee2b39fea8a0dbf342d7758733a7d8.file to ex.1.gz: File exists

[jlebon]

Right, this is the same issue though the output is different because v2017.9 has https://github.com/projectatomic/rpm-ostree/pull/974. A hacky workaround is to manually download the matching vim packages (https://koji.fedoraproject.org/koji/buildinfo?buildID=969318) and locally install that. Hacky because on the next upgrade that bumps vim-minimal again, you'll have to uninstall them and possibly get the latest RPMs if they're still not in sync (though if the tree is freshly composed, that shouldn't be an issue).

[jlebon]

@miabbott, I think you're hitting something different. I opened a separate issue for your error here: https://github.com/projectatomic/rpm-ostree/issues/1047.

[cgwalters]

This problem will be comprehensively fixed by jigdo ♲📦 https://github.com/projectatomic/rpm-ostree/issues/1081 because the rpm-md and ostree commit will move at the same cadence by default.

[dustymabe]

This problem will be comprehensively fixed by jigdo ♲📦 #1081 because the rpm-md and ostree commit will move at the same cadence by default.

Can you explain this a little more? if yesterday's bodhi run had a new fedora-atomic-host rpm in it (i.e. a new release), and today's run includes new packages (and new base packages that are deps) then won't an rpm-ostree install still fail in the same way if a base package is part of the update?

[cgwalters]

The jigdoRPM should be generated from the same rpm-md reposet as contains it. That's how it works today in FAHC.

[cgwalters]

Yes, this means you need to do: createrepo_c, compose tree, copy in jigdoRPM, createrepo_c.

[jlebon]

But that still means you need to update to the new jigdoRPM first, right? (Which also assumes that there's a new jigdoRPM to update to). Just like OSTree mode right now?

Jigdo does improve the situation in the sense that, at the moment that the jigdoRPM is released, it is in sync with the rest of the RPMs, which is not a guarantee right now. But otherwise, this split issue is still something people can hit, right?

[dustymabe]

Yes, this means you need to do: createrepo_c, compose tree, copy in jigdoRPM, createrepo_c.

yeah I figured that much. Specifically for this issue, though, tomorrows bodhi run fires and no new fedora-atomic-host rpm is in that, now there are newer rpms in the rpm-md than what was used to build the fedora-atomic-host rpm and we have this same problem all over again?

[cgwalters]

But that still means you need to update to the new jigdoRPM first, right?

Yes, but we do that by default on upgrade right?

Specifically for this issue, though, tomorrows bodhi run fires and no new fedora-atomic-host rpm is in that, now there are newer rpms in the rpm-md than what was used to build the fedora-atomic-host rpm and we have this same problem all over again?

I don't quite understand that. Again the design is that the jigdoRPM is contained within and versioned by the rpm-md repo. The jigdoRPM should be regenerated whenever the RPM content changes. However what you may be getting at here is that today the compose tree process doesn't yet support "pure jigdo" mode; the canonical history is still stored in an ostree repo.

[jlebon]

The jigdoRPM should be regenerated whenever the RPM content changes.

Right, assuming that all Bodhi updates are batched and each batch has a jigdoRPM, then there's no issue. But e.g. right now for FAH, we release every two weeks. Whereas Bodhi batches are every week, right? So there's potentially a one week window during which we can get mismatches. Isn't that the same issue we're hitting right now with the stable vs updates ref?

[dustymabe]

I don't quite understand that.

Let me explain. Today we have the stable ref fedora/27/x86_64/atomic-host and the updates ref that gets updated every day fedora/27/x86_64/updates/atomic-host. In jigdo future we have the stable jigdoRPM fedora-atomic-host and an updates jigdoRPM fedora-atomic-host-updates. If we release the stable fedora-atomic-host jigdoRPM today then the rpm-md it was generated from will be out of date tomorrow, just like the stable ostree ref that we use today fedora/27/x86_64/atomic-host.

[dustymabe]

Isn't that the same issue we're hitting right now with the stable vs updates ref?

yes, i think so

[cgwalters]

But e.g. right now for FAH, we release every two weeks. Whereas Bodhi batches are every week, right?

Yes; making full use of jigdo requires that we synchronize those in actuality.

[cgwalters]

I see two paths:

1) We don't differ from Bodhi at all - this may mean we have an ostree update 3 days in a row if one of the RPMs changes each of those days 2) We create /etc/yum.repos.d/fedora-atomic.repo

[jlebon]

Alternatively, we could also add a kind of --allow-replacements switch so we automatically add replace overrides.

[cgwalters]

Alternatively, we could also add a kind of --allow-replacements switch so we automatically add replace overrides.

Yeah true, but I really like the protection we have today against the depsolver deciding to swap out something from the base by default. By default we shouldn't offer depsolve errors and a magic switch "make it work" right?

[dustymabe]

Ok, just so i'm clear on everything.

• this is a problem with jigdo AND ostree transports (i.e. jigdo doesn't solve this problem)
• the most desirable solution is actually in the releng side to make sure rpm repos sync with when we do releases
• an alternative solution is to allow forcing overrides in the base via a command line switch

Some more responses:

We don't differ from Bodhi at all - this may mean we have an ostree update 3 days in a row if one of the RPMs changes each of those days

This takes us back to where we were before we 'slowed down' the cadence. Not saying that is a bad thing, but it is different.

We create /etc/yum.repos.d/fedora-atomic.repo

The problem with this is we don't get all of the rpms from the vast ecosystem that is Fedora. Is there a solution to this that would allow us to keep pulling rpms from the Everything package set but still have a subset of rpms in a separate repo?

[dustymabe]

and one more:

Yeah true, but I really like the protection we have today against the depsolver deciding to swap out something from the base by default. By default we shouldn't offer depsolve errors and a magic switch "make it work" right?

I like that protection too, but we're going to hit this over time. I think we should possibly give a really nasty message to the user about swapping out base packages and then tell them the option they can use to force it to happen.

Another option would be that we don't give them an option but make it an interactive Y/n, which means they are less likely to script it??

[jlebon]

Yeah, I don't think I want to go back to having upgrades everyday. No one wants to upgrade everyday. It seems like the best we can do for this is to detect it as best we can and be helpful about it? E.g. when we hit it, we can check if the cached update has those pkgs and suggest upgrading first. If not, we can suggest an override switch if they really want it.

[dustymabe]

cached update

meaning the update that is cached because of the auto-update checker?

[cgwalters]

The problem with this is we don't get all of the rpms from the vast ecosystem that is Fedora.

Nothing stops us from having that rpm-md repo be Everything, although personally I'd like it to be a subset. I really am pretty sure that it'd be quite cheap to maintain N to N-4 history for even Everything. Just throw S3/Cloudfront/etc at it really.

Another option would be that we don't give them an option but make it an interactive Y/n, which means they are less likely to script it??

I'm not opposed to adding that (you mentioned it above), I just think it's on the provider to synchronize the OS(es) and extensions.

[cgwalters]

Personally though, I favor automation pushing back (strongly?) against non-batched updates in Bodhi that affect Atomic {Host,Workstation}.

[cgwalters]

this is a problem with jigdo AND ostree transports (i.e. jigdo doesn't solve this problem)

Not magically, but it makes it way easier for tooling to keep things in sync.

[cgwalters]

For Fedora, one can work around this today by explicitly using versioned repositories client side, like this:

[fedora-compose]
baseurl=https://kojipkgs.fedoraproject.org/compose/updates/Fedora-28-updates-20180503.0/compose/Everything/$basearch/os
gpgcheck=1
enabled=1

Disable the updates repo.

[cgwalters]

Yeah, I don't think I want to go back to having upgrades everyday. No one wants to upgrade everyday.

I used to agree, but...once one goes to staged automatic updates, it's more about when you want to reboot, and that's obviously user controlled.

this is a problem with jigdo AND ostree transports (i.e. jigdo doesn't solve this problem)

And let me emphasize again - topics like this https://discussion.fedoraproject.org/t/forbidden-base-package-replacements/387/1 wouldn't come up because the things would be in sync always.

Today we cache rpm-md and don't update it within the expiry period, but that's not true of ostree repos, etc.

And yes, big picture to really gain parity we need to version the rpm-md repos too. As I've mentioned elsewhere, in practice Fedora does this today via pungi but libdnf has no idea what a compose is - there's no dnf distro-sync --version 20180831.0/ etc. But fixing that would benefit everything, including container builds.

[dustymabe]

And let me emphasize again - topics like this https://discussion.fedoraproject.org/t/forbidden-base-package-replacements/387/1 wouldn't come up because the things would be in sync always.

i guess I fundamentally don't understand what mechanism is making them be in sync. Is it the cached rpm-md you mention below?

Today we cache rpm-md and don't update it within the expiry period, but that's not true of ostree repos, etc.

What is preventing us from caching rpm-md at the time that we do an rpm-ostree upgrade from an ostree repo and achieving the same goal?

[cgwalters]

i guess I fundamentally don't understand what mechanism is making them be in sync. Is it the cached rpm-md you mention below?

With rojig, there's only one place content comes from - so there's nothing to go out of sync.

[dustymabe]

With rojig, there's only one place content comes from - so there's nothing to go out of sync.

This is the part that i'd like to explore:

Let's say I installed my system with the latest version of rojig rpm/ostree two days ago 29.20180911.0 and today I decide to rpm-ostree install htop. In yesterday's compose (29.20180912.0) new glibc got updated (and let's say htop from today's compose depends on that new version of glibc). Where does htop in the rpm-ostree install htop operation come from? The options are 29.20180911.0 repo, 29.20180912.0 repo, 29.20180913.0 repo, or latest repo (what's currently on the mirrors today).

[jlebon]

I used to agree, but...once one goes to staged automatic updates, it's more about when you want to reboot, and that's obviously user controlled.

Yeah, it's true that AutomaticUpdatePolicy=stage changes the picture a bit.

I think we discussed this at some point, but... another way of solving this would be if Fedora didn't prune previous versions from the repos, right? Which could be easier to implement than "officially" publishing versioned repos. (Versioned metadata could still be a layer on top of that.)

[cgwalters]

and let's say htop from today's compose depends on that new version of glibc

Ah. I see what you're getting at. It is likely indeed that with rojig we probably want install to really be rpm-ostree upgrade --install.

And to follow up...with automatic updates enabled, you probably already have the updated OS data staged.

[dustymabe]

Ah. I see what you're getting at. It is likely indeed that with rojig we probably want install to really be rpm-ostree upgrade --install.

good to know :+1:

Assumption: Fedora updates yum repos and OSTree repo gets updated at approximately the same time, which is the case today.

In the case where you say we want install to be rpm-ostree upgrade --install I don't think rojig is going to help us with this exact problem over what we have today since (I think, please tell me if I'm wrong) that rpm-ostree upgrade --install would solve our problems today with the OSTree repo (at least throughout a normal fedora cycle (exceptions include when we have stable/beta|final/testing composes going on at the same time)).

Could we just solve most of our problem today by detecting a conflict and telling the user to try rpm-ostree upgrade --install instead ? or we could give them a Installing this RPM would require updateing the base OSTree. Continue? (y/N)

[cgwalters]

Assumption: Fedora updates yum repos and OSTree repo gets updated at approximately the same time, which is the case today.

Yeah, but this whole issue is about the current gaps in "approximately". Not to mention that when e.g. enabling testing repositories, users currently need to understand how to simutaneously enable the updates-testingrepostiory and rebase to a testing ref.

I could imagine teaching rpm-ostree and the server side enough intelligence to "tie together" these things - something like each ref having metadata about which rpm-md repos generated it. But IMO the solution is rojig.

or we could give them a Installing this RPM would require updateing the base OSTree. Continue? (y/N)

In general, I am not a fan of these sorts of dynamic interactive questions - I think they're very often indicative of bad design.

See also https://ometer.com/preferences.html

That doesn't mean we shouldn't have any questions like this, but in this specific case, I think we should take the "hard path" (rojig) and save our users confusing questions.

(Another important thing here is that any questions like this would have to be reimplemented in gnome-software and Cockpit)

[dustymabe]

Not to mention that when e.g. enabling testing repositories, users currently need to understand how to simutaneously enable the updates-testingrepostiory and rebase to a testing ref.

I could imagine teaching rpm-ostree and the server side enough intelligence to "tie together" these things - something like each ref having metadata about which rpm-md repos generated it.

yeah that would be nice

or we could give them a Installing this RPM would require updateing the base OSTree. Continue? (y/N)

In general, I am not a fan of these sorts of dynamic interactive questions - I think they're very often indicative of bad design.

See also https://ometer.com/preferences.html

That doesn't mean we shouldn't have any questions like this, but in this specific case, I think we should take the "hard path" (rojig) and save our users confusing questions.

(Another important thing here is that any questions like this would have to be reimplemented in gnome-software and Cockpit)

Then we just make it not an option and upgrade the user by default just like in rojig. Solves the same problem. Then all that's left is the updates vs updates-testing problem mentioned above.

[dustymabe]

Then we just make it not an option and upgrade the user by default just like in rojig.

although I think this would render rpm-ostree install && rpm-ostree ex livefs useless because we would always be updating the base and requiring a reboot, right?

[edwintorok]

Edit: this actually seems to be a problem with the repository itself, libxcrypt-devel-4.4.2-3.fc29 is claimed to be not available at all, although it is available on koji: https://github.com/projectatomic/rpm-ostree/issues/1724

It would be useful if rpm-ostree printed more details on which package is causing the problem, I just ran into this problem today with a very generic Some base packages would be replaced:

sudo rpm-ostree override reset fuse-overlayfs
sudo rpm-ostree upgrade
note: automatic updates (stage) are enabled
1 metadata, 0 content objects fetched; 569 B transferred in 7 seconds
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
Updating metadata for 'fedora'... done
rpm-md repo 'fedora'; generated: 2018-10-24T22:20:15Z
Updating metadata for 'updates'... done
rpm-md repo 'updates'; generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
⠁ Resolving dependencies... 
Forbidden base package replacements:
Resolving dependencies... done
error: Some base packages would be replaced

I tried using --uninstall, but it still complained:

sudo rpm-ostree upgrade --uninstall ansible bcc-tools chromium dconf-editor evince evolution fedora-toolbox firefox-wayland gmp-devel gnome-photos gnome-tweak-tool kernel-tools latencytop libreoffice mozilla-ublock-origin ncurses-term neovim ocaml ocamldoc opam pass perf perl-Pod-Html python3-psutil ripgrep stow strace tig weston zsh
[sudo] password for edwin: 
note: automatic updates (stage) are enabled
1 metadata, 0 content objects fetched; 569 B transferred in 5 seconds
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'updates' (cached); generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
⠁ Resolving dependencies... 
Forbidden base package replacements:
Resolving dependencies... done
error: Some base packages would be replaced

Finally I pinned the current deployment, uninstalled all layers, upgraded, used rpm-ostree install -n to narrow down the package (in this case ocaml):

sudo rpm-ostree uninstall --all
[...]
sudo rpm-ostree upgrade
note: automatic updates (stage) are enabled
1 metadata, 0 content objects fetched; 569 B transferred in 3 seconds
Staging deployment... done
Upgraded:
  container-selinux 2:2.76-1.git87fae85.fc29 -> 2:2.77-1.git2c57a17.fc29
  libxcrypt 4.4.2-1.fc29 -> 4.4.2-3.fc29
  runc 2:1.0.0-59.dev.gitccb5efd.fc29 -> 2:1.0.0-66.dev.gitbbb17ef.fc29
  zchunk-libs 0.9.17-1.fc29 -> 1.0.0-1.fc29
[...]
for i in bcc-tools dconf-editor evince evolution fedora-toolbox firefox-wayland gnome-photos gnome-tweak-tool kernel-tools latencytop mozilla-ublock-origin ncurses-term neovim ocaml ocamldoc opam perf ripgrep stow strace tig weston zsh; do echo $i; sudo rpm-ostree install -n $i; done
[...]
sudo rpm-ostree install ocaml
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'updates' (cached); generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
⠁ Resolving dependencies... 
Forbidden base package replacements:
Resolving dependencies... done
error: Some base packages would be replaced

However it still doesn't show which package in the base packages would be the problem...

Here is my current status:

sudo rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
  ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181227.0 (2018-12-27T01:09:18Z)
                BaseCommit: a6d00a98f2bfb16e3cdae4f89f306a5035c7e67ae15012ccf3d757c11c4b8624
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
           LayeredPackages: ansible bcc-tools chromium dconf-editor evince evolution fedora-toolbox firefox-wayland gmp-devel gnome-photos gnome-tweak-tool kernel-tools
                            latencytop libreoffice mozilla-ublock-origin ncurses-term neovim pass perf perl-Pod-Html python3-psutil ripgrep stow strace tig weston zsh

● ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181225.0 (2018-12-25T01:13:26Z)
                BaseCommit: 9c2273bb044c6b0a36c0e1549011091afd79f95700f27a568f5964ac12c7eb32
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
      ReplacedBasePackages: fuse-overlayfs 0.1-6.dev.git3d48bf9.fc29 -> 0.1-8.dev.git91bb401.fc29
           LayeredPackages: ansible bcc-tools chromium dconf-editor evince evolution fedora-toolbox firefox-wayland gmp-devel gnome-photos gnome-tweak-tool kernel-tools
                            latencytop libreoffice mozilla-ublock-origin ncurses-term neovim ocaml ocamldoc opam pass perf perl-Pod-Html python3-psutil ripgrep stow strace
                            tig weston zsh
                    Pinned: yes

  ostree://fedora-workstation:fedora/29/x86_64/silverblue
                   Version: 29.20181224.0 (2018-12-24T01:17:14Z)
                BaseCommit: a071235dd160f942385a1c2472521ef1906e066b26083e69fce734e1ac71a803
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
      ReplacedBasePackages: fuse-overlayfs 0.1-6.dev.git3d48bf9.fc29 -> 0.1-8.dev.git91bb401.fc29
           LayeredPackages: ansible bcc-tools chromium dconf-editor evince evolution fedora-toolbox firefox-wayland gmp-devel gnome-photos gnome-tweak-tool kernel-tools
                            latencytop libreoffice mozilla-ublock-origin ncurses-term neovim ocaml ocamldoc opam pass perf perl-Pod-Html python3-psutil ripgrep stow strace
                            tig weston zsh

AvailableUpdate:
        Version: 29.20181227.0 (2018-12-27T01:09:18Z)
         Commit: a6d00a98f2bfb16e3cdae4f89f306a5035c7e67ae15012ccf3d757c11c4b8624
   GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4
           Diff: 4 upgraded, 232 removed

Edit: in this case the problem was libxcrypt-devel (needed by glibc-devel, needed by gcc, needed by ocaml-runtime, needed by ocaml):

sudo rpm-ostree install -n libxcrypt
error: "libxcrypt" is already provided by: libxcrypt-4.4.2-3.fc29.x86_64. Use --allow-inactive to explicitly require it.
edwin@bolt:~ % sudo rpm-ostree install -n libxcrypt-devel
Checking out tree a6d00a9... done
Enabled rpm-md repositories: fedora updates
rpm-md repo 'fedora' (cached); generated: 2018-10-24T22:20:15Z
rpm-md repo 'updates' (cached); generated: 2018-12-26T02:39:50Z
Importing rpm-md... done
⠁ Resolving dependencies... 
Forbidden base package replacements:
Resolving dependencies... done
error: Some base packages would be replaced

Note that rpm-ostree refresh-md and rpm-ostree cleanup -m didn't fix the problem, libxcrypt-devel is still uninstallable.

[jistr]

I'm seeing this when trying to install gcc on Atomic host 29.20190219.0:

Forbidden base package replacements:
  libgcc 8.2.1-6.fc29 -> 8.3.1-2.fc29 (updates)
  libxcrypt 4.4.3-2.fc29 -> 4.4.3-10.fc29 (updates)
  libgomp 8.2.1-6.fc29 -> 8.3.1-2.fc29 (updates)

As soon as the update repos get out of sync from the latest base layer, installing some RPMs becomes impossible. (rpm-ostree cleanup -m doesn't help.) IMO this issue shouldn't be tagged just "enhancement", it's a significant usability bug.

Until a real fix is implemented, would it help to build Atomic host base layers with a nightly cadence?

[cgwalters]

Until a real fix is implemented, would it help to build Atomic host base layers with a nightly cadence?

That exists; just rpm-ostree rebase fedora-atomic:fedora/29/x86_64/updates/atomic-host

(But I'd recommend doing development in a container, not the host)

[jistr]

@cgwalters Thanks! Rebasing to the updates base layer helped.

(Re development in container i agree, we're perhaps using Atomic in a weird way here. We're trying to make a multi-user workstation without forcing everyone to work in containers all the time. We picked Atomic mainly because multiple users have admin rights on the machine, and in this situation Atomic should accumulate less bit rot over time than traditional distros.)

[dustymabe]

@cgwalters Thanks! Rebasing to the updates base layer helped.

@jistr - I do want to point out that using the updates base layer (fedora/29/x86_64/updates/atomic-host) works, but it goes through less testing than the stable ref (fedora/29/x86_64/atomic-host) that only gets releases every two weeks. Just something to consider when making this decision.

[jlebon]

It would be useful if rpm-ostree printed more details on which package is causing the problem, I just ran into this problem today with a very generic Some base packages would be replaced:

This will be solved by https://github.com/coreos/rpm-ostree/pull/2125.