Support "deep" updates

[mickaelistria]

Currently, p2 updates simply look up for updates of top-level dependencies. But many fixes and improvements do come from transitive dependencies for which newer version may be available without the installed or update content actually requiring an update. In the current case, such updates are ignored; p2 should instead provide a way to get the transitive dependencies updated as well.

In term of implementation, I suspect it's mostly a matter of feeding the planner with all units from the profile instead of only the top-level ones.

[HannesWell]

I assume that this would be important to really get rid of third-party deps in features. As far as I know (but I have to admit I didn't check it), without this P2 would only update a transitive dependency if a 'root' unit would require it (e.g. through a corresponding version range), otherwise the dependency is not updated.

@mickaelistria do you plan to implement this? If not maybe @merks or @laeubi are interested?

[mickaelistria]

Yes, p2 is by default "economic", ie it will update as little as possible to satisfy the constraints of existing+provisioning request. I don't plan to work on it anytime soon, and would really be pleased to see someone trying to tackle it. However, I don't think it's an easy task. p2 actually builds a SAT problem and SAT problems which are NP-problems. So there is a risk that if we can't deal with such updates without trimming the problem to a simpler subset, the complexity gets exponential and basically update would take ages to complete... IMO, 1 approach would be that update first: update "roots" as they're doing right now, and once they have a working plan, "just" try whether upgrading individual bundles from this plan allows to create another valid provisioning plan an iterate on all bundles; but even that would take ages I'm afraid...

[laeubi]

I think the main "problem" is that P2 uses the lowest possible version (as far as I know), so actually the solution would be that the consumer uses a proper version range and raises the lower bound if there is a critical problem, but I think @merks is the better person to ask here because the Update-Stuff is a mystery to me and how P2 decides what needs updates and its a mess to test without a deployed product.

[merks]

As I understand it, p2 basically makes a full plan based on the root IUs that one wants to install or update. So definitely it won't update anything if there is no new root IU under consideration. I'm not sure it really cares about keeping the set of updates minimal, because I'm pretty sure the SAT4J solver doesn't know about the current state but rather is asked to compute a new state...

The key point here, as @mickaelistria suggests, is about the root IUs. So in this minimal case, for example, the SDK does require some of EMF's features to be installed. These features are imported, not included, so they can be updated without needing the SDK to be updated. Checking for updates, when there are definitely updates available, does not report any updates:

I can try to install what's already installed first:

Which doesn't actually install anything new but afterwards it's listed as a root:

And now check for updates will find updates:

What's not entirely clear from this is that if there were updates available to IUs that are not included in a feature being updated whether or not that IU would be included in the solution. What is clear that that if any IU actually being updated requires an update to some other IU, because a lower bound requires that, the solution would necessarily include that update too.

One thing that Oomph does differently with its use of p2 relative to the Platform's direct use of p2, is that Oomph builds an artificial root IU within which it creates requirements with version ranges on all the root IUs. So for example, when we do Help -> Perform Setup Tasks and there is a new version of EMF available in the available repositories we see this:

The computed solution does find updates for EMF and installs them.

Executing manual tasks OpenJDK Runtime Environment 17.0.4.1+1 Product org.eclipse.applications.eclipse.platform.sdk.latest.released Workspace D:\Users\test20\workspace-4 Bundle org.eclipse.oomph.setup 1.26.0.v20230509-0723, build=Local, branch=Unknown Bundle org.eclipse.oomph.setup.core 1.27.0.v20230402-0730, build=Local, branch=Unknown Bundle org.eclipse.oomph.setup.p2 1.19.0.v20230416-1150, build=Local, branch=Unknown Performing P2 Director (Eclipse SDK (4.27 - 2023-03)) Offline = false Mirrors = true Resolving 2 requirements from 4 repositories to D:\Users\test20\sdk-latest-released\eclipse Requirement org.eclipse.equinox.p2.iu:org.eclipse.sdk.ide 2.28.0.v20230315-1321(&(osgi.arch=aarch64)(osgi.os=macosx)(osgi.ws=cocoa))(&(osgi.arch=ppc64le)(osgi.os=linux)(osgi.ws=gtk))(&(osgi.arch=x86_64)(osgi.os=linux)(osgi.ws=gtk))(&(osgi.arch=x86_64)(osgi.os=macosx)(osgi.ws=cocoa))(&(osgi.arch=x86_64)(osgi.os=win32)(osgi.ws=win32)))) Requirement org.eclipse.equinox.p2.iu:org.eclipse.oomph.setup.feature.group Repository https://download.eclipse.org/eclipse/updates/4.27/R-4.27-202303020300 Repository file:/D:/Users/merks/oomph-p2-1.29/git/org.eclipse.oomph/sites/org.eclipse.oomph.site/target/repository/ Repository https://download.eclipse.org/modeling/emf/emf/builds/release/2.33.0 Repository https://download.eclipse.org/modeling/emf/emf/builds/milestone/latest Adding metadata repository https://download.eclipse.org/modeling/emf/emf/builds/release/2.33.0 Adding metadata repository file:/D:/Users/merks/oomph-p2-1.29/git/org.eclipse.oomph/sites/org.eclipse.oomph.site/target/repository/ Adding metadata repository https://download.eclipse.org/eclipse/updates/4.27/R-4.27-202303020300 Fetching p2.index from https://download.eclipse.org/modeling/emf/emf/builds/milestone/latest/ (126B) Adding metadata repository https://download.eclipse.org/modeling/emf/emf/builds/milestone/latest Fetching compositeContent.jar from https://download.eclipse.org/modeling/emf/emf/builds/milestone/latest/ (437B) Fetching p2.index from https://download.eclipse.org/modeling/emf/emf/builds/milestone/S202304250538/ (172B) Fetching content.xml.xz from https://download.eclipse.org/modeling/emf/emf/builds/milestone/S202304250538/ (29,84kB) Calculating requirements and dependencies. Computing prerequisite plan Adding artifact repository file:/D:/Users/test20/.p2/pool/ Adding artifact repository file:/D:/Users/merks/oomph-p2-1.29/git/org.eclipse.oomph/sites/org.eclipse.oomph.site/target/repository/ Adding artifact repository https://download.eclipse.org/modeling/emf/emf/builds/release/2.33.0 Adding artifact repository https://download.eclipse.org/eclipse/updates/4.27/R-4.27-202303020300 Adding artifact repository https://download.eclipse.org/modeling/emf/emf/builds/milestone/latest Fetching compositeArtifacts.jar from https://download.eclipse.org/modeling/emf/emf/builds/milestone/latest/ (442B) Fetching artifacts.xml.xz from https://download.eclipse.org/modeling/emf/emf/builds/milestone/S202304250538/ (18,42kB) Installing org.eclipse.emf.common [2.28.0.v20230315-1321] Installing org.eclipse.emf.common.feature.group [2.29.0.v20230315-1321] Installing org.eclipse.emf.common.source [2.28.0.v20230315-1321] Installing org.eclipse.emf.common.source.feature.group [2.29.0.v20230315-1321] Installing org.eclipse.emf.ecore [2.34.0.v20230406-1203] Installing org.eclipse.emf.ecore.feature.group [2.32.0.v20230406-1203] Installing org.eclipse.emf.ecore.source [2.34.0.v20230406-1203] Installing org.eclipse.emf.ecore.source.feature.group [2.32.0.v20230406-1203] Installing org.eclipse.emf.common.feature.jar [2.29.0.v20230315-1321] Installing org.eclipse.emf.common.source.feature.jar [2.29.0.v20230315-1321] Installing org.eclipse.emf.ecore.feature.jar [2.32.0.v20230406-1203] Installing org.eclipse.emf.ecore.source.feature.jar [2.32.0.v20230406-1203] Preparing to commit the provisioning operation. Committing the provisioning operation. Took 4 seconds. A restart is needed for the following reasons:

• New software has been installed. Press Finish to restart now or Cancel to restart later.

Note that we did not need to feed p2 all the IUs. That this would not even work properly because at the top level p2 is given specific IUs (with specific versions) which is it supposed to try to install/update. What we did in Oomph is feed p2 an IU that requires the two root features with version ranges that allow updates to that root features. The overall solution finds deeply nested updates that are a new and improved valid solution to the problem.

Probably we want at least an option for p2 to behave this way and we have in Oomph a working example. Alternatively we would need to consider directly more than just the root IUs, i.e., perhaps all the features IUs in tree being displayed.

[laeubi]

In term of implementation, I suspect it's mostly a matter of feeding the planner with all units from the profile instead of only the top-level ones.

One also need to take into account that there could be multiple versions of a unit installed (e.g. different bundle version) and even if thats not the case that updating a bundle is not always valid (e.g. it is a new major version but import ranges do not allow to use them), so I think an implementation has to work the following way:

1. Collect the root IUs as it is currently done
2. Compute all requirements as if it would install these root IUs
3. Compare the result with the current state and see if any of the existing items would use a new version and offer this for update as well

Maybe this is what is described with the Oomph case above, but I think we should not limit this to features but any IU, I think the Eclipse-SDK IDE uses something similar when it computes the target by just collecting all dependencies of the current bundles.