[Features] Leaverage transitive inclusion and build source-features

HannesWell commented 2 years ago

This PR proposes to build source-features for all non-sdk features of equinox-bundles and to simplify the feature content by leveraging transitive inclusion and the now generated source features.

Leveraging transitive inclusion avoids effective duplicates and reduces the maintenance effort. Furthermore it becomes more unlikely to forget to include a source-plugin. My intention is also to reduce the subsequent maintenance effort of https://github.com/eclipse-equinox/equinox.framework/pull/41.

Besides that, it removes the .feature suffix from the project-name org.eclipse.equinox.server.core.feature to match the feature name.

HannesWell commented 2 years ago

The initial target-platform resolution fails with the following errors:

[INFO] {osgi.os=linux, osgi.ws=gtk, org.eclipse.update.install.features=true, osgi.arch=x86_64}
[ERROR] Cannot resolve project dependencies:
[ERROR]   Software being installed: org.eclipse.equinox.sdk.feature.group 3.23.400.qualifier
[ERROR]   Missing requirement: org.eclipse.equinox.core.sdk.feature.group 3.23.400.qualifier requires 'org.eclipse.equinox.p2.iu; org.eclipse.equinox.core.feature.source.feature.group 0.0.0' but it could not be found
[ERROR]   Cannot satisfy dependency: org.eclipse.equinox.sdk.feature.group 3.23.400.qualifier depends on: org.eclipse.equinox.p2.iu; org.eclipse.equinox.core.sdk.feature.group 0.0.0
[ERROR] 
[ERROR] See https://wiki.eclipse.org/Tycho/Dependency_Resolution_Troubleshooting for help.

[INFO] Resolving dependencies of MavenProject: org.eclipse.equinox.feature:org.eclipse.equinox.core.sdk:3.23.400-SNAPSHOT @ /home/jenkins/agent/workspace/rt.equinox.bundles_PR-20/features/org.eclipse.equinox.core.sdk/pom.xml
[INFO] {osgi.os=linux, osgi.ws=gtk, org.eclipse.update.install.features=true, osgi.arch=x86_64}
[ERROR] Cannot resolve project dependencies:
[ERROR]   Software being installed: org.eclipse.equinox.core.sdk.feature.group 3.23.400.qualifier
[ERROR]   Missing requirement: org.eclipse.equinox.core.sdk.feature.group 3.23.400.qualifier requires 'org.eclipse.equinox.p2.iu; org.eclipse.equinox.core.feature.source.feature.group 0.0.0' but it could not be found
[ERROR] 
[ERROR] See https://wiki.eclipse.org/Tycho/Dependency_Resolution_Troubleshooting for help.

[INFO] Resolving dependencies of MavenProject: org.eclipse.equinox.feature:org.eclipse.equinox.serverside.sdk:3.23.400-SNAPSHOT @ /home/jenkins/agent/workspace/rt.equinox.bundles_PR-20/features/org.eclipse.equinox.serverside.sdk/pom.xml
[INFO] {osgi.os=linux, osgi.ws=gtk, org.eclipse.update.install.features=true, osgi.arch=x86_64}
[ERROR] Cannot resolve project dependencies:
[ERROR]   Software being installed: org.eclipse.equinox.serverside.sdk.feature.group 3.23.400.qualifier
[ERROR]   Missing requirement: org.eclipse.equinox.serverside.sdk.feature.group 3.23.400.qualifier requires 'org.eclipse.equinox.p2.iu; org.eclipse.equinox.server.core.source.feature.group 0.0.0' but it could not be found
[ERROR] 
[ERROR] See https://wiki.eclipse.org/Tycho/Dependency_Resolution_Troubleshooting for help.

I'm puzzled why this fails. @laeubi @mickaelistria do you have any idea? In m2e we use the same pattern for the m2e.sdk feature that also includes source-features that are generated during build: https://github.com/eclipse-m2e/m2e-core/blob/master/org.eclipse.m2e.sdk.feature/feature.xml

The only difference I see is that Tycho-pomless is not (yet but I'm working on it) used in equinox-bundles and the features of their own pom.xml.

HannesWell commented 2 years ago

Adding the following configuration to the tycho-p2-plugin did not help either (I didn't expect much, but hope dies last).

        <configuration>
          <defaultP2Metadata>false</defaultP2Metadata>
          <baselineReplace>common</baselineReplace>
        </configuration>

But I think I have to add <baselineReplace>common</baselineReplace> anyway until it has been added to the base-line.

laeubi commented 2 years ago

I have no idea, but would start with the bare minimum and only remove the source plugins/features instead of any "transitive" for me this is causing more trouble as is really solves and the real issue we should care is https://github.com/eclipse/tycho/issues/898

merks commented 2 years ago

I've run into problems similar to this too. I think it's a bit of a chicken and egg problem because the source is generated so is not available up front somehow. It seems to me I had to do this to make the problem go away at the root level.

https://git.eclipse.org/c/oomph/org.eclipse.oomph.git/tree/features/org.eclipse.oomph.all-feature/p2.inf

This is the particular feature that strictly controls what is and is not in Oomph's update site via transitive inclusion...

Let me look if there are any other things I can think of...

merks commented 2 years ago

I assume that this is configured for these things in the right places:

  <build>
    <plugins>
      <plugin>
        <groupId>org.eclipse.tycho.extras</groupId>
        <artifactId>tycho-source-feature-plugin</artifactId>
        <executions>
          <execution>
            <id>generate-source-feature</id>
            <goals>
              <goal>source-feature</goal>
            </goals>
            <configuration>
              <excludes>
                <plugin id="org.apache.commons.codec"/>
                <plugin id="org.apache.commons.logging"/>
                <plugin id="org.apache.httpcomponents.client5.httpclient5"/>
                <plugin id="org.apache.httpcomponents.core5.httpcore5"/>
                <plugin id="org.apache.httpcomponents.core5.httpcore5-h2"/>
                <plugin id="org.apache.httpcomponents.httpclient"/>
                <plugin id="org.apache.httpcomponents.httpcore"/>
                <plugin id="org.bouncycastle.bcpg"/>
                <plugin id="org.bouncycastle.bcprov"/>
                <plugin id="org.slf4j.api"/>
                <plugin id="org.eclipse.nebula.widgets.tablecombo"/>
                <plugin id="org.eclipse.oomph.releng.helpcenter"/>
              </excludes>
            </configuration>
          </execution>
        </executions>
      </plugin>

HannesWell commented 2 years ago

Thanks for your help. I will look into this tomorrow, for today I unfortunately have no time left.

HannesWell commented 2 years ago

The first reason of the failure was that Tycho requires a specific execution-id for the tycho-source-plugin:feature-source goal in order to consider the source-features that will be generated during the initial dependency resolution. I have already created a PR to improve that: https://github.com/eclipse/tycho/pull/912

A second reason this is failing is that the tycho-p2-plugin is removing source-features that are not present in the baseline. I hope this is solved with corresponding enforced qualifier updates.

HannesWell commented 2 years ago

Great, that worked well. I just reverted the unnecessary qualifier updates for the o.e.equinox.console.jaas.fragment (it doesn't need a source bundle) as well as the unnecessary version bumps of the o.e.equinox.core.feature. The the remaining enforced qualifier updates are necessary because the p2-plugins baseline is the latest I-build repo. So in order not get the use the newly introduced source features removed their build qualifier has to be updated.

The new source features are:

org.eclipse.equinox.core.feature.source
org.eclipse.equinox.server.core.source
org.eclipse.equinox.server.jetty.source
org.eclipse.equinox.server.p2.source

HannesWell commented 2 years ago

I double checked that no plug-ins are effectively removed, so I think this is ready for submission.

HannesWell commented 2 years ago

I assume that this is configured for these things in the right places:
      <plugin>
        <groupId>org.eclipse.tycho.extras</groupId>
        <artifactId>tycho-source-feature-plugin</artifactId>
      </plugin>

@merks the tycho-source-feature-plugin is deprecated and I assume it is will be removed in Tycho 3.0.0 (@laeubi can you confirm this?). In Tycho 2.7.2/3.0.0 you can replace it with the org.eclipse.tycho:tycho-source-plugin and its feature-source goal. For older versions of Tycho you have to make sure the goal's execution has the id feature-source. :)

merks commented 2 years ago

@HannesWell Yes, I remember updating some of my builds to use the new one, but the Oomph build still uses the old one.

There are still these uses in the platform repos:

And I thought the builds were being done with some 3.0.* version of Tycho...

HannesWell commented 2 years ago

Thanks for the hints. I just created the following PRs to address this:

And I thought the builds were being done with some 3.0.* version of Tycho...

That's right, the tycho-source-feature-plugin is not yet removed. But I expect it will be removed before the 3.0.0 release. At least a new major version would be a good opportunity to remove deprecated code. But I don't know if there are reasons against a removal or if it just was not done yet, maybe the remaining usages are the 'blocker'?

laeubi commented 2 years ago

But I don't know if there are reasons against a removal or if it just was not done yet, maybe the remaining usages are the 'blocker'?

No, there is a clear warning and it is marked for removal in 3.x and there is a replacement for it so no need to retain it.

eclipse-equinox / equinox.bundles

[Features] Leaverage transitive inclusion and build source-features #20