search

eclipse-packaging / packages

Eclipse IDE product definitions.
Eclipse Public License 2.0
4 stars 11 forks source link

Exclude httpclientjava #81

Closed jonahgraham closed 9 months ago

jonahgraham commented 10 months ago

Because the Eclipse p2 changed to use a new ECF provider that is known to not work for some consumers we need to exclude that new provider in EPP by adding:

-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclientjava

For additional details see https://github.com/eclipse-equinox/p2/issues/381 and N&N entry

cc: @merks

merks commented 10 months ago

I’ve confirmed via tracing in the installer that this works.

Also a user has confirmed that the installer works again in their network

https://github.com/eclipse-oomph/oomph/issues/46

Of course this should be done for rc1 not m3.

merks commented 10 months ago

Of course if you include the old provider in the products for m3 then doing this for m3 is fine and good...

jonahgraham commented 10 months ago

Something has gone wrong - I didn't get the same results on the build machine as I got when building locally and the produced builds are not working. I tested locally with the rcp package. Best I can tell because of this change.

I get runtime wiring errors related to httpclient, for rcp they seem to be relatively minor (broken, but not too much broken), but for cpp it is more catastrophic.

I may have to roll back this fix for M3 and keep trying.

merks commented 10 months ago

Is MPC’s stale dependency on older versions a problem?

jonahgraham commented 10 months ago

I'm running a build without the change to make sure I have something for M3 today while I try to find the cause.

jonahgraham commented 10 months ago

Is MPC’s stale dependency on older versions a problem?

Yes - I think so. I'm away from my main dev computer at the moment trying to solve this. I will post a full error message soon.

jonahgraham commented 10 months ago

Here are some of the wiring errors:


!ENTRY org.eclipse.epp.mpc.core 4 0 2023-11-16 15:04:11.456
!MESSAGE FrameworkEvent ERROR
!STACK 0
org.osgi.framework.BundleException: Could not resolve module: org.eclipse.epp.mpc.core [228]
  Unresolved requirement: Import-Package: org.apache.hc.client5.http; version="[5.1.0,5.2.0)"

!ENTRY org.eclipse.epp.mpc.core.win32 4 0 2023-11-16 15:04:11.456
!MESSAGE FrameworkEvent ERROR
!STACK 0
org.osgi.framework.BundleException: Could not resolve module: org.eclipse.epp.mpc.core.win32 [229]
  Unresolved requirement: Import-Package: org.apache.hc.client5.http.auth; version="[5.1.0,5.2.0)"

I had many more errors on my Linux build and I will retry with that later.

merks commented 10 months ago

I noticed that looking at duplicates. I didn’t consider it might cause wiring problems. It’s beyond frustrating that 1/2 the projects just don’t bother to provide updates and have crazy restrictive ranges.

jonahgraham commented 10 months ago

I tried to "just" include the same feature that Eclipse Platform will be delivering in RC1 - so I guess RC1 from Platform won't solve the problem and we'll be in the same boat.

merks commented 10 months ago

I will test the egg laying wool milk pig tomorrow which I expect will have the same problem. Then I will need to poke MPC to update their contribution ASAP. It’s beyond annoying…

jonahgraham commented 10 months ago

It looks like when Tycho/p2 pulled in httpclient 5.1 and 5.2. And with both of them installed we end up with uses violations:

  Bundle was not resolved because of a uses constraint violation.
  org.apache.felix.resolver.reason.ReasonException: Uses constraint violation. Unable to resolve resource org.eclipse.epp.mpc.core [osgi.identity; type="osgi.bundle"; version:Version="1.10.1.v20221110-1841"; osgi.identity="org.eclipse.epp.mpc.core"; singleton:="true"] because it is exposed to package 'org.apache.hc.client5.http' from resources org.apache.httpcomponents.client5.httpclient5 [osgi.identity; type="osgi.bundle"; version:Version="5.1.3.v20221013-1742"; osgi.identity="org.apache.httpcomponents.client5.httpclient5"] and org.apache.httpcomponents.client5.httpclient5 [osgi.identity; type="osgi.bundle"; version:Version="5.2.1.v20230802-0806"; osgi.identity="org.apache.httpcomponents.client5.httpclient5"] via two dependency chains.

Chain 1:
  org.eclipse.epp.mpc.core [osgi.identity; type="osgi.bundle"; version:Version="1.10.1.v20221110-1841"; osgi.identity="org.eclipse.epp.mpc.core"; singleton:="true"]
    import: (&(osgi.wiring.package=org.apache.hc.client5.http)(&(version>=5.1.0)(!(version>=5.2.0))))
     |
    export: osgi.wiring.package: org.apache.hc.client5.http
  org.apache.httpcomponents.client5.httpclient5 [osgi.identity; type="osgi.bundle"; version:Version="5.1.3.v20221013-1742"; osgi.identity="org.apache.httpcomponents.client5.httpclient5"]

Chain 2:
  org.eclipse.epp.mpc.core [osgi.identity; type="osgi.bundle"; version:Version="1.10.1.v20221110-1841"; osgi.identity="org.eclipse.epp.mpc.core"; singleton:="true"]
    import: (&(osgi.wiring.package=org.apache.hc.client5.http.impl.classic)(&(version>=5.1.0)(!(version>=5.2.0))))
     |
    export: osgi.wiring.package=org.apache.hc.client5.http.impl.classic; uses:=org.apache.hc.client5.http.routing
  org.apache.httpcomponents.client5.httpclient5 [osgi.identity; type="osgi.bundle"; version:Version="5.1.3.v20221013-1742"; osgi.identity="org.apache.httpcomponents.client5.httpclient5"]
    import: (&(osgi.wiring.package=org.apache.hc.client5.http.routing)(&(version>=5.1.0)(!(version>=6.0.0))))
     |
    export: osgi.wiring.package: org.apache.hc.client5.http.routing; uses:=org.apache.hc.client5.http
    export: osgi.wiring.package=org.apache.hc.client5.http
  org.apache.httpcomponents.client5.httpclient5 [osgi.identity; type="osgi.bundle"; version:Version="5.2.1.v20230802-0806"; osgi.identity="org.apache.httpcomponents.client5.httpclient5"]
        at org.eclipse.osgi.container.Module.start(Module.java:463)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1852)
        at org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$1$1.execute(EquinoxContainerAdaptor.java:136)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1845)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1786)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1750)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1672)
        at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1)
        at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234)
        at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345)
merks commented 10 months ago

Yes that’s what the *.aggran showed as the reason for the multiple versions. MPC excludes the latest version. Goodness knows why the narrow bounds. Goodness knows why so many just ignore the information they receive.

jonahgraham commented 10 months ago

@merks :-( there is an error in apache's http 5 components. The import packages have too wide a range. http 5 components should allow 5.1 and 5.2 to coexist as they have all the uses clauses needed to make sure that it is wired up properly. But its imports are problematic as it allows imports to mix and match versions.

Because one part of httpcomponents imports other parts with lines like (simplified):

Bundle-Version: 5.1.3.v20221013-1742
Import-Package: org.apache.hc.core5.http;version="[5.1.3,6)"

But that means p2 can use org.apache.httpcomponents.core5.httpcore5 v5.2.3 to resolve the import. That can lead to use case problems and leads to strange resolutions by p2 where, for example, I have this set of plug-ins in one of my EPPs (CPP on Windows):

  1. org.apache.httpcomponents.client5.httpclient5_5.1.3.v20221013-1742.jar
  2. org.apache.httpcomponents.core5.httpcore5-h2_5.2.3.v20230922-1600.jar
  3. org.apache.httpcomponents.client5.httpclient5-win_5.1.3.v20221013-1742.jar
  4. org.apache.httpcomponents.httpclient_4.5.14.jar
  5. org.apache.httpcomponents.core5.httpcore5_5.1.4.v20221013-1742.jar
  6. org.apache.httpcomponents.httpcore_4.4.16.v20221207-1049.jar
  7. org.apache.httpcomponents.core5.httpcore5_5.2.3.v20230922-1600.jar

So EPP for CPP on Windows with M3 simrel (and #82 reverted) MPC still doesn't work:


!ENTRY org.eclipse.epp.mpc.core 4 0 2023-11-16 18:55:20.933
!MESSAGE FrameworkEvent ERROR
!STACK 0
org.osgi.framework.BundleException: Could not resolve module: org.eclipse.epp.mpc.core [224]
  Unresolved requirement: Import-Package: org.apache.hc.core5.http; version="[5.1.0,5.2.0)"

But I can see p2 coming to different solutions

merks commented 10 months ago

FYI, I have generated the catalogs for m3 to add the -D option and to add the fixed MPC update site...

jonahgraham commented 10 months ago

You also need to add a dependency on the httpclient ECF provider otherwise the only provider will be excluded IIUC.

merks commented 10 months ago

Good point! That's added too now.

jonahgraham commented 9 months ago

I have tested the current staging repo with the CPP package and this change looks good. I don't have a way to test with the problematic proxies, but I can see that with this change a non-existent host URL throws and exception in Apache's client now:

With this change you can see org.eclipse.ecf.provider.filetransfer.httpclient5

ENTRY org.eclipse.equinox.p2.transport.ecf 2 0 2023-11-22 12:20:13.252
!MESSAGE Connection to https://asdfasdfasfsdfdsfwerfwefwefwef.com/p2.index failed on asdfasdfasfsdfdsfwerfwefwefwef.com: Name or service not known. Retry attempt 0 started
!STACK 0
java.net.UnknownHostException: asdfasdfasfsdfdsfwerfwefwefwef.com: Name or service not known
    at java.base/java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
    at java.base/java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934)
    at java.base/java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543)
    at java.base/java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852)
    at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1533)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1385)
    at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1306)
    at org.apache.hc.client5.http.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:43)
    at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141)
    at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:447)
    at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:162)
    at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:172)
    at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:142)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:192)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:96)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:152)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:115)
    at org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51)
    at org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:170)
    at org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:106)
    at org.eclipse.ecf.provider.filetransfer.httpclient5.HttpClientRetrieveFileTransfer.performConnect(HttpClientRetrieveFileTransfer.java:1009)
    at org.eclipse.ecf.provider.filetransfer.httpclient5.HttpClientRetrieveFileTransfer$1.performFileTransfer(HttpClientRetrieveFileTransfer.java:997)
    at org.eclipse.ecf.filetransfer.FileTransferJob.run(FileTransferJob.java:76)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

With the -D removed from eclipse.ini you can see org.eclipse.ecf.provider.filetransfer.httpclientjava

!ENTRY org.eclipse.equinox.p2.core 4 0 2023-11-22 12:21:11.478
!MESSAGE Provisioning exception
!STACK 1
org.eclipse.equinox.p2.core.ProvisionException: Unable to read repository at https://asdfasdfasfsdfdsfwerfwefwefwef.com/site.xml.
    at org.eclipse.equinox.internal.p2.updatesite.UpdateSite.loadActualSiteFile(UpdateSite.java:255)
    at org.eclipse.equinox.internal.p2.updatesite.UpdateSite.load(UpdateSite.java:155)
    at org.eclipse.equinox.internal.p2.updatesite.metadata.UpdateSiteMetadataRepositoryFactory.initializeRepository(UpdateSiteMetadataRepositoryFactory.java:108)
    at org.eclipse.equinox.internal.p2.updatesite.metadata.UpdateSiteMetadataRepositoryFactory.load(UpdateSiteMetadataRepositoryFactory.java:62)
    at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.factoryLoad(MetadataRepositoryManager.java:63)
    at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:791)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.eclipse.oomph.util.ReflectUtil.invokeMethod(ReflectUtil.java:119)
    at org.eclipse.oomph.p2.internal.core.CachingRepositoryManager.loadRepository(CachingRepositoryManager.java:448)
    at org.eclipse.oomph.p2.internal.core.CachingRepositoryManager.loadRepository(CachingRepositoryManager.java:231)
    at org.eclipse.oomph.p2.internal.core.CachingRepositoryManager$Metadata.loadRepository(CachingRepositoryManager.java:520)
    at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:110)
    at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:105)
    at org.eclipse.equinox.p2.ui.LoadMetadataRepositoryJob.doLoad(LoadMetadataRepositoryJob.java:126)
    at org.eclipse.equinox.p2.ui.LoadMetadataRepositoryJob.runModal(LoadMetadataRepositoryJob.java:110)
    at org.eclipse.equinox.internal.p2.ui.sdk.PreloadingRepositoryHandler$1.runModal(PreloadingRepositoryHandler.java:84)
    at org.eclipse.equinox.p2.operations.ProvisioningJob.run(ProvisioningJob.java:188)
    at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)
Caused by: java.util.concurrent.ExecutionException: java.net.ConnectException
    at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396)
    at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2096)
    at org.eclipse.ecf.provider.filetransfer.httpclientjava.HttpClientRetrieveFileTransfer.performConnect(HttpClientRetrieveFileTransfer.java:1012)
    at org.eclipse.ecf.provider.filetransfer.httpclientjava.HttpClientRetrieveFileTransfer$1.performFileTransfer(HttpClientRetrieveFileTransfer.java:999)
    at org.eclipse.ecf.filetransfer.FileTransferJob.run(FileTransferJob.java:76)
merks commented 9 months ago

Both the platform sdk and the installer have been tested with this system property and confirmed to work in the proxy environment where the new provider was failing. So I think we are good.