Not able to communicate through authenticated proxy

[SebasHein]

I had a discussion about oomph not able to communicate through proxy anymore with Ed Merks already - see here: https://www.eclipse.org/forums/index.php/m/1861818/#msg_1861818

He suggests to open an issue here and to reproduce the behaviour with the latest eclipse build.

I downloaded I-Build of Eclipse from here https://download.eclipse.org/eclipse/downloads/drops4/I20231105-1800/

My environment is behind a proxy that requests to authentificate with user and password. In the eclipse preferences I set the same settings as was working well with Eclipse 4-29 ( 2023-09). Active Provider: Manual HTTP and HTTPS: Host and Port, user and password according to what was working in former release. The SOCKS is activated (checkbox is checked) but host, port etc. is empty. No proxy bypasses except localhost and 127.0.0.1

I then tried to install new software with help → install new software The update sites I tried were

• https://download.eclipse.org/eclipse/updates/4.30
• https://download.eclipse.org/releases/2023-12

I immediately get Proxy authentication required exceptions org.eclipse.equinox.p2.core.ProvisionException: HTTP Proxy Authentication Required: https://download.eclipse.org/releases/2023-12/content.xml at org.eclipse.equinox.internal.p2.repository.CacheManager.createCache(CacheManager.java:250) at org.eclipse.equinox.internal.p2.metadata.repository.SimpleMetadataRepositoryFactory.getLocalFile(SimpleMetadataRepositoryFactory.java:69) at org.eclipse.equinox.internal.p2.metadata.repository.SimpleMetadataRepositoryFactory.load(SimpleMetadataRepositoryFactory.java:89) at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.factoryLoad(MetadataRepositoryManager.java:63) at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:791) at org.eclipse.equinox.internal.p2.repository.helpers.AbstractRepositoryManager.loadRepository(AbstractRepositoryManager.java:689) at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:110) at org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager.loadRepository(MetadataRepositoryManager.java:105) at org.eclipse.equinox.p2.ui.ProvisioningUI.loadMetadataRepository(ProvisioningUI.java:498) at org.eclipse.equinox.internal.p2.ui.model.MetadataRepositoryElement.getMetadataRepository(MetadataRepositoryElement.java:138) at org.eclipse.equinox.internal.p2.ui.model.MetadataRepositoryElement.getRepository(MetadataRepositoryElement.java:127) at org.eclipse.equinox.internal.p2.ui.model.MetadataRepositoryElement.getQueryable(MetadataRepositoryElement.java:120) at org.eclipse.equinox.internal.p2.ui.QueryProvider.getQueryDescriptor(QueryProvider.java:92) at org.eclipse.equinox.internal.p2.ui.model.QueriedElement.fetchChildren(QueriedElement.java:104) at org.eclipse.equinox.internal.p2.ui.model.MetadataRepositoryElement.fetchChildren(MetadataRepositoryElement.java:86) at org.eclipse.equinox.internal.p2.ui.model.RemoteQueriedElement.fetchDeferredChildren(RemoteQueriedElement.java:38) at org.eclipse.ui.progress.DeferredTreeContentManager$1.run(DeferredTreeContentManager.java:219) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63) Caused by: org.eclipse.ecf.filetransfer.BrowseFileTransferException: Proxy Authentication Required at org.eclipse.ecf.provider.filetransfer.httpclientjava.HttpClientFileSystemBrowser.runRequest(HttpClientFileSystemBrowser.java:292) at org.eclipse.ecf.provider.filetransfer.browse.AbstractFileSystemBrowser$DirectoryJob.run(AbstractFileSystemBrowser.java:71) ... 1 more

Please let me know what else I can do or information I can share that helps you track down the root cause of this issue. As far as I understood you switched to Java 11 version of HttpClient. Is there anything I can do on my side that would be helpful?

[merks]

Thank you for testing and confirming the problem is reproducible purely with the Eclipse SDK 4.30.

The issue likely related to, or caused by, the same underlying problem as this one:

https://github.com/eclipse-equinox/p2/issues/357

[l3-g5]

I did a few tests with different kinds of proxies (2023-12 M2):

• SOCKS with and without auth seems to work
• Https Proxy without authentication works
• Https Proxy with authentication (digest) not working
• Https Proxy with authentication (basic auth) not working

All kind of proxies are working with 2023-09, so it seems that we introduced a severe problem with ECF's new httpclientjava.

[jonahgraham]

What is the way out here? IIUC the default changed to be java http client (via ECF) and the alternative apache implementation (also via ECF) is still in place. Can we change the default back to apache for 2023-12?

M3 has already been published, so we need to resolve this ASAP to have a working RC1.

cc: @laeubi @akurtakov

[jonahgraham]

Note for those that are only following this issue by email, @l3-g5's full comment was (side rant about how GitHub email notifications are insufficient):

I did a few tests with different kinds of proxies (2023-12 M2):

• SOCKS with and without auth seems to work
• Https Proxy without authentication works
• Https Proxy with authentication (digest) not working
• Https Proxy with authentication (basic auth) not working

All kind of proxies are working with 2023-09, so it seems that we introduced a severe problem with ECF's new httpclientjava.

[sratz]

Can we change the default back to apache for 2023-12?

I would also suggest to go for this, otherwise this would be a highly critical issue:

Since the updater itself would be affected by this regression it would be impossible to get out of a broken setup by updating. And since the default update sites contain /releases/latest, people with basic-auth proxies will end up in such broken setups.

[laeubi]

@merks @akurtakov @jonahgraham I think the safest way forward would be to include both httpjava+http5

@scottslewis mentioned somewhere that one can set a system property to select the provider used but I can't found that comment right now.

So one both are there we only need a N&N entry to show how to switch between those, so people that are affected have a way out but we get qualified feedback if there are any problems.

[merks]

In my environment with both ECF implementations present, I tested that this option works:

-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclientjava

[jonahgraham]

In my environment with both ECF implementations present, I tested that this option works:

-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclientjava

Should we make that the default for Oomph and EPP product files and that would allow a soft launch of the backend change for the generally more advanced users that get platform or SDK directly?

[merks]

Yes I think so. We will have to hold off until rc1 because it probably breaks m3 disabling the only provider. And we should consider a future where we maybe need a touch point to remove it.

I will do more testing with the installer tomorrow.

[laeubi]

I don't think making this default makes much sense because then no one will use it and no one will test it "in the wild" adding it to the eclipse ini in case of problems seems easy enough and everyone without special authenticated proxy won't notice anyways.

[jonahgraham]

Our advanced users will test it first since Eclipse SDK and Eclipse Platform downloads will use it.

But we know that the new version doesn't work in existing cases. I don't think we can ship such a product.

Once we have fixed the known bugs in the Java http version we can revisit changing the default in EPP / Oomph.

[laeubi]

@jonahgraham of course its the decision of EPP / Oomph, I just would then ask to adjust the N&N accordingly with whatever you think fits to encourage people to try that out in regular EPP / Oomph setups so we get better feedback, because even if we can fix this there might be more to discover:

• https://github.com/eclipse-platform/www.eclipse.org-eclipse/pull/86

[merks]

@SebasHein

Thanks so much for your error reporting and testing efforts!

The latest I-Build contains what's potentially a fix for the problem you've reported:

https://download.eclipse.org/eclipse/downloads/

Currently that's this version:

https://download.eclipse.org/eclipse/downloads/drops4/I20231115-0150/

The Eclipse SDK will continue to use the new provider as it's default so you could test whether this latest version works better than before...

[SebasHein]

I downloaded it and run just two short tests:

1. Just start eclipse sdk, put in proxy host, -port and user credentials and tried to fetch an update site (with install new software and link to https://download.eclipse.org/eclipse/updates/4.30 ) In this test the proxy authentification failed message is still present.
2. I added -Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclientjava to eclipse.ini and repeated what I did in point 1. This time everything works and I was able to fetch the eclipse project update site.

Therefore it looks like what you did functions as expected at least for the proxy configuration that is present in the network I am logged in.

Thanks again for your fast response to my issue and the extra work you had with it - I highly appreciate it!

[scottslewis]

I downloaded it and run just two short tests:

1. Just start eclipse sdk, put in proxy host, -port and user credentials and tried to fetch an update site (with install new software and link to https://download.eclipse.org/eclipse/updates/4.30 )
   In this test the proxy authentification failed message is still present.

Question: Is the stack trace/message from this fail the same as when you first opened this issue? If you have the stack trace from this failed run (or can run it again), perhaps you could paste it in here so I can verify.

Thanks.

[SebasHein]

Sure I can.

To summarize: I compared eclipse-SDK-I20231105-1800-win32-x86_64 (this was the not working version) with eclipse-SDK-I20231115-0150-win32-x86_64 (working with -Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclientjava in eclipse.ini) The exceptions are exactly the same. Of course no exception is thrown if I use it. But a warning appears (4 times) : org.eclipse.core.runtime.Status[plugin=org.eclipse.ecf.provider.filetransfer;code=2;message=Plugin org.eclipse.ecf.provider.filetransfer.httpclientjava excluded from contributing browse factory;severity2;exception=null;children=[]]

I added a txt file containing the thrown exceptions for either case: exceptionsThrown.txt

[jonahgraham]

I don't understand why there is an exception in httpclientjava when it is excluded. I'm glad the install works, but it sounds like something else is going on.

I have added the required -D for the Eclipse EPP packages (in a way that it can be removed once httpclientjava supports this use case) in https://github.com/eclipse-packaging/packages/pull/82

[jonahgraham]

I just would then ask to adjust the N&N accordingly with whatever you think fits to encourage people to try that out in regular EPP / Oomph setups so we get better feedback, because even if we can fix this there might be more to discover:

I added it to the EPP N&N: https://github.com/eclipse-packaging/packages/blob/master/CHANGELOG.md#2023-12

[ZebrasTeam]

The order seems to be the solution. In the original eclipse.ini of build.id: 20231201-2043 there is '-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclientjava' in line 15 and '-Dorg.eclipse.ecf.provider.filetransfer.excludeContributors=org.eclipse.ecf.provider.filetransfer.httpclient4' in line 28.

When you move the httpclientjava-entry from line 15 to the last line everything works fine with the proxy.

The question is, why?