eclipse / ecf

ECF project repository
7 stars 14 forks source link

Update 3rd party dependencies to the latest Orbit content #95

Closed merks closed 4 months ago

merks commented 6 months ago

I'm really not sure the best path and strategy for moving forward to updating the ECF's 3rd party dependencies. Currently ECF's target platform relies on bundles from old Orbit versions, even in cases where there are newer bundles available in the latest Orbit builds, e.g., json versus org.json.

This target platform is current being used, at least by default:

https://github.com/eclipse/ecf/blob/28753b9b4da377be0b51d273428ba4193fec44c8/releng/org.eclipse.ecf.releng.target/ecf-2022-03.target#L40-L69

The following are things that could be pulled into a newer Orbit build

I've been prototyping creating new Orbit versions for the above and using them in ECF:

image

But it feels like a bit of an uphill battle and I'm not sure if the project is actually going to update to new content from Orbit. The zookeeper in particular looks like a big can of works.

@scottslewis

I would be good to understand the expectations around this topic. ECF does contribute a significant amount of content to SimRel and that is currently dragging in some really old content. I can certainly understand if you (or the project) have other priorities, in which case it doesn't make so much sense for me to invest significant time in providing things in Orbit that won't actually be consumed and redistributed by ECF.

scottslewis commented 6 months ago

Hi Ed.

I'm really not sure the best path and strategy for moving forward to updating the ECF's 3rd party dependencies. Currently ECF's target platform relies on bundles from old Orbit versions, even in cases where there are newer bundles available in the latest Orbit builds, e.g., json versus org.json.

* https://repo1.maven.org/maven2/org/json/json/20230227/

This target platform is current being used, at least by default:

https://github.com/eclipse/ecf/blob/28753b9b4da377be0b51d273428ba4193fec44c8/releng/org.eclipse.ecf.releng.target/ecf-2022-03.target#L40-L69

The following are things that could be pulled into a newer Orbit build

* https://repo1.maven.org/maven2/commons-httpclient/commons-httpclient/3.1 - This is ancient and is not updated beyond 2007.  It's a problem waiting to happen.   

* https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient-win/4.5.14/  - This should have been updated from 4.5.13 but that never happened.  

* https://repo1.maven.org/maven2/rome/rome/0.9 or https://repo1.maven.org/maven2/rome/rome/1.0 - This is for com.sun.syndication which also appears to have gone dormant in 2010.  This looks like a problem waiting to happen.

Hi @merks

I will take a look at the dependencies above and determine whether to remove and/or deprecate the depending test code and/or old filetransfer or other providers.

* https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.9.2 - This looks very much alive, but the latest breaks APIs.  Also, it seem so drag in everything but the kitchen sink.  I could spend days trying to determine which dependencies are optional or not

The zookeeper discovery provider was originally created by Wim Jongman aka @wimjongman . I've sent an email to Wim and ecf-dev mailing list asking that Wim let me/us know if something other than just deprecating this provider and removing the old third party dependencies should be done.

* https://repo1.maven.org/maven2/dnsjava/dnsjava/3.5.3 - This looks pretty lively.  The latest breaks APIs.  Added to Orbit:

The dnssd discovery provider was created by Markus Kuppe aka @lemmy. The same ecf-dev email was sent to Markus also and hopefully he will respond on ecf-dev, to me directly, or even here about what should be done with dnssd provider and the dnsjava dependency.

> But it feels like a bit of an uphill battle and I'm not sure if the project is actually going to update to new content from Orbit. The zookeeper in particular looks like a big can of works. > > @scottslewis > > I would be good to understand the expectations around this topic. ECF does contribute a significant amount of content to SimRel and that is currently dragging in some really old content. I can certainly understand if you (or the project) have other priorities, in which case it doesn't make so much sense for me to invest significant time in providing things in Orbit that won't actually be consumed and redistributed by ECF. @merks thanks for the heads up. Until we hear from @wimjongman and @lemmy let's not expend any effort on zookeeper provider dependencies (apache zookeeper and old version of log4j I believe) nor dnssd discovery provider (org.xbill dns dependency). I expect that we will need to update the httpclient5 provider's target dependencies (httpclient/httpcomponents), so that we can continue to build and distribute it for a while, even though platform is now not using it. There may be a few others that are consuming and may want to stay with it. So if you could focus on updating the Orbit dependencies for that provider specifically (httpclient5), I will deal with the older providers (httpclient4 and 45) and json.
lemmy commented 6 months ago

@scottslewis Hi, are you aware of any active users of DNSSD? If not, I don't see the value in porting it to a new version, although it should be fairly straightforward given that DNS is well standardized.

scottslewis commented 6 months ago

@lemmy I'm currently not aware of any active users of DNSSD...although as I've found out before that doesn't mean that there aren't any.

WRT DNS well-standardized: Are you aware of an OSS xbill lib replacement that it could be ported to? If there is one somewhere I would be willing to do the porting and some testing if you advise/assist. And is there a simple way to test on lan without a working DNS server?

lemmy commented 6 months ago

org.xbill.dns

https://github.com/dnsjava/dnsjava appears to be a renamed org.xbill.dns, which suggests its API should be largely the same.

Unfortunately, you won't be able to test it without a DNS server, but the zone files for the dnssd.ecf-project.org tests are at https://github.com/eclipse/ecf/tree/master/tests/bundles/org.eclipse.ecf.tests.provider.dnssd/zones/bind9. However, OSUOSL doesn't seem to serve these zones anymore (it is authoritative for ecf-project.org), leaving the tests defunct.

scottslewis commented 6 months ago

Hi @merks

First, I'm a little confused about how to create a new target...using only the 'new' Orbit way (I presume this would be preferred).

I've been prototyping creating new Orbit versions for the above and using them in ECF:

Do you have a prototype ECF .target file? If so, would it be possible to get a hold of that .target file and the changes you have in your screenshot above so I can fuss with the dnssd provider specifically?

I would like to eliminate these dependencies:

https://repo1.maven.org/maven2/commons-httpclient/commons-httpclient/3.1 - This is ancient and is not updated beyond 2007. It's a problem waiting to happen. -- This is used for starting a very simple httpserver in the ECF filetransfer testing code:. bundle: org.eclipse.ecf.tests.apache.httpclient.server. I will figure out how we can remove or replace this test code, and remove this dependency.

https://repo1.maven.org/maven2/rome/rome/0.9 or https://repo1.maven.org/maven2/rome/rome/1.0 - This is for com.sun.syndication which also appears to have gone dormant in 2010. This looks like a problem waiting to happen. -- My belief is that this is used by a should-be-deprecated provider (ECF .synd provider)...so I'll just remove that old provider feature and the com.sun.syndication dependency can be removed.

https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.9.2 - This looks very much alive, but the latest breaks APIs. Also, it seem so drag in everything but the kitchen sink. I could spend days trying to determine which dependencies are optional or not. - Unless Wim comes in with resources and ideas for replacing the zookeeper provider I want to deprecate it...and remove all the third party dependencies. that is uses. I believe that is apache hadoop zookeeper (3.3.3 version I believe) ...which itself depends upon log4j old version I think.

I would like to work on/keep these dependencies:

https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient-win/4.5.14/ - This should have been updated from 4.5.13 but that never happened. Added to Orbit:

https://repo1.maven.org/maven2/dnsjava/dnsjava/3.5.3 - This looks pretty lively. The latest breaks APIs. Added to Orbit:

@merks if provide the prototype work you've been doing (.target and changes to source/features/products etc from your screenshots) I'll work on the elimination of the dependencies as described above.

Thanks.

merks commented 6 months ago

@scottslewis

Thanks for the responsiveness!!

Unfortunately, I'm very busy today with this:

https://github.com/eclipse-packaging/packages/issues/111

But I hope to be completed with that tomorrow some time and then revisit this ECF issue. I will reduce the scope of my changes to include only the things I've added to Orbit, so without rome/zookeeper changes, and will create a PR for your review.

scottslewis commented 6 months ago

Hi @merks

Wanted to let you know what I did about the old third party deps. I removed zookeeper and a number of other bundles from ECF's build, and removed the associated third-party dependencies from the currently-used target file (ecf-2022-03.target in releng project).

You can see the deletions from ecf-2022-03.target here:

https://github.com/eclipse/ecf/pull/97/commits/6cfb392cd7d848e72634123414646e3bf33c0823

Also gone is the commons-httpclient 3.1

The ones remaining that I would like to fix are:

httpclient45 (version upgrade as you specify above) json (some other version in Orbit now?) org.xbill.dns -> dnsjava 3.5.8

These are the ones that you started fixing/prototyping.

Feel free to create a brand new target platform rather than merging this new one in. I'm completely comfortable with having our target platform use up to 2023-06 release.

If you do create a new target platform the only pom that has to be modified to use it is the top-level pom.xml...in the target-platform property (current value: 2022-03)

Thanks.

scottslewis commented 6 months ago

Hi @merks

I can go ahead and fix these three third party deps that you list above

json: https://repo1.maven.org/maven2/org/json/json/20230227/

httpclient45: https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient-win/4.5.14/

dnsjava: https://repo1.maven.org/maven2/dnsjava/dnsjava/3.5.3

But with the new Orbit, I'm not sure how to get these new dependencies into the ECF target platform. Are there any docs to look at for getting these three new third party libs into (or accessing existing) the ECF target platform?

Thanksinadvance.

merks commented 6 months ago

Sorry for the delay. So much to do for EPP.

Those libraries are now available in the milestone build.

General documentation

https://github.com/eclipse-orbit

in this update site

https://download.eclipse.org/tools/orbit/simrel/orbit-aggregation/milestone/latest

tabular content summary

https://download.eclipse.org/tools/orbit/simrel/orbit-aggregation/table.html

You can use them just like any other bundle you use.

the Repository Explorer view is very handy in advanced mode to show the iu ids in a repo. The generated update site docs also show this information.

scottslewis commented 5 months ago

org.xbill.dns

https://github.com/dnsjava/dnsjava appears to be a renamed org.xbill.dns, which suggests its API should be largely the same.

Unfortunately, you won't be able to test it without a DNS server, but the zone files for the dnssd.ecf-project.org tests are at https://github.com/eclipse/ecf/tree/master/tests/bundles/org.eclipse.ecf.tests.provider.dnssd/zones/bind9. However, OSUOSL doesn't seem to serve these zones anymore (it is authoritative for ecf-project.org), leaving the tests defunct.

Hi @lemmy . @merks and I have updated the dnssd provider and test code (thanks @merks).

To test: I could contact the folks at OSUOSL...if you happen to know a contact name (I know it's been years so there may not be any personnel overlap) could you pass it along? And if you have any more details on what it would take at OSUOSL to run the dnssd test code please let me know. Thanks @lemmy.

Also for both: By foundation current rules a progress review has to be done for ECF 3.15.0 (which I'm ready for release for the platform/filetransfer...and if dnssd ends up broken then I'll produce another release).

Here's the progress review issue: https://gitlab.eclipse.org/eclipsefdn/emo-team/emo/-/issues/743

lemmy commented 5 months ago

Hi @lemmy . @merks and I have updated the dnssd provider and test code (thanks @merks).

To test: I could contact the folks at OSUOSL...if you happen to know a contact name (I know it's been years so there may not be any personnel overlap) could you pass it along? And if you have any more details on what it would take at OSUOSL to run the dnssd test code please let me know. Thanks @lemmy.

Also for both: By foundation current rules a progress review has to be done for ECF 3.15.0 (which I'm ready for release for the platform/filetransfer...and if dnssd ends up broken then I'll produce another release).

Here's the progress review issue: https://gitlab.eclipse.org/eclipsefdn/emo-team/emo/-/issues/743

I checked my old emails and Lance Albertson used to work with us on our DNS infra in 2017. The dnssd test suite just requires a DNS server that serves the zones.

merks commented 5 months ago

FYI, for the future, you can just do a progress review independent of any specific release such that no release is held up by a review:

https://gitlab.eclipse.org/eclipsefdn/emo-team/emo/-/issues/1#note_2307015

merks commented 5 months ago

@scottslewis

The runway is growing very short with the Platform producing RC1 this week. We could contribute a snapshot build so that we can at least properly test this for the Platform, e.g.,

https://download.eclipse.org/rt/ecf/snapshot/site.p2/3.15.0.v20240502-2230/

But that only makes sense if we are sure there will definitely be a release repository before next week's RC2...

FYI, @akurtakov

scottslewis commented 5 months ago

@scottslewis

The runway is growing very short with the Platform producing RC1 this week. We could contribute a snapshot build so that we can at least properly test this for the Platform, e.g.,

https://download.eclipse.org/rt/ecf/snapshot/site.p2/3.15.0.v20240502-2230/

But that only makes sense if we are sure there will definitely be a release repository before next week's RC2...

FYI, @akurtakov

I cannot commit to having a release repository available early next week. I'm still waiting on the EF progress review for 3.15.0, and have personal/medical and work commitments over the next week.

scottslewis commented 5 months ago

Here is the progress review issue for reference: https://gitlab.eclipse.org/eclipsefdn/emo-team/emo/-/issues/743

wimjongman commented 4 months ago

Hey Scott, AFAIC you can drop the zookeeper implementation. We do not use it anymore, and I don't know anyone who does.