Closed merks closed 4 months ago
Hi Ed.
I'm really not sure the best path and strategy for moving forward to updating the ECF's 3rd party dependencies. Currently ECF's target platform relies on bundles from old Orbit versions, even in cases where there are newer bundles available in the latest Orbit builds, e.g., json versus org.json.
* https://repo1.maven.org/maven2/org/json/json/20230227/
This target platform is current being used, at least by default:
The following are things that could be pulled into a newer Orbit build
* https://repo1.maven.org/maven2/commons-httpclient/commons-httpclient/3.1 - This is ancient and is not updated beyond 2007. It's a problem waiting to happen. * https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient-win/4.5.14/ - This should have been updated from 4.5.13 but that never happened. * https://repo1.maven.org/maven2/rome/rome/0.9 or https://repo1.maven.org/maven2/rome/rome/1.0 - This is for com.sun.syndication which also appears to have gone dormant in 2010. This looks like a problem waiting to happen.
Hi @merks
I will take a look at the dependencies above and determine whether to remove and/or deprecate the depending test code and/or old filetransfer or other providers.
* https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.9.2 - This looks very much alive, but the latest breaks APIs. Also, it seem so drag in everything but the kitchen sink. I could spend days trying to determine which dependencies are optional or not
The zookeeper discovery provider was originally created by Wim Jongman aka @wimjongman . I've sent an email to Wim and ecf-dev mailing list asking that Wim let me/us know if something other than just deprecating this provider and removing the old third party dependencies should be done.
* https://repo1.maven.org/maven2/dnsjava/dnsjava/3.5.3 - This looks pretty lively. The latest breaks APIs. Added to Orbit:
The dnssd discovery provider was created by Markus Kuppe aka @lemmy. The same ecf-dev email was sent to Markus also and hopefully he will respond on ecf-dev, to me directly, or even here about what should be done with dnssd provider and the dnsjava dependency.
@scottslewis Hi, are you aware of any active users of DNSSD? If not, I don't see the value in porting it to a new version, although it should be fairly straightforward given that DNS is well standardized.
@lemmy I'm currently not aware of any active users of DNSSD...although as I've found out before that doesn't mean that there aren't any.
WRT DNS well-standardized: Are you aware of an OSS xbill lib replacement that it could be ported to? If there is one somewhere I would be willing to do the porting and some testing if you advise/assist. And is there a simple way to test on lan without a working DNS server?
org.xbill.dns
https://github.com/dnsjava/dnsjava appears to be a renamed org.xbill.dns, which suggests its API should be largely the same.
Unfortunately, you won't be able to test it without a DNS server, but the zone files for the dnssd.ecf-project.org
tests are at https://github.com/eclipse/ecf/tree/master/tests/bundles/org.eclipse.ecf.tests.provider.dnssd/zones/bind9. However, OSUOSL doesn't seem to serve these zones anymore (it is authoritative for ecf-project.org), leaving the tests defunct.
Hi @merks
First, I'm a little confused about how to create a new target...using only the 'new' Orbit way (I presume this would be preferred).
I've been prototyping creating new Orbit versions for the above and using them in ECF:
Do you have a prototype ECF .target file? If so, would it be possible to get a hold of that .target file and the changes you have in your screenshot above so I can fuss with the dnssd provider specifically?
I would like to eliminate these dependencies:
https://repo1.maven.org/maven2/commons-httpclient/commons-httpclient/3.1 - This is ancient and is not updated beyond 2007. It's a problem waiting to happen. -- This is used for starting a very simple httpserver in the ECF filetransfer testing code:. bundle: org.eclipse.ecf.tests.apache.httpclient.server. I will figure out how we can remove or replace this test code, and remove this dependency.
https://repo1.maven.org/maven2/rome/rome/0.9 or https://repo1.maven.org/maven2/rome/rome/1.0 - This is for com.sun.syndication which also appears to have gone dormant in 2010. This looks like a problem waiting to happen. -- My belief is that this is used by a should-be-deprecated provider (ECF .synd provider)...so I'll just remove that old provider feature and the com.sun.syndication dependency can be removed.
https://repo1.maven.org/maven2/org/apache/zookeeper/zookeeper/3.9.2 - This looks very much alive, but the latest breaks APIs. Also, it seem so drag in everything but the kitchen sink. I could spend days trying to determine which dependencies are optional or not. - Unless Wim comes in with resources and ideas for replacing the zookeeper provider I want to deprecate it...and remove all the third party dependencies. that is uses. I believe that is apache hadoop zookeeper (3.3.3 version I believe) ...which itself depends upon log4j old version I think.
I would like to work on/keep these dependencies:
https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient-win/4.5.14/ - This should have been updated from 4.5.13 but that never happened. Added to Orbit:
https://repo1.maven.org/maven2/dnsjava/dnsjava/3.5.3 - This looks pretty lively. The latest breaks APIs. Added to Orbit:
@merks if provide the prototype work you've been doing (.target and changes to source/features/products etc from your screenshots) I'll work on the elimination of the dependencies as described above.
Thanks.
@scottslewis
Thanks for the responsiveness!!
Unfortunately, I'm very busy today with this:
https://github.com/eclipse-packaging/packages/issues/111
But I hope to be completed with that tomorrow some time and then revisit this ECF issue. I will reduce the scope of my changes to include only the things I've added to Orbit, so without rome/zookeeper changes, and will create a PR for your review.
Hi @merks
Wanted to let you know what I did about the old third party deps. I removed zookeeper and a number of other bundles from ECF's build, and removed the associated third-party dependencies from the currently-used target file (ecf-2022-03.target in releng project).
You can see the deletions from ecf-2022-03.target here:
https://github.com/eclipse/ecf/pull/97/commits/6cfb392cd7d848e72634123414646e3bf33c0823
Also gone is the commons-httpclient 3.1
The ones remaining that I would like to fix are:
httpclient45 (version upgrade as you specify above) json (some other version in Orbit now?) org.xbill.dns -> dnsjava 3.5.8
These are the ones that you started fixing/prototyping.
Feel free to create a brand new target platform rather than merging this new one in. I'm completely comfortable with having our target platform use up to 2023-06 release.
If you do create a new target platform the only pom that has to be modified to use it is the top-level pom.xml...in the target-platform property (current value: 2022-03)
Thanks.
Hi @merks
I can go ahead and fix these three third party deps that you list above
json: https://repo1.maven.org/maven2/org/json/json/20230227/
httpclient45: https://repo1.maven.org/maven2/org/apache/httpcomponents/httpclient-win/4.5.14/
dnsjava: https://repo1.maven.org/maven2/dnsjava/dnsjava/3.5.3
But with the new Orbit, I'm not sure how to get these new dependencies into the ECF target platform. Are there any docs to look at for getting these three new third party libs into (or accessing existing) the ECF target platform?
Thanksinadvance.
Sorry for the delay. So much to do for EPP.
Those libraries are now available in the milestone build.
General documentation
https://github.com/eclipse-orbit
in this update site
https://download.eclipse.org/tools/orbit/simrel/orbit-aggregation/milestone/latest
tabular content summary
https://download.eclipse.org/tools/orbit/simrel/orbit-aggregation/table.html
You can use them just like any other bundle you use.
the Repository Explorer view is very handy in advanced mode to show the iu ids in a repo. The generated update site docs also show this information.
org.xbill.dns
https://github.com/dnsjava/dnsjava appears to be a renamed org.xbill.dns, which suggests its API should be largely the same.
Unfortunately, you won't be able to test it without a DNS server, but the zone files for the
dnssd.ecf-project.org
tests are at https://github.com/eclipse/ecf/tree/master/tests/bundles/org.eclipse.ecf.tests.provider.dnssd/zones/bind9. However, OSUOSL doesn't seem to serve these zones anymore (it is authoritative for ecf-project.org), leaving the tests defunct.
Hi @lemmy . @merks and I have updated the dnssd provider and test code (thanks @merks).
To test: I could contact the folks at OSUOSL...if you happen to know a contact name (I know it's been years so there may not be any personnel overlap) could you pass it along? And if you have any more details on what it would take at OSUOSL to run the dnssd test code please let me know. Thanks @lemmy.
Also for both: By foundation current rules a progress review has to be done for ECF 3.15.0 (which I'm ready for release for the platform/filetransfer...and if dnssd ends up broken then I'll produce another release).
Here's the progress review issue: https://gitlab.eclipse.org/eclipsefdn/emo-team/emo/-/issues/743
Hi @lemmy . @merks and I have updated the dnssd provider and test code (thanks @merks).
To test: I could contact the folks at OSUOSL...if you happen to know a contact name (I know it's been years so there may not be any personnel overlap) could you pass it along? And if you have any more details on what it would take at OSUOSL to run the dnssd test code please let me know. Thanks @lemmy.
Also for both: By foundation current rules a progress review has to be done for ECF 3.15.0 (which I'm ready for release for the platform/filetransfer...and if dnssd ends up broken then I'll produce another release).
Here's the progress review issue: https://gitlab.eclipse.org/eclipsefdn/emo-team/emo/-/issues/743
I checked my old emails and Lance Albertson used to work with us on our DNS infra in 2017. The dnssd test suite just requires a DNS server that serves the zones.
FYI, for the future, you can just do a progress review independent of any specific release such that no release is held up by a review:
https://gitlab.eclipse.org/eclipsefdn/emo-team/emo/-/issues/1#note_2307015
@scottslewis
The runway is growing very short with the Platform producing RC1 this week. We could contribute a snapshot build so that we can at least properly test this for the Platform, e.g.,
https://download.eclipse.org/rt/ecf/snapshot/site.p2/3.15.0.v20240502-2230/
But that only makes sense if we are sure there will definitely be a release repository before next week's RC2...
FYI, @akurtakov
@scottslewis
The runway is growing very short with the Platform producing RC1 this week. We could contribute a snapshot build so that we can at least properly test this for the Platform, e.g.,
https://download.eclipse.org/rt/ecf/snapshot/site.p2/3.15.0.v20240502-2230/
But that only makes sense if we are sure there will definitely be a release repository before next week's RC2...
FYI, @akurtakov
I cannot commit to having a release repository available early next week. I'm still waiting on the EF progress review for 3.15.0, and have personal/medical and work commitments over the next week.
Here is the progress review issue for reference: https://gitlab.eclipse.org/eclipsefdn/emo-team/emo/-/issues/743
Hey Scott, AFAIC you can drop the zookeeper implementation. We do not use it anymore, and I don't know anyone who does.
I'm really not sure the best path and strategy for moving forward to updating the ECF's 3rd party dependencies. Currently ECF's target platform relies on bundles from old Orbit versions, even in cases where there are newer bundles available in the latest Orbit builds, e.g., json versus org.json.
This target platform is current being used, at least by default:
https://github.com/eclipse/ecf/blob/28753b9b4da377be0b51d273428ba4193fec44c8/releng/org.eclipse.ecf.releng.target/ecf-2022-03.target#L40-L69
The following are things that could be pulled into a newer Orbit build
I've been prototyping creating new Orbit versions for the above and using them in ECF:
But it feels like a bit of an uphill battle and I'm not sure if the project is actually going to update to new content from Orbit. The zookeeper in particular looks like a big can of works.
@scottslewis
I would be good to understand the expectations around this topic. ECF does contribute a significant amount of content to SimRel and that is currently dragging in some really old content. I can certainly understand if you (or the project) have other priorities, in which case it doesn't make so much sense for me to invest significant time in providing things in Orbit that won't actually be consumed and redistributed by ECF.