Open chirontt opened 2 years ago
Did you update the features to include the new fragments?
@merks thanks for the tip. Your picture is worth more than a thousand words to me. Now I can build the Eclipse SDKs for WoA and they work!
Next is where/what is the repo to build the Eclipse IDE (which should include other tools like JGit/EGit, m2e, etc.) for WoA?
The repository used to build the IDE are composed from many projects. There's information about that here:
https://ci.eclipse.org/simrel/
This file has information about all the projects involved:
https://git.eclipse.org/c/simrel/org.eclipse.simrel.build.git/tree/simrel.aggran
Quite of a few of the project has Oomph setups that you can use to setup an an IDE; many of them you'd have to do manually.
I don't think you'll need to change many other projects. The very vast majority of them wouldn't have Platform specific artifacts, and I don't have the impression fixing those is an immediate requirement for you. As @merks pointed, you might want to make the SimRel build capable of including the win32/aarch64 artifacts; adding a relevant line at https://git.eclipse.org/c/simrel/org.eclipse.simrel.build.git/tree/simrel.aggr#n74 . But SimRel references p2 repositories, so you'll need to tweak it to reference a p2 repository that contains the win32/aarch64 SDK, which might not be easy. Then, from SimRel, the "Eclipse IDE" is build with https://git.eclipse.org/c/epp/org.eclipse.epp.packages.git/ ; there are some pom.xml and .product files in here you'll also need to modify for this new platform to be considered.
But this is a multi-step process. My advice would be that -if you're willing to make your work sustainable and to contribute it to Eclipse Project-, now that you have a build working, you find a way to get a win32/aarch64 machine contributed to the build farm and contribute your changes to the Eclipse Platform/SDK project to get this Platform supported. Going to "official" SDK and p2 repo will already be a major milestones and will make it easier to cascade later on to SimRel/EPP...
As to the problem with JNA bundle in WoA:
PPS. Especially and crucially for WoA,
eclipse.exe
can only start up after I add the-Dorg.osgi.framework.os.name=win32,win
entry to theeclipse.ini
file, to help it resolve the criticalcom.sun.jna
bundle at runtime.
I've raised an issue with the JNA project: java-native-access/jna#1465
@mickaelistria My Arm64 box is an entry-level, slow box, suitable for some light, manual testing only. I had started building this repo with this Arm64 box, and it was a big exercise in patience. I eventually gave it up and switched to building on a Linux (x64) box instead. It takes around 1 hour to build this repo on the Linux box without running the tests (i.e. with -DskipTests=true
). The Arm64 box becomes a test machine for some manual testing of the resulting SDKs. So I don't think it's a good addition to your build farm.
The good news is that you don't need an Arm64 box to build these SDKs for WoA. I can build and produce all SDKs for all os/arch (WoA included) with my Linux (x64) box, so your existing build farm can build the WoA distribution too, once my work on WoA are accepted in to the official repos. The recompiled binaries for Arm64 would be checked in to the repos like other existing binaries currently for other os/arch, so that the build wouldn't depend on any platform-specific tools to run.
From the look of it, building the Eclipse IDE for WoA is more than I can chew at the moment, so for now I will concentrate on submitting PRs for this WoA work to various repos (9 total, including this repo), to get the official Platform/SDK build started for WoA!
@chirontt OK, thanks for the details. So if you have things that you could already contribute, please do so and we'll see what we can merge easily. But at some point, to guarantee good enough quality, we'll need to have the SDK tests running automatically on a win32/aarch64 machine. Without it, then the project cannot prevent from potential regressions easily, and we basically cannot provide good enough support ot make it reliable; which IMO kind of question the value of hosting this specific code in the Eclipse project instead of eg in some fork of yours.
Can this aggregator build be run for Windows on Arm64 only, i.e. for win32/win32/aarch64
environment triplet only? How should it be configured?
If so, I would try to build just for this WoA platform on my Arm64 box, including running the tests, to see how it would fare in producing the SDKs for WoA. I can't give any promise at the moment, though.
BTW, I've started sending out the PRs for this WoA work.
Some of the CI issues such as the ones about not found bundles in https://ci.eclipse.org/equinox/job/equinox/job/PR-121/1/console are caused by the missing win32/win32/aarch64 declaration in the parent pom: https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/master/eclipse-platform-parent/pom.xml#L254 . Please try adding the environment there and if build is fine, submitting a PR for that. If you want to try a local build skipping other environments, you can comment them out.
We can have the bundles but can't include them in features/products until we have the builder to ensure that they are rebuild on source changes.
Sorry for the confusion in my partial submission of PRs. I ran out of time yesterday, and only managed to submit the PRs for eclipse-equinox
repos.
Now I've done all other PRs for eclipse-platform
repos, to complete this WoA submission to mainline.
My guess is that all these PRs need to go in together for a successful build, as they are inter-related, but you may have better idea about them.
I'll repeat it: We can not include native win32 aarch fragments in features unless we have a builder in our JIPPs allowing to rebuild them on e.g. source change in SWT. Until this happens we can only have sources that can't end up in deliverables.
@daliasheasha You was interested in Eclipse on Windows AArch64. Maybe you can help with some builder?
@akurtakov I'll resubmit the PRs if you can show me what can be submitted now, and what should be submitted later when the build farm would have a builder available to (re)build Windows on Arm64 artifacts. This requires in-depth knowledge of the overall build requirements of all the repos, knowledge that I don't have.
Although I've withdrawn all the PRs, for the records, here was the list of them again that contains all of the WoA work in my forks:
https://github.com/eclipse-equinox/equinox/pull/121 https://github.com/eclipse-equinox/equinox.binaries/pull/2 https://github.com/eclipse-equinox/p2/pull/147 https://github.com/eclipse-platform/eclipse.platform/pull/189 https://github.com/eclipse-platform/eclipse.platform.releng/pull/117 https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/pull/595 https://github.com/eclipse-platform/eclipse.platform.swt/pull/413 https://github.com/eclipse-platform/eclipse.platform.swt.binaries/pull/16 https://github.com/eclipse-platform/eclipse.platform.ui/pull/365
The branches in my forks for them are still available, to be resubmitted to mainline if needed. From my point of view, submitting them all together would lessen the chance of build breakage, but you may have better idea.
I know that all changes in those PRs would produce a successful Eclipse SDK for Windows on Arm64 platform, but separating them to do partial submits to mainline is a tough job for me, not to mention (God forbid!) build breakage. I'm willing to create new branches in my forks to submit the PRs again, if you can give me a precise list of what can be accepted, or wouldn't be needed (natives?), out of the above (withdrawn) PRs.
FYI, my fork of this umbrella repo, for Windows-on-Arm64 support, is now available:
https://github.com/chirontt/eclipse.platform.releng.aggregator
The fork includes links to other forked submodules, to make up my whole Windows-on-Arm64 work.
To build it is like normal (it can be built on any OS, be it Linux, Windows, or MacOS (haven't actually tried this myself)):
> git clone --recurse-submodules https://github.com/chirontt/eclipse.platform.releng.aggregator
> cd eclipse.platform.releng.aggregator
> mvn verify -DskipTests=true
The eventual build artifacts for all Eclipse SDKs of all os/arch will be produced in the eclipse.platform.releng.tychoeclipsebuilder/eclipse.platform.repository/target/products/
directory.
And the specific artifact of my interest would be org.eclipse.sdk.ide-win32.win32.aarch64.zip
, which should be created in that directory. Note that, in order to run this artifact successfully on a WoA machine, you must apply the workaround I mentioned above, for Eclipse to successfully resolve the com.sun.jna
bundle at startup.
I will try to keep this fork tracking the upstream repo closely, to keep it from becoming outdated, perhaps by merging it with upstream repos every weekend.
@waynebeaton Maybe you can try foundation MS connections for having win/aarch64 machine on the build farm?
There's no need to have win/aarch64 machine to build bundles, see https://github.com/eclipse-platform/eclipse.platform.swt/pull/413#issuecomment-1270658605
Without such machine, we won't be able to run tests against built binaries, but this is only a minor issue: it's unlikely to have any new bugs compared to x64 bundle.
I expect we'll want to be able to test that this stuff actually works. Also, if we want SimRel/EPP to deliver these things, will need a JustJ JRE for which we need to run jlink natively. If no one is willing to commit to providing hardware, what does that say about depth of the commitment?
I said it already - it's fine to have sources for anyone willing to work on it. But it's not fine to ship any user binaries without being able to at least smoketest it. We have went through such hoops with linux/[aarch64|ppc64le] - binaries produced were not even starting in real world until smoketesting infrastructure was present.
Hi folks - Martijn from Microsoft (and Eclipse Adoptium) here. I'm wondering if some of the Windows on ARM hosts we have at Eclipse Adoptium could be utilized for this project? I'll check with George Adams when he's back from leave and see what we can do.
@karianna so is there progress on your arm hosts being available for releng jipp?
An option is Project Volterra - https://blogs.windows.com/windowsdeveloper/2022/10/24/available-today-windows-dev-kit-2023-aka-project-volterra/
Not sure if we can do the Adoptium loan - CC @gdams
I'm closing this one as there is clearly no interest from anyone to step up and do some work. If/when one does please reopen or create new issue.
FYI: In the meantime, I'm putting out unofficial Eclipse SDK builds for Windows on Arm64 in my fork.
I guess my fork will exist until Eclipse starts officially supporting Windows on Arm64 in its IDE releases.
FYI: In the meantime, I'm putting out unofficial Eclipse SDK builds for Windows on Arm64 in my fork.
I guess my fork will exist until Eclipse starts officially supporting Windows on Arm64 in its IDE releases.
Thanks a lot, @chirontt ! That's the way forward as long as no one with resources cares enough to provide a single Windows Arm64 builder to the project to use.
@chirontt FWIW, if you want to make your work official and would be ready to commit in maintaining it as part of Eclipse Platform, I think you'd be very welcome to do so. The only thing I don't know is whether the Eclipse infra offers a Windows/Arm64 machine to take care of compiling the natives and testing. However, if you're interested, we may also consider just including your code directly in Platform, without building it if we can't. It can be a 1st steps towards an official project delivery.
The sticking point here is the compiling of the natives for WoA, where there is no official WoA machine (real or virtual) to do it. But @SyntevoAlex has suggested a way forward: cross-compiling, from an official x64
Windows host, to produce natives for the target arm64
of WoA. I've done this cross-compiling successfully for the last few months now, for the regular recompile of SWT natives for WoA.
I need to work with @SyntevoAlex to submit these cross-compile scripts first. Once the scripts are accepted and the WoA natives are officially created and committed to various repos, the rest of my WoA work can be submitted thereafter.
Once the Platform provides it, most likely EPP will want to provide it and then JustJ will want to provide JREs for it, and it needs to run JLink natively. I really think we need some hardware. I would also think we'd want to run tests with it natively as well. I wonder that Adoptium wouldn't have such hardware. I'll ask around again...
For the record - getting support in the whole stack is/should be the end goal. But if cross compiling allows getting support at Platform level now and there are people to do the work and test it- we should get that in Platform now and further work can/should be handled later. After all Linux AArch64 support has been in Platform for years before it made it to EPPs. If/when someone starts doing the work for cross compiling and needs tracking issue for the various things to change feel free to reopen this one or create a new issue for that purpose.
@chirontt please prepare a PR and I could have a look at it. Not soon, though. Maybe in a month.
@merks @mickaelistria How can I contribute a a Windows on ARM CI runner to the project? At least for a few months until you guys get a permanent one from ARM (they are donating hardware for projects trying to be ported).
@SyntevoAlex I got here actually because I need SmartGit for Windows on ARM 😉
@hmartinez82 great to hear you want help out, to contribute a runner you need to open a ticket on the helpdesk here is an example:
there you should note that you want your runner be added to the Platform Releng Jenkins:
the IT team will then ask you for additional details in my case it was
Additionally it would be good to mention how many concurrent build should be allowed on this node, you maybe also want to take a look here to get some idea what needs to be setup: https://www.jenkins.io/doc/book/managing/nodes/#installing-a-jenkins-agent-on-windows
At least for a few months until you guys get a permanent one from ARM (they are donating hardware for projects trying to be ported).
If you have further information please share them here, or you can even mention them in the help-desk so the IT team probably can take the necessary actions!
After unzipping the
org.eclipse.sdk.ide-win32.win32.aarch64.zip
file, I discover that, in itseclipse/plugins
folder, the following 6 bundles are missing:org.eclipse.core.filesystem.win32.aarch64_<version>.jar org.eclipse.core.net.win32.aarch64.source_<version>.jar org.eclipse.core.net.win32.aarch64_<version>.jar org.eclipse.core.resources.win32.aarch64_<version>.jar org.eclipse.equinox.security.win32.aarch64.source_<version>.jar org.eclipse.equinox.security.win32.aarch64_<version>.jar
First of all the source bundles are created by the build, so you don't have to take care about them.
And org.eclipse.core.filesystem.win32.aarch64
should also not be necessary to make Eclipse work on WoA. The native fragment for Windows x86_64 is only to improve the performance of filesystem access but there is also a default handler available, which is IIRC 5~10% slower.
If org.eclipse.core.resources.win32.aarch64
is not available you only loose auto-refresh with native polling, so unless you change files outside Eclipse you shouldn't notice it is missing.
I haven't checked org.eclipse.core.net.win32.aarch64
and org.eclipse.equinox.security.win32.aarch64
yet, but maybe this one can also be saved with only some conveniences lost.
On the long run we can probably replace all the native binaries with Java's new Foreign Function and Memory Access API and solve all of this in pure Java without the need to compile for each architecture. Therefore I'm not sure its worth the effort to work towards that direction in the meantime.
The only definitive essentials I'm aware of are SWT and the Equinox launcher, but for both you have already created PRs 👍🏽
Once the basic are available we should also set-up an I-build for win32.aarch64.
With all the recent changes tonight's I-build should produce a complete Eclipse-SDK product for Windows on ARM. @chirontt could you please test them and report back if they work fine?
@merks since all requirements on win32.aarch64 fragments are now strict again (in SWT it was optional for some time), could you test if the eclipse sdk contribution for SimRel still works?
- [ ] add support for Windows on Arm64 to
eclipse.platform
submodule
- [ ] eclipse-platform/eclipse.platform#1311
Explicit win32.aarch64
fragments for the org.eclipse.core.filesystem
and org.eclipse.core.resources
plugins are the only things that have not yet been implemented, but they are not strictly necessary.
If no o.e.core.filesystem.win32.X
fragment is available instead a Java NIO file API based file-system NativeHandler
is used. The custom native filesystem-access implementations used by Eclipse were created to be faster, but I don't know if it has been reevaluated recently and if this is still the case respectivly worth the effort.
It would be great if you could benchmark with your custom builds?
If no o.e.core.resources.win32.X
fragment is available only the installation of the Win32RefreshProvider
fails, which leads to the following error log and the auto-refresh feature being not available:
Both should not block one from using Eclipse on Windows on ARM. The auto-refresh feature would be nice to have, but to implement a native file-system access I think we should really have some numbers first if it is worth the effort. If a custom native file-system access is still significantly faster, in some time in the future it will probably be replaced by the Java-22 FFM API any ways and then will be automatically available on all architecturs for which JVMs are available.
I've downloaded the Eclipse SDK for WoA from the latest I-build and ran it on my WoA box; seems to work fine, but here's its screenshot in dark mode:
Note the lighted scrollbars, they require the fix in eclipse-platform/eclipse.platform.swt#1172 to be dark again.
Seems both the o.e.core.filesystem.win32.aarch64
and o.e.core.resources.win32.aarch64
fragments are not needed for the SDK to work.
How do I check the auto-refresh feature? like, checking the auto-sync of the IDE with changes in the file system?
If no
o.e.core.filesystem.win32.X
fragment is available instead a Java NIO file API based file-system NativeHandler is used. The custom native filesystem-access implementations used by Eclipse were created to be faster, but I don't know if it has been reevaluated recently and if this is still the case respectivly worth the effort. It would be great if you could benchmark with your custom builds?
How should I benchmark this filesystem feature? Import a sizable project into the IDE and time it? (I have a sizable RCP project here to test it if needed.) Also, I have a slow box, so it may clearly show the difference as compared to a faster box. But I agree that we should remove these dependencies in native code as much as possible, given reasonably fast hardware available nowadays.
How do I check the auto-refresh feature?
Go to Preferences > Workspace > Refresh Using native Hooks or polling
What will happen is that you open expand a project folder, then you go to the Windows Explorer and create a new file there. If Refresh/Polling is activated the file will be shown in Eclipse after some (hopefully short) time, otherwhise you need to press F5 on the folder.
I think that's a part worth porting to aarch64
as there is no NIO Impl and the Java abstraction is rather complicated to be used while the native windows allows to watch a whole folder including its subfolders.
You can find some discussions here:
How should I benchmark this filesystem feature?
There was recently a benchmark and the NIO is slightly ~faster~slower (a few seconds over a very large count of operations) because NIO uses Exceptions to signal non existing files while we use in the native integration a boolean flag:
You can find some discussions here including the benchmarks:
I think its up to you if you think its worth to migrate this for aarch64
or not, or wait unless there are real complaints, inside an "usual" IDE setup I suspect one would never notice as there are other factors that slow down things considerably more than such micro optimization.
The https://download.eclipse.org/eclipse/updates/4.32-I-builds/I20240508-1800 aggregates:
Thank you Ed for checking.
I've downloaded the Eclipse SDK for WoA from the latest I-build and ran it on my WoA box; seems to work fine, but here's its screenshot in dark mode: Note the lighted scrollbars, they require the fix in eclipse-platform/eclipse.platform.swt#1172 to be dark again.
That's great news. The SWT issue will hopefully be resolved soon.
If no
o.e.core.filesystem.win32.X
fragment is available instead a Java NIO file API based file-system NativeHandler is used. The custom native filesystem-access implementations used by Eclipse were created to be faster, but I don't know if it has been reevaluated recently and if this is still the case respectivly worth the effort. It would be great if you could benchmark with your custom builds?How should I benchmark this filesystem feature? Import a sizable project into the IDE and time it? (I have a sizable RCP project here to test it if needed.) Also, I have a slow box, so it may clearly show the difference as compared to a faster box. But I agree that we should remove these dependencies in native code as much as possible, given reasonably fast hardware available nowadays.
Yes that would one option. You could for example compare the runtime of an explicit refreseh of all projects in the workspace or of a full clean+rebuild both should include a lot of IO (altough I have to admit I have not yet checked for which operations exactly the native file-system handler is used). And yes using a slow computer will probably show any difference better.
The king-way would be to create a JMH benchmark case of the different org.eclipse.core.internal.filesystem.local.NativeHandler
implementations, but that is probably more work.
But lets discuss that further in https://github.com/eclipse-platform/eclipse.platform/pull/1311 (or its successors) lets keep this for high-level discussions.
I think all other questions have been answered by Christoph already.
With that I think support for WoA is ready to be announced on the cross-project mailing list so that more people can try it out and give feedback.
There was recently a benchmark and the NIO is slightly faster (a few seconds over a very large count of operations) because NIO uses Exceptions to signal non existing files while we use in the native integration a boolean flag:
You can find some discussions here including the benchmarks:
Per occasion, we re-run "our" real life application benchmark on Eclipse 4.30 platform and we still see native Eclipse code performing faster as Java 21 NIO on xfs.
JNI vs Java 21 NIO 50x runs | RH7 xfs, Xeon W-2145 | RH9 xfs, Xeon w5-2465X |
---|---|---|
JNI eclipse.filesystem.useNatives=true |
146.9 s (σ 3.9) | 90.3 s (σ 1.7) |
Java I/O eclipse.filesystem.useNatives=false |
153.9 s (σ 3.7) | 92.5 s (σ 1.5) |
NIO vs JNI | 104,8% | 102,4% |
I think its up to you if you think its worth to migrate this for
aarch64
or not, or wait unless there are real complaints, inside an "usual" IDE setup I suspect one would never notice as there are other factors that slow down things considerably more than such micro optimization.
Correct.
Just created a New&Noteworthy entry for this:
and an issue in EPP to track the delivery of all prepacked Eclipse packages for WoA:
With https://github.com/eclipse-platform/eclipse.platform/pull/1422 being submitted I think everything functional is done to support Windows on ARM. The only missing pieces to fully complete this are I-builds on WoA and a win32.aarch64 Eclipse package. The latter mainly depends on Temurin providing such JDKs: https://github.com/adoptium/adoptium-support/issues/616
The 2024-12 M2 EPP build that contains WoA packages is now available for testing. If you have access to WoA, please let me know if these new packages work. See announcement in https://www.eclipse.org/lists/epp-dev/msg06939.html
I did some testing today on my WoA laptop and the packages including the new Oomph installer for win32-aarch64 worked flawlessly.
So the only thing missing to complete this with perfection is setting up the I-builds for WoA, but I'm working on this.
Update: now that adding support for Windows on Arm64 is under way for various Eclipse repositories, here's the central TODO list to keep track of various tasks done and/or under way:
eclipse.platform.swt
submoduleequinox
submodulert.equinox.p2
submoduleeclipse.platform
submoduleeclipse.platform.ui
submoduleeclipse.platform.releng.aggregator
I'm trying to build the Eclipse SDKs for Windows 11 on Arm64 (WoA), using this repo. I hope this is the right repo to start with, which is the umbrella repo for all others.
The environment triplet I use for it is
win32/win32/aarch64
, following the existingwin32/win32/x86_64
triplet as a guide.After re-compiling many .exe and .dll native files for WoA in various repos (from existing C code mostly unchanged), and adding many new
win32.aarch64
folders (most of them are duplicates of the existing win32.x86_64 ones), I've managed to build successfully this aggregator repo, and got the following new zip files ineclipse.platform.releng.tychoeclipsebuilder/eclipse.platform.repository/target/products
(in addition to other supporting os/arch zip files there):After unzipping the
org.eclipse.sdk.ide-win32.win32.aarch64.zip
file, I discover that, in itseclipse/plugins
folder, the following 6 bundles are missing:When compared to the
org.eclipse.sdk.ide-win32.win32.x86_64.zip
distribution, those bundles (of relevantx86_64
arch) do exist there. So somehow they got missing from the final zip file foraarch64
during the build.The missing bundles are crucial for the Eclipse SDK to properly start up, but I have no idea how they got missed out from the build artifacts. I think I must have skipped over some crucial file(s) somewhere which produced this missing result.
Is there a file somewhere in the repo to indicate which bundles to be included (in the
plugins
folder of the zip file) for each SDK?PS. The missing bundles are actually produced in their own Maven modules during the build, in the
eclipse.platform
&equinox
submodules. Once I manually add 4 of them (the other 2 are source bundles) to theplugins
folder, and also add their missing entries to theconfiguration\org.eclipse.equinox.simpleconfigurator\bundles.info
file, starting it up witheclipse.exe
works nicely!PPS. Especially and crucially for WoA,
eclipse.exe
can only start up after I add the-Dorg.osgi.framework.os.name=win32,win
entry to theeclipse.ini
file, to help it resolve the criticalcom.sun.jna
bundle at runtime.