Arm64 variant?

[jeremyd2019]

@mstorsjo has mentioned before that he uses wine on arm64 to help debug arm64 windows binaries (I think this was when we were having unwinding issues, but that's not important now).

There has been some mention of betas of arm64 runners (probably just linux, but the hope is always there that they'll have a windows one), but for right now I think we just have an arm64 macos for public runners. Unfortunately, I'm not a mac user, a docker user, or a wine developer/builder, so I don't really know where to start. It seems like the published mac binaries of wine don't have arm64 windows binary support enabled.

[jeremyd2019]

It looks like macos docker does not have a lightweight "container" like on linux or windows, so maybe this is not practical until there are other arm64 runners publicly available.

[rimrul]

probably just linux, but the hope is always there that they'll have a windows one

https://github.blog/2024-06-03-arm64-on-github-actions-powering-faster-more-efficient-build-systems/

GitHub is ecstatic to unveil ArmⓇ-based Linux and Windows runners for GitHub Actions are now in Public Beta. […] These runners are available to our customers on our GitHub Team and Enterprise Cloud plans. We expect to begin offering Arm runners for open source projects by the end of the year.

Official arm64 Windows runners should become a thing in the next 6 months. If you need an interim solution, what Git for Windows does is automatically create azure vms. Azure does have an offering for free credits for open source projects that Msys2 should qualify for. https://opensource.microsoft.com/azure-credits/#credits-overview

[jeremyd2019]

That's great news! An arm64 linux docker image might still be a useful thing to have, once runners become available to produce one in gha.

[jeremyd2019]

I figured I'd give a try to building wine on arm64 mac runner. It builds successfully, but fails to run even so far to output usage or anything. It just says Killed: 9.

https://github.com/jeremyd2019/macos-wine-test/actions/runs/9456100637/workflow

(it doesn't matter if I include --enable-archs configure option or not: https://github.com/jeremyd2019/macos-wine-test/actions/runs/9456668761/job/26049019420)

[mstorsjo]

I figured I'd give a try to building wine on arm64 mac runner. It builds successfully, but fails to run even so far to output usage or anything. It just says Killed: 9.

Native arm64 wine on macOS is problematic. (I did experiment with this early in the macOS/arm64 developer preview period actually.) Even if you can get things running, the main blocker is that the Apple arm HW uses 16 KB pages, while most windows binaries assume 4 KB pages. For binaries that you link yourself, it's possible to overcome this by passing arguments for setting a different page alignment, but that doesn't help for running any other binary.

(IIRC this issue also extends to when running native Linux on Apple HW, i.e. Asahi Linux.)

Arm64 wine on linux (on HW with 4 KB pages) works quite well these days though. So once there are arm64 linux github runners, we should be all set.

FWIW, there's also a premade docker image linaro/wine-arm64, set up with qemu+wine, that lets you run windows arm64 binaries on x86 linux. But I'm not sure if that's usable/suitable for msys2-docker.

[jeremyd2019]

Thanks for chiming in here.

Another issue occurred to me: msys2 itself is still x86_64, so we'd need something that could run both x86_64 and arm64 windows binaries. I kind of glossed over that because I knew macos on arm had emulation like windows on arm does, so it could run both, but if we use linux we'd have to hook up qemu user emulation for one or the other architecture anyway, and figure out how to configure wine in such a way that both types of windows executables could be run.

[jeremyd2019]

I forked wine-mirror/wine on github, grabbed the jhol msys2-hacks-17 branch (verified still the same commit ref), and rebased it onto wine-9.10. A couple of the commits had conflicts, but they seemed to have a different version of the fix applied upstream, so I skipped those commits in the rebase. I also applied the revert done by the wine-arm64 docker image from linaro to that branch. https://github.com/jeremyd2019/wine/tree/msys2-wine-9.10. It seems to work about the same as the original msys2-hacks-17 on x86_64 for msys2 (I'm hitting some issues in both...). I am still trying to build it for arm64 (using qemu-user-static on x86_64, so it's going slowly) to see how it works for that. Using debian's wine package on arm64 results in 100% cpu and inability to run anything, so clearly needs some patch to work (and the wine-arm64 docker image does seem to work)

I realize now that I have wsl on the windows dev kit machine, I could probably build wine for arm64 much faster there... Oh well

[jeremyd2019]

build finally finished, and it can run (I just download mingw-w64-clang-aarch64-zstd and use its zstd.exe to decompress itself as a test). So now I just need to see if there's some way to hook arm64 and x86_64 wines together so I can run msys2 and clangarm64 the way I can on windows on arm.

[jeremyd2019]

It doesn't look as easy as I hoped (was thinking: write a program that can tell what arch a PE file is for (easy), make it run the right wine binary for the file, and hook that up to binfmt-misc). But looking at the code, it looks like wine already identifies PE files, and errors out with STATUS_INVALID_IMAGE_FORMAT for arches it can't run, rather than trying to fork/exec like it would for a non-PE file.

I'm thinking @1480c1's idea of using a linux pacman to install clangarm64 packages might be easier...

[jeremyd2019]

it looks like x86_64 -> arm64 would allow using "normal" fork/exec by:

diff --git a/dlls/ntdll/unix/process.c b/dlls/ntdll/unix/process.c
index 5debe98..c173657 100644
--- a/dlls/ntdll/unix/process.c
+++ b/dlls/ntdll/unix/process.c
@@ -795,7 +795,9 @@ NTSTATUS WINAPI NtCreateUserProcess( HANDLE *process_handle_ptr, HANDLE *thread_

     if ((status = get_pe_file_info( &attr, &file_handle, &pe_info )))
     {
-        if (status == STATUS_INVALID_IMAGE_NOT_MZ && !fork_and_exec( &attr, unixdir, params ))
+        if ((status == STATUS_INVALID_IMAGE_NOT_MZ ||
+             status == STATUS_INVALID_IMAGE_FORMAT ||
+             status == STATUS_INVALID_IMAGE_WIN_64) && !fork_and_exec( &attr, unixdir, params ))
         {
             memset( info, 0, sizeof(*info) );
             free( redir.Buffer );

But arm64 -> x86_64 is hitting some "experimental" code for arm64ec that lets it load x86_64 images, and even try to start an x86_64 process, but fails with c00000bb loading ntdll. I don't know what they hope to accomplish starting an x86_64 process on arm64, unless they have an x86_64 emulator hidden somewhere I don't know about. (which would be cool - that's really what I want is a single wine that can run both arm64 and x86_64).

[mstorsjo]

But arm64 -> x86_64 is hitting some "experimental" code for arm64ec that lets it load x86_64 images, and even try to start an x86_64 process, but fails with c00000bb loading ntdll. I don't know what they hope to accomplish starting an x86_64 process on arm64, unless they have an x86_64 emulator hidden somewhere I don't know about. (which would be cool - that's really what I want is a single wine that can run both arm64 and x86_64).

That's indeed what they're working on. They don't have an emulator hidden in wine itself, but they have an interface for plugging in an existing emulator - e.g. FEX. AFAIK this isn't quite done for x86_64/arm64ec yet, but it's progressing a lot. For i386, it should be working though, if I understand correctly. There's been a lot of work done upstream in LLVM, by @cjacek, to implement all the arm64ec bits in the compiler, in order to be able to make wine do this. See e.g. https://gitlab.winehq.org/wine/wine/-/releases/wine-9.0#arm64 for some notes on this, and how to plug in an emulator.

[jeremyd2019]

Nice! So yet again, probably best to wait for something else to be ready... but probably keep hacking at things anyway 😛

[jeremyd2019]

OK, that sorta works, but doesn't seem to block when I run a console arm64 exe from x86_64 msys2 (on x86_64 linux). Haven't tried running x86_64 from an arm64 process yet.

https://github.com/jeremyd2019/wine/commit/9666c8290f1fd0c7a88f377a7005bd5a00ecda04 https://gist.github.com/jeremyd2019/f1a3aff7a46e953dec38035e87f265c0

[jeremyd2019]

Maybe I could use i686 msys2, if i686 emulation works on arm64, but I don't see any doc on where to get the FEX binary that I'm supposed to plug into the wine wow64 registry key

[jeremyd2019]

I forked wine-mirror/wine on github, grabbed the jhol msys2-hacks-17 branch (verified still the same commit ref), and rebased it onto wine-9.10. A couple of the commits had conflicts, but they seemed to have a different version of the fix applied upstream, so I skipped those commits in the rebase.

I just noticed syncing my fork that wine 9.11 was just tagged. rebased my 9.10 branch onto 9.11, another fix from that branch was applied upstream. Looks like more building is in my future :wink:

[jeremyd2019]

Is there any interest of making a wine fork in ie msys2-contrib, and rebasing msys2-hacks-17 onto new wine releases there?

For fun, range-diffs of my branches:

git range-diff wine-9.0-rc1..msys2-hacks-17 wine-9.10..msys2-wine-9.10

```diff 1: d7cb27fc91e = 1: 6d31ddfc750 ntdll/test: Add tests for FILE_RENAME_POSIX_SEMANTICS. 2: d8379c81b07 = 2: 998e1857c8a ntdll/test: Add tests for FILE_LINK_POSIX_SEMANTICS. 3: 974aaff8646 = 3: c12827d2566 ntdll: Factor out get_inode_open_sharing. 4: a0fad3c3b02 = 4: 1be3d31bf72 ntdll: Add support for FILE_{RENAME,LINK}_POSIX_SEMANTICS. 5: 1d21c9ca9ae = 5: 7f2fff6b438 ntdll/server: Make robust to spurious short writes 6: 12a0ff70eb0 = 6: 33a840a4ab8 RR-DEBUG: ntdll: Print unix pid and tid of each thread, commandline, wait on exit and Qt environment. 7: 5dd8d878c81 = 7: 40440864aa0 RR-DEBUG: ntdll: Enable recording with rr of cygwin executables. 8: 4b12c49c705 = 8: e40e461a41b RR-DEBUG: winemenubuilder: Exit early if run under rr-debugger. 9: d6e41453440 = 9: a03708529fa TEMP: uxtheme: Print fixme just once. 10: 2cf3669f2c6 = 10: 3136f6b22f6 TEMP: win32u: Print fixme just once. 11: 91f00fb5d3a ! 11: 2a6e074e780 TEMP: ntdll: Print fixme just once. @@ Commit message ## dlls/ntdll/unix/process.c ## @@ dlls/ntdll/unix/process.c: NTSTATUS WINAPI NtQueryInformationProcess( HANDLE handle, PROCESSINFOCLASS class - else ret = STATUS_INVALID_PARAMETER; - break; + break; + } + case ProcessHandleInformation: + { 12: d20a76328dd = 12: 604064b7136 ntdll/tests: Test opening a named pipe with \Device\NamedPipe\ as RootDirectory. 13: 562a066606b = 13: 24921346efe server: Inline open_named_object() call in open_file_object handler. 14: ec325118eb7 = 14: 4168007ff1a server: Pass the object subpath to open_file instead of rejecting it. 15: 1880d09aa76 = 15: d0c76d87ef3 server: Add stub lookup_name implementation for named_pipe_device_file_ops. 16: 53b1b75601c = 16: 48e36b67fa1 server: Allow creating named pipes using \Device\NamedPipe\ as RootDirectory. 17: ae7415fa9dc = 17: 15bae410b50 ntdll: Implement reading multiple entries in NtQueryDirectoryObject. 18: da6e0259209 = 18: bd9a5732742 server: Allow access with PROCESS_QUERY_INFORMATION. 19: e4831f0809e < -: ----------- ntdll: Ignore pointers outside of the current stack in exception handlers. 20: f9dce259053 = 19: d1e5a96e872 ws2_32: Disable check for local socket handles. 21: b9dc706f026 = 20: 9ca3526e234 server: Reset reported_events if socket gets reused. 22: a867b0f8cfd = 21: e87acdc88b5 ntdll: Add read and write to commited guard pages. 23: aed38b3fea2 < -: ----------- kernelbase: Let GetFileInformationByHandle succeed on unix redirected handle. -: ----------- > 22: 36d5394c91e Revert "loader: Build the preloader as PIE on 64-bit." ```

git range-diff wine-9.10..msys2-wine-9.10 wine-9.11..msys2-wine-9.11

```diff 1: 6d31ddfc750 = 1: 3967751c4c7 ntdll/test: Add tests for FILE_RENAME_POSIX_SEMANTICS. 2: 998e1857c8a = 2: db98f55615c ntdll/test: Add tests for FILE_LINK_POSIX_SEMANTICS. 3: c12827d2566 = 3: 1f0a1ce847a ntdll: Factor out get_inode_open_sharing. 4: 1be3d31bf72 = 4: 12abe93525d ntdll: Add support for FILE_{RENAME,LINK}_POSIX_SEMANTICS. 5: 7f2fff6b438 = 5: 03b319d7749 ntdll/server: Make robust to spurious short writes 6: 33a840a4ab8 = 6: 4cad10a7d70 RR-DEBUG: ntdll: Print unix pid and tid of each thread, commandline, wait on exit and Qt environment. 7: 40440864aa0 < -: ----------- RR-DEBUG: ntdll: Enable recording with rr of cygwin executables. -: ----------- > 7: 4ee8e1234ff RR-DEBUG: ntdll: Enable recording with rr of cygwin executables. 8: e40e461a41b = 8: e9cc28571e5 RR-DEBUG: winemenubuilder: Exit early if run under rr-debugger. 9: a03708529fa = 9: 7339273b15d TEMP: uxtheme: Print fixme just once. 10: 3136f6b22f6 = 10: bc00110853d TEMP: win32u: Print fixme just once. 11: 2a6e074e780 = 11: f9149785692 TEMP: ntdll: Print fixme just once. 12: 604064b7136 = 12: 0e0fa49867d ntdll/tests: Test opening a named pipe with \Device\NamedPipe\ as RootDirectory. 13: 24921346efe = 13: d5fd617c8d8 server: Inline open_named_object() call in open_file_object handler. 14: 4168007ff1a = 14: c8a43b11059 server: Pass the object subpath to open_file instead of rejecting it. 15: d0c76d87ef3 = 15: 395bee44f7c server: Add stub lookup_name implementation for named_pipe_device_file_ops. 16: 48e36b67fa1 = 16: 02632f036dd server: Allow creating named pipes using \Device\NamedPipe\ as RootDirectory. 17: 15bae410b50 < -: ----------- ntdll: Implement reading multiple entries in NtQueryDirectoryObject. 18: bd9a5732742 = 17: 158baa8aa7e server: Allow access with PROCESS_QUERY_INFORMATION. 19: d1e5a96e872 = 18: 8c7be8a8500 ws2_32: Disable check for local socket handles. 20: 9ca3526e234 = 19: b30590d667f server: Reset reported_events if socket gets reused. 21: e87acdc88b5 = 20: 2ade156021d ntdll: Add read and write to commited guard pages. 22: 36d5394c91e = 21: 3610019aade Revert "loader: Build the preloader as PIE on 64-bit." ```

[cjacek]

As @mstorsjo said, we're working on supporting such emulation, but it's not complete yet. ARM64EC port, which is a requirement for 64-bit support, is getting there, but emulator integration is a largely missing.

@jeremyd2019 wrappers like that may work for simple cases, but ultimately if you want to share Wine prefix between ARM64 and x64 processes, Wine needs to be aware of it. For example, .wine/drive_c/windows/system32 needs to be understood by both process types (which needs ARM64X).

For experimenting with 32-bit, I think upstream Wine still needs a little bit of patching to work with FEX. Probably the easiest way to experiment with it is using https://github.com/AndreRH/hangover (although most of its Wine patches should not be needed).

[jeremyd2019]

Technically, I don't think I was sharing the wine prefix in my testing, because the wine-arm64 wrapper I borrowed from the linaro repo sets WINEPREFIX: https://gitlab.com/Linaro/windowsonarm/woa-linux/-/blob/master/containers/unified.Dockerfile#L72 Of course, it only sets WINEPREFIX if it is not already set, and I didn't explicitly set it (I was using the default ~/.wine for the x86_64 while testing).

It looks like that hangover repo has instructions on how to build FEX for wine. So I can try to build FEX dll and plug it into wine-9.11, and see what breaks. Though one of the limitations I saw mentioned is that FEX doesn't work with console applications... so that's probably a pretty big limitation for me

[jeremyd2019]

It appears that i686 msys2 doesn't work properly, with "experimental wow64 mode" on x86_64 -> i386. It seems to trigger some sort of exception situation that then hangs on fork. Just calling pacman.exe directly works well enough to update the base distribution, if I disable sig checking, until it tries to fork for hooks or install scripts. I can kill the hung pacman.exe child and it gives a dofork error and moves on.

[mstorsjo]

It appears that i686 msys2 doesn't work properly, with "experimental wow64 mode" on x86_64 -> i386. It seems to trigger some sort of exception situation that then hangs on fork. Just calling pacman.exe directly works well enough to update the base distribution, if I disable sig checking, until it tries to fork for hooks or install scripts. I can kill the hung pacman.exe child and it gives a dofork error and moves on.

I guess cygwin/msys2 is one of the cases that exercise a lot of tricky details/corner cases, so it's probably not going to be the easiest to get working, either in experimental wow64, or emulated i386 (or x86_64 for that matter) on arm64. Not sure if this is a known issue upstream though, but if it's possible to distill into a minimal-ish testcase for them to reproduce on, it sounds to me that it'd be worth reporting.

[jeremyd2019]

Haha, yeah, minimal-ish testcase on one hand might just be a cygwin program that calls fork... but I bet you meant minimal in terms of Windows APIs not Cygwin... Probably next I need to get a 32-bit wine build and see if i686 msys2 even works there (with the patches/hacks that are necessary for x86_64 msys2), to see if it's the wow64 mode or just another cygwin issue that just didn't get addressed because it was i686.

[jeremyd2019]

OK, "native" i386 wine from my wine fork msys2-wine-9.11 branch can run msys2 i686 without this issue, so it's some wow64 situation I guess.