Add Windows on AArch64 support

[Nzoth9]

Hello! First of all, thank you for making Dynamorio. I want to simply get code coverage data, but my environment is like this.

• M1 Windows 11 ARM
• DynamoRIO 9.0.19349 Release
• VisualCode 17 2022
• My stupid head

I'm trying to get code coverage on my Windows 11 with the following command (for use to IDA's lighthouse)

C:\Users\Eo\Desktop\DynamoRIO-Windows-9.0.19349\bin64>drrun.exe -t drcov -- notepad

C:\Users\Eo\Desktop\DynamoRIO-Windows-9.0.19349\bin64>

And then there is no response. The error appears even when I use pinTOOL on my Windows 11.

.\pin.exe -t .\CodeCoverage.dll -- notepad.exe
E: Failed to allocate Injector, Error = INJECTOR_ERR
E: Pin is exiting due to fatal error

Is there any other way to do code coverage on my windows? I will eagerly await reply.

[abhinav92003]

I don't know about other ways to get code coverage, but as far as the DynamoRIO issue is concerned: I don't think Windows on ARM is well-supported by DynamoRIO yet.

It will be useful to get more details about the issue you're seeing. Try the debug build (pass -DDEBUG=ON to cmake), which may hit some assert somewhere. Also try getting more logs using -loglevel x to drrun. See https://dynamorio.org/page_debugging.html for more tips.

[Nzoth9]

Thank you! @abhinav92003

[Nzoth9]

@abhinav92003! First of all thank you for your reply.

I ran it with the -debug option as you said, but in version 7.1.0-1 ERROR: unable to inject: exec of |(null)| failed is output.

On the other hand, in the latest release version 9.0.19349, there is no response (no output is shown).

Should I give up simply getting the code coverage of an EXE file in my ARM environment?

[abhinav92003]

On the other hand, in the latest release version 9.0.19349, there is no response (no output is shown).

That's surprising. What is the exit code? Is it non-zero?

Should I give up simply getting the code coverage of an EXE file in my ARM environment?

While DR does support Linux on AArch64, I don't think there's any quick solution yet to make DR work for Windows on AArch64. I updated this issue's title to cover that. If you make progress on the Windows support, we welcome contributions.

[Nzoth9]

Thank you for reply <3 As you said, I tried using DynamoRIO's drstrace to see the exit code.

PS C:\Users\Eo\Desktop\DrMemory-Windows-2.5.0\bin64> .\drstrace.exe -- calc
PS C:\Users\Eo\Desktop\DrMemory-Windows-2.5.0\bin64> (Nothing is output.)

I'll try with Cygwin strace.

• strace /cygdrive/c/Users/Eo/Desktop/DrMemory-Windows-2.5.0/bin/drstrace.exe

5.0/bin/drstrace.exe/Users/Eo/Desktop/DrMemory-Windows-2.5.0/bin/drstrace.exe
--- Process 16212 created
--- Process 16212 loaded C:\Windows\System32\ntdll.dll at 00007ffd0dc10000
--- Process 16212 loaded C:\Windows\SyChpe32\ntdll.dll at 0000000077820000
--- Process 16212 loaded C:\Windows\System32\wow64.dll at 00007ffd0daf0000
--- Process 16212 loaded C:\Windows\System32\wow64base.dll at 00007ffd0a490000
--- Process 16212 loaded C:\Windows\System32\wow64win.dll at 00007ffd0af00000
--- Process 16212 loaded C:\Windows\System32\wow64con.dll at 00007ffd0d220000
--- Process 16212 loaded C:\Windows\System32\xtajit.dll at 0000000077710000
--- Process 16212 loaded C:\Windows\SyChpe32\kernel32.dll at 0000000077190000
--- Process 16212 loaded C:\Windows\SyChpe32\KernelBase.dll at 00000000772a0000
--- Process 16212 thread 20324 created
--- Process 16212 thread 19528 created
--- Process 16212 thread 8820 created
--- Process 16212 loaded C:\Users\Eo\Desktop\DrMemory-Windows-2.5.0\bin\dbghelp.dll at 0000000061ad0000
--- Process 16212 loaded C:\Users\Eo\Desktop\DrMemory-Windows-2.5.0\bin\drconfiglib.dll at 0000000061c60000
--- Process 16212 loaded C:\Users\Eo\Desktop\DrMemory-Windows-2.5.0\bin\drinjectlib.dll at 0000000061ca0000
--- Process 16212 loaded C:\Windows\SyChpe32\ucrtbase.dll at 0000000076ad0000
--- Process 16212 loaded C:\Windows\SyChpe32\advapi32.dll at 00000000765e0000
--- Process 16212 loaded C:\Windows\SyChpe32\advapi32.dll at 0000000007bc0000
--- Process 16212 loaded C:\Windows\SyChpe32\msvcrt.dll at 0000000075910000
--- Process 16212 unloaded DLL at 0000000007bc0000
--- Process 16212 loaded C:\Users\Eo\Desktop\DrMemory-Windows-2.5.0\bin\dynamorio.dll at 0000000061950000
--- Process 16212 loaded C:\Windows\SyChpe32\rpcrt4.dll at 00000000775e0000
--- Process 16212 loaded C:\Windows\SysWOW64\sechost.dll at 0000000073e10000
--- Process 16212 loaded C:\Windows\SysWOW64\sechost.dll at 0000000007d40000
--- Process 16212 loaded C:\Windows\SysWOW64\sechost.dll at 0000000007dd0000
--- Process 16212 unloaded DLL at 0000000007dd0000
--- Process 16212 unloaded DLL at 0000000007d40000
--- Process 16212, exception 4000001f at 00000000779a905b
--- Process 16212, exception c0000005 at 0000000007c17be8
--- Process 16212, exception c0000005 at 00000000619f506f
--- Process 16212 thread 20412 created
--- Process 16212 thread 15132 exited with status 0xc0000005
--- Process 16212 thread 20324 exited with status 0xc0000005
--- Process 16212 thread 19528 exited with status 0xc0000005
--- Process 16212 thread 8820 exited with status 0xc0000005
--- Process 16212 exited with status 0xc0000005

Next, let's test drrun again. (It was so nice to have it because it was a small progress.)

• strace /Desktop/DynamoRIO-Windows-7.1.0-1/bin64/drrun.exe -t drcov -- calc

strace /cygdrive/c/Users/Eo/Desktop/DynamoRIO-Windows-7.1.0-1/bin64/drrun.exe -t drcov -- calc
--- Process 8260 created
--- Process 8260 loaded C:\Windows\System32\ntdll.dll at 00007ffd0dc10000
--- Process 8260 loaded C:\Windows\System32\xtajit64.dll at 00007ffd0cb00000
--- Process 8260 loaded C:\Windows\System32\kernel32.dll at 00007ffd0c730000
--- Process 8260 loaded C:\Windows\System32\KernelBase.dll at 00007ffd09b30000
--- Process 8260 loaded C:\Windows\System32\apphelp.dll at 00007ffd08d70000
--- Process 8260 thread 11656 created
--- Process 8260 thread 18508 created
--- Process 8260 loaded C:\Users\Eo\Desktop\DynamoRIO-Windows-7.1.0-1\bin64\drconfiglib.dll at 00007ffc58eb0000
--- Process 8260 loaded C:\Users\Eo\Desktop\DynamoRIO-Windows-7.1.0-1\bin64\drinjectlib.dll at 00007ffc4de80000
--- Process 8260 loaded C:\Windows\System32\advapi32.dll at 00007ffd0b3a0000
--- Process 8260 loaded C:\Windows\System32\advapi32.dll at 0000014357f70000
--- Process 8260 loaded C:\Windows\System32\msvcrt.dll at 00007ffd0ab50000
--- Process 8260 unloaded DLL at 0000014357f70000
--- Process 8260 loaded C:\Windows\System32\sechost.dll at 00007ffd0cf10000
--- Process 8260 loaded C:\Windows\System32\rpcrt4.dll at 00007ffd0c8a0000
--- Process 8260 thread 20132 created
ERROR: unable to inject: exec of |(null)| failed
--- Process 8260, exception c0000008 at 00007ffd0dd82888
--- Process 8260 thread 15600 exited with status 0xc0000008
--- Process 8260 thread 20132 exited with status 0xc0000008
--- Process 8260 thread 18508 exited with status 0xc0000008
--- Process 8260 exited with status 0xc0000008

• strace /Desktop/DynamoRIO-Windows-9.0.19349 2/bin64/drrun.exe -t drcov -- notepad

Here is the strace log from the latest version (9.0.19349):

--- Process 15976 created
--- Process 15976 loaded C:\Windows\System32\ntdll.dll at 00007ffd0dc10000
--- Process 15976 loaded C:\Windows\System32\xtajit64.dll at 00007ffd0cb00000
--- Process 15976 loaded C:\Windows\System32\kernel32.dll at 00007ffd0c730000
--- Process 15976 loaded C:\Windows\System32\KernelBase.dll at 00007ffd09b30000
--- Process 15976 loaded C:\Windows\System32\apphelp.dll at 00007ffd08d70000
--- Process 15976 thread 15868 created
--- Process 15976 thread 11108 created
--- Process 15976 loaded C:\Users\Eo\Desktop\DynamoRIO-Windows-9.0.19349 2\bin64\drconfiglib.dll at 00007ffc58ea0000
--- Process 15976 loaded C:\Users\Eo\Desktop\DynamoRIO-Windows-9.0.19349 2\bin64\drinjectlib.dll at 00007ffc4cbb0000
--- Process 15976 loaded C:\Windows\System32\advapi32.dll at 00007ffd0b3a0000
--- Process 15976 loaded C:\Windows\System32\advapi32.dll at 000001b83d430000
--- Process 15976 loaded C:\Windows\System32\msvcrt.dll at 00007ffd0ab50000
--- Process 15976 unloaded DLL at 000001b83d430000
--- Process 15976 loaded C:\Windows\System32\sechost.dll at 00007ffd0cf10000
--- Process 15976 loaded C:\Windows\System32\rpcrt4.dll at 00007ffd0c8a0000
--- Process 15976 thread 15420 created
--- Process 15976 loaded C:\Windows\System32\ApiSetHost.AppExecutionAlias.dll at 00007ffcfef40000
--- Process 15976 loaded C:\Windows\System32\ucrtbase.dll at 00007ffd09380000
--- Process 15976 loaded C:\Windows\System32\combase.dll at 00007ffd0d230000
--- Process 15976 loaded C:\Windows\System32\msvcp_win.dll at 00007ffd097f0000
--- Process 15976 loaded C:\Windows\System32\profapi.dll at 00007ffd09290000
--- Process 15976 loaded C:\Windows\System32\windows.storage.dll at 00007ffd052b0000
--- Process 15976 loaded C:\Windows\System32\WinTypes.dll at 00007ffd05000000
--- Process 15976 loaded C:\Windows\System32\SHCore.dll at 00007ffd0c4d0000
--- Process 15976 loaded C:\Windows\System32\shlwapi.dll at 00007ffd0b2f0000
--- Process 15976 thread 11504 created
--- Process 15976 thread 13096 created
--- Process 15976 loaded C:\Windows\System32\kernel.appcore.dll at 00007ffd075e0000
--- Process 15976 loaded C:\Windows\System32\daxexec.dll at 00007ffce4740000
--- Process 15976 loaded C:\Windows\System32\oleaut32.dll at 00007ffd0b140000
--- Process 15976 loaded C:\Windows\System32\container.dll at 00007ffce66b0000
--- Process 15976 loaded C:\Windows\System32\userenv.dll at 00007ffd07e20000
--- Process 15976 loaded C:\Windows\System32\bcryptprimitives.dll at 00007ffd09720000
--- Process 15976 loaded C:\Windows\System32\clbcatq.dll at 00007ffd0aa10000
--- Process 15976 thread 17180 created
--- Process 15976 loaded C:\Windows\System32\Windows.StateRepositoryPS.dll at 00007ffcf39e0000
--- Process 15976 loaded C:\Windows\System32\AppxAllUserStore.dll at 00007ffce62a0000
--- Process 15976 loaded C:\Windows\System32\Windows.StateRepositoryClient.dll at 00007ffcf3f50000
--- Process 15976 loaded C:\Windows\System32\Windows.StateRepositoryCore.dll at 00007ffcf57a0000
--- Process 15976 loaded C:\Windows\System32\capauthz.dll at 00007ffce4a80000
--- Process 15976 loaded C:\Windows\System32\ntmarta.dll at 00007ffd07610000
--- Process 15976 loaded C:\Users\Eo\Desktop\DynamoRIO-Windows-9.0.19349 2\lib64\release\dynamorio.dll at 0000000071000000
--- Process 14428 created
--- Process 14428 loaded C:\Windows\System32\ntdll.dll at 00007ffd0dc10000
--- Process 14428 loaded C:\Windows\System32\kernel32.dll at 00007ffd0c730000
--- Process 14428 loaded C:\Windows\System32\KernelBase.dll at 00007ffd09b30000
--- Process 14428 loaded C:\Windows\System32\shlwapi.dll at 00007ffd0b2f0000
--- Process 14428 loaded C:\Windows\System32\msvcrt.dll at 00007ffd0ab50000
--- Process 14428 thread 16432 created
--- Process 14428, exception c000001d at 00007ffd0dc82db0
--- Process 14428 thread 16432 exited with status 0xc000001d
--- Process 14428 loaded C:\Windows\System32\user32.dll at 00007ffd0aca0000
--- Process 14428 exited with status 0xc000001d
--- Process 15976 thread 17180 exited with status 0xc000001d
--- Process 15976 thread 13096 exited with status 0xc000001d
--- Process 15976 thread 11504 exited with status 0xc000001d
--- Process 15976 thread 15420 exited with status 0xc000001d
--- Process 15976 thread 11108 exited with status 0xc000001d
--- Process 15976 thread 15868 exited with status 0xc000001d
--- Process 15976 exited with status 0xc000001d
Illegal instruction

And when Intel PT's pin is also executed, many libraries are loaded and it ends as follows.

• strace pin.exe -- notepad

--- Process 16188 thread 19044 created
--- Process 16188 loaded C:\Users\Eo\Desktop\pin-3.25-98650-g8f6168173-msvc-windows\pin-3.25-98650-g8f6168173-msvc-windows\inncrt.dll at 00007ffc3e6c0000
--- Process 16188 thread 14764 exited with staE: [tid:6088] Failed to Inject
tus 0x0
E: --- Process 13468 createPin is exiting due to fatal error
d
--- Process 7676 thread 17740 exited with status 0xffffffff
--- Process 7676 thread 5616 exited with status 0xffffffff
--- Process 7676 thread 6088 exited with status 0xffffffff
--- Process 7676 thread 15272 exited with status 0xffffffff
--- Process 7676 thread 13676 exited with status 0xffffffff
--- Process 7676 thread 20128 exited with status 0xffffffff
--- Process 7676 thread 4416 exited with status 0xffffffff
--- Process 7676 thread 3932 exited with status 0xffffffff
--- Process 7676 thread 11480 exited with status 0xffffffff
--- Process 7676 thread 20144 exited with status 0xffffffff
--- Process 13468 loaded C:\Windows\System32\ntdll.dll at 00007ffd0dc10000
--- Process 7676 exited with status 0xffffffff
--- Process 13468 exited with status 0x1
--- Process 16188 thread 17272 exited with status 0x0
--- Process 13444 loaded C:\Windows\SysWOW64\kernel.appcore.dll at 00000000732e0000
--- Process 16188 thread 19044 exited with status 0x0
--- Process 13444 loaded C:\Windows\SyChpe32\msvcrt.dll at 0000000075910000
--- Process 16188 thread 2260 exited with status 0x0
--- Process 13444 thread 15828 exited with status 0xffffffff
--- Process 13444 thread 12836 exited with status 0xffffffff
--- Process 16188 thread 14896 exited with status 0x0
--- Process 16188 thread 19088 exited with status 0x0
--- Process 16188 thread 18272 exited with status 0x0
--- Process 13444 exited with status 0xffffffff
--- Process 16188 exited with status 0x0

Thank you. dear @abhinav92003, I hope you have a day full of God's happiness.

[DebuggingSystems]

I am interested in this feature as well. I noticed that on one of the stack traces, threads are exiting with 0xC000001D (STATUS_ILLEGAL_INSTRUCTION).

This appears expected when running ARM64X binaries (containing x64 and ARM64 code) without support from the loader.

https://learn.microsoft.com/en-us/cpp/build/arm64ec-windows-abi-conventions

https://learn.microsoft.com/en-us/windows/arm/arm64x-pe

@derekbruening I read some of the code in core/win32 and would like to help, if that is okay.

If you have any pointers for an attempt at this issue, I would appreciate them. 🙂

[derekbruening]

Thank you for your interest and willingness to help.

I believe that most of the work would involve porting x86 code in the core/win32 directory:

• ntdll.dll hooks: there are multiple x86 code sequences in https://github.com/DynamoRIO/dynamorio/blob/master/core/win32/callback.c#L305 used for Ki routine hooks and system call hooks. These would need to be ported to aarch64.
• injection: there are a lot of x86 pieces of generated code that need to be ported

There are various other x86-assuming pieces of code in core/win32: run grep -r -i eax core/win32 e.g. to see some.

There are probably a number of places elsewhere in the code where aarch64 is assumed to be Linux.

Happy to answer further questions.

[DebuggingSystems]

@derekbruening Updating so I don't leave others waiting for this.

I believe this was too large an issue for me to take on while I'm still learning how DynamoRIO works, and I should look into simpler issues such as the ones in the private loader comments. Either way, I left the build configuration changes I needed here: https://github.com/DebuggingSystems/dynamorio/commits/windows-arm64/

Sorry!

[yuhui518]

@DebuggingSystems Have you compiled a program that can run on Win11 ARM？

@Nzoth9 I am interested in this feature as well.But I use opencppcoverage tool to get code coverage.Opencppcoverage can only obtain code coverage for x64.As we know,Win11 ARM also support to run x64 application.

set srcpath=D:\mycompilepath\x64\vmgt
set dstpath=C:\run_exe_path\x64\vmgt
set OpenOption=--cover_children --continue_after_cpp_exception
set modules=--modules=%dstpath%\x64\Debug
set sources=--sources=%dstpath% --excluded_sources=*.h 
set substitute=--substitute_pdb_source_path=%srcpath%?%dstpath%
set inputfile=--input_coverage=1.cov

OpenCppCoverage %OpenOption% --export_type=binary:1.cov %sources% %substitute% -- 1.exe
OpenCppCoverage %OpenOption% --export_type=html %inputfile% %sources% %substitute% -- 2.exe

I also hope that DynamoRIO could support win11 arm64.Opencppcoverage still has many imperfections.

[yuhui518]

@DebuggingSystems Thanks for your configuration changes,I could make it ,but I can't build it. I use vs2019 vcvarsamd64_arm64.bat.

mkdir build && cd build
cmake -G"Visual Studio 16 2019" -A ARM64 -DDEBUG=ON ..
cmake --build .

There were many errors that occurred when I was building.

[DebuggingSystems]

@yuhui518 I used VS 2022, I don’t believe the previous version supports ARM64 as well.

Have you compiled a program that can run on Win11 ARM？

I only managed to compile the binaries, the difficult part was due to the x64 emulation, all system libraries being compiled as ARM64X for compatibility with both architectures didn’t make it easier.

See the files @derekbruening mentioned above, and this. I hope it helps.