search

XboxDev / nxdk

The cross-platform, open-source SDK to develop for original Xbox: *new* xdk
445 stars 68 forks source link

Recreate import libs locally #119

Open JayFoxRox opened 5 years ago

JayFoxRox commented 5 years ago

Users should be able to modify libxboxkrnl.lib and others. Our makefile should generate these files.

This was originally part of nxdk, but we removed it because the LLVM tools didn't create compatible import libs.

I've now created a patch for llvm-dlltool that makes this possible. I try to get this upstreamed, although I'm not motivated to use their mailing-list.

It's optional for xboxkrnl.lib, because that's considered complete. However, we might want other import libs in the future (I'm working on xbdm.lib for example).

(Edit: This section was added in September 2019)

I'm removing my fork of LLVM; this was the description for my LLVM patch (which has been upstreamed; see comment from May below):

The patch was aimed to fix issues with an import libraries for DXT (dynamically linked debugger extension) but also affects the xboxkrnl import library.

On original Xbox, debugger extensions will try to resolve imports from the debugger.

  • That debugger only exports by ordinal.
  • The symbol resolver for plugins will ignore the ordinal if a name is specified for an import.

With LLVM master, the NONAME keyword in the .def for the import-lib (generated with llvm-dlltool) is ignored, so all imports will happen by name. This prevents extensions, that were build using LLVM, from being loaded.

With this patch, the NONAME option works as intended; import will happen by ordinal.

Generate import lib:

(This is the broken step)

llvm-dlltool -d "xbdm.dll.def" -m i386 -l "libxbdm.lib" -k -f=-s

With the following def:

LIBRARY xbdm.dll

EXPORTS
    DmNotify@12                              @  24 NONAME
    DmOpenNotificationSession@8              @  25 NONAME
    DmRegisterCommandProcessor@8             @  30 NONAME
    DmSendNotificationString@4               @  36 NONAME

Before (via objdump -x):

The Import Tables (interpreted .data section contents)
 vma:            Hint    Time      Forward  DLL       First
                 Table   Stamp     Chain    Name      Thunk
 00054370 000543ac 00000000 00000000 00054594 0005448c

  DLL Name: xbdm.dll
  vma:  Hint/Ord Member-Name Bound-To
  5456c      24  DmNotify
  54578      25  DmOpenNotificationSession

After (via objdump -x):

The Import Tables (interpreted .data section contents)
 vma:            Hint    Time      Forward  DLL       First
                 Table   Stamp     Chain    Name      Thunk
 00054370 000543ac 00000000 00000000 0005456c 0005448c

  DLL Name: xbdm.dll
  vma:  Hint/Ord Member-Name Bound-To
  80000018       24  <none>
  80000019       25  <none>
JayFoxRox commented 5 years ago

My patch has landed in https://github.com/llvm-mirror/llvm/commit/7a38c439b6df87ff25c3c67004679ebd4c6ac823 thanks to @thrimbor for tests and upstreaming.

This should be part of LLVM 9.0, which should be released around September 2019.

As result, this should be in Ubuntu 19.10 / Ubuntu 20.04 LTS. There are nightly builds for Ubuntu though, so we can probably upgrade before that, or make it an optional feature.

I'm not sure when MSYS will adapt it, when MSVS adapts it, or how releases work on macOS.

JayFoxRox commented 4 years ago

It would be nice if people could test their installation of llvm-dlltool by following the steps described in this post.

I'm not sure which version of LLVM ships in MSYS2 or on Ubuntu, but if thrimbors patch has propagated, then I'd like to go forward with removing the import-lib binaries from our repository.

You should be able to test your LLVM by compiling a sample with https://github.com/JayFoxRox/nxdk/pull/52 Please delete lib/xboxkrnl/libxboxkrnl.lib before trying this. Also delete your samples main.exe / main.lib / main.obj or any other leftovers from previous compilations.

I did this (after deleting the mentioned files):

$ cd samples/triangle
$ make -B
[...]
$ objdump -x main.exe 
[...]

The Import Tables (interpreted .rdata section contents)
 vma:            Hint    Time      Forward  DLL       First
                 Table   Stamp     Chain    Name      Thunk
 0001e0ac   0001e0d4 00000000 00000000 0001e2ac 0001e1c0

    DLL Name: xboxkrnl.exe
    vma:  Hint/Ord Member-Name Bound-To
    80000036       54  <none>
    800000fa      250  <none>
    80000001        1  <none>
    80000002        2  <none>
    80000003        3  <none>
    80000018       24  <none>
    8000002c       44  <none>
    8000002e       46  <none>
    8000002f       47  <none>
    80000031       49  <none>
    80000043       67  <none>
    80000045       69  <none>
    80000062       98  <none>
    80000063       99  <none>
    80000064      100  <none>
    80000068      104  <none>
    8000006b      107  <none>
[...]

You can also use llvm-objdump if you don't have objdump, but the output isn't as meaningful (as it doesn't tell us if the import is by empty name or ordinal - which we care about):

$ llvm-objdump -x main.exe 
[...]
The Import Tables:
  lookup 0001e0d4 time 00000000 fwd 00000000 name 0001e2ac addr 0001e1c0

    DLL Name: xboxkrnl.exe
    Hint/Ord  Name
          54
         250
           1
           2
           3
          24
          44
          46
          47
          49
          67
          69
          98
          99
         100
         104
         107
[...]

The built default.xbe should still work.

All of my tests were done on Arch Linux llvm 9.0.0-4 and GNU objdump (GNU Binutils) 2.33.1.

Interestingly, inspecting xboxkrnl.lib seems to imply that the -k option broke (-k: Kill @n Symbol from export)? This might be important for DXT support, so please also do this (after building libxboxkrnl.lib with my linked branch):

Edit: -k option works, it's just not visible in nm. The linked binaries will have the correct names (Tested on LLVM 9; @dracc also confirmed this behaviour for LLVM 6 on Discord).

Click here for my faulty test for the `-k` option and results from `nm` (which doesn't show the difference, even though it exists): ``` $ llvm-nm lib/xboxkrnl/libxboxkrnl.lib [...] xboxkrnl.exe: 00000000 T _AvGetSavedDataAddress@0 00000000 T __imp__AvGetSavedDataAddress@0 xboxkrnl.exe: 00000000 T _AvSendTVEncoderOption@16 00000000 T __imp__AvSendTVEncoderOption@16 xboxkrnl.exe: 00000000 T _AvSetDisplayMode@24 00000000 T __imp__AvSetDisplayMode@24 xboxkrnl.exe: 00000000 T _AvSetSavedDataAddress@4 00000000 T __imp__AvSetSavedDataAddress@4 xboxkrnl.exe: 00000000 T _DbgBreakPoint@0 00000000 T __imp__DbgBreakPoint@0 xboxkrnl.exe: 00000000 T _DbgBreakPointWithStatus@4 00000000 T __imp__DbgBreakPointWithStatus@4 xboxkrnl.exe: 00000000 T _DbgLoadImageSymbols@12 00000000 T __imp__DbgLoadImageSymbols@12 [...] ```
dracc commented 4 years ago

I've tested building and running according to the instructions above using llvm-dlltool version 6.0, 7, 8, 9 and 10. 9 and 10 seems to work (as was expected). Do note that none of these packages creates a symlink for llvm-dlltool by default in Debian - you have to create one yourself.

llvm-dlltool-6.0: Version: **1:6.0.1-11** ``` $ objdump -x main.exe [...] There is an import table in .rdata at 0x1001f034 The Import Tables (interpreted .rdata section contents) vma: Hint Time Forward DLL First Table Stamp Chain Name Thunk 0001f034 0001f05c 00000000 00000000 0001f740 0001f148 DLL Name: xboxkrnl.exe vma: Hint/Ord Member-Name Bound-To 1f234 54 InterlockedExchange 1f24a 250 ObfDereferenceObject 1f262 1 AvGetSavedDataAddress 1f27a 2 AvSendTVEncoderOption 1f292 3 AvSetDisplayMode 1f2a6 24 ExQueryNonVolatileSetting 1f2c2 44 HalGetInterruptVector [...] ``` ``` $ nm lib/xboxkrnl/libxboxkrnl.lib [...] xboxkrnl.exe: 00000000 T _AvGetSavedDataAddress@0 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__AvGetSavedDataAddress@0 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSendTVEncoderOption@16 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__AvSendTVEncoderOption@16 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSetDisplayMode@24 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__AvSetDisplayMode@24 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSetSavedDataAddress@4 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__AvSetSavedDataAddress@4 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _DbgBreakPoint@0 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__DbgBreakPoint@0 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text [...] ```
llvm-dlltool-7: Version: **1:7.0.1-9+b2** ``` $ objdump -x main.exe [...] There is an import table in .rdata at 0x1001f034 The Import Tables (interpreted .rdata section contents) vma: Hint Time Forward DLL First Table Stamp Chain Name Thunk 0001f034 0001f05c 00000000 00000000 0001f740 0001f148 DLL Name: xboxkrnl.exe vma: Hint/Ord Member-Name Bound-To 1f234 54 InterlockedExchange 1f24a 250 ObfDereferenceObject 1f262 1 AvGetSavedDataAddress 1f27a 2 AvSendTVEncoderOption 1f292 3 AvSetDisplayMode 1f2a6 24 ExQueryNonVolatileSetting 1f2c2 44 HalGetInterruptVector 1f2da 46 HalReadWritePCISpace 1f2f2 47 HalRegisterShutdownNotification 1f314 49 HalReturnToFirmware 1f32a 67 IoCreateSymbolicLink 1f342 69 IoDeleteSymbolicLink 1f35a 98 KeConnectInterrupt [...] ``` ``` $ nm lib/xboxkrnl/libxboxkrnl.lib [...] xboxkrnl.exe: 00000000 T _AvGetSavedDataAddress@0 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__AvGetSavedDataAddress@0 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSendTVEncoderOption@16 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__AvSendTVEncoderOption@16 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSetDisplayMode@24 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__AvSetDisplayMode@24 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSetSavedDataAddress@4 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__AvSetSavedDataAddress@4 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _DbgBreakPoint@0 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__DbgBreakPoint@0 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _DbgBreakPointWithStatus@4 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__DbgBreakPointWithStatus@4 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text [...] ```
llvm-dlltool-8: Version: **1:8.0.1-4** ``` $ objdump -x main.exe [...] There is an import table in .rdata at 0x1001f034 The Import Tables (interpreted .rdata section contents) vma: Hint Time Forward DLL First Table Stamp Chain Name Thunk 0001f034 0001f05c 00000000 00000000 0001f740 0001f148 DLL Name: xboxkrnl.exe vma: Hint/Ord Member-Name Bound-To 1f234 54 InterlockedExchange 1f24a 250 ObfDereferenceObject 1f262 1 AvGetSavedDataAddress 1f27a 2 AvSendTVEncoderOption 1f292 3 AvSetDisplayMode 1f2a6 24 ExQueryNonVolatileSetting 1f2c2 44 HalGetInterruptVector 1f2da 46 HalReadWritePCISpace 1f2f2 47 HalRegisterShutdownNotification 1f314 49 HalReturnToFirmware [...] ``` ``` $ nm lib/xboxkrnl/libxboxkrnl.lib [...] xboxkrnl.exe: 00000000 T _AvGetSavedDataAddress@0 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__AvGetSavedDataAddress@0 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSendTVEncoderOption@16 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__AvSendTVEncoderOption@16 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSetDisplayMode@24 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__AvSetDisplayMode@24 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSetSavedDataAddress@4 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__AvSetSavedDataAddress@4 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _DbgBreakPoint@0 00000000 I .idata$4 00000000 I .idata$5 00000000 I .idata$6 00000000 I __imp__DbgBreakPoint@0 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text [...] ```

llvm-dlltool-9: Version: **1:9.0.0-4** ``` $ objdump -x main.exe [...] There is an import table in .rdata at 0x1001f034 The Import Tables (interpreted .rdata section contents) vma: Hint Time Forward DLL First Table Stamp Chain Name Thunk 0001f034 0001f05c 00000000 00000000 0001f234 0001f148 DLL Name: xboxkrnl.exe vma: Hint/Ord Member-Name Bound-To 80000036 54 800000fa 250 80000001 1 80000002 2 80000003 3 80000018 24 8000002c 44 8000002e 46 8000002f 47 80000031 49 [...] ``` ``` $ nm lib/xboxkrnl/libxboxkrnl.lib [...] xboxkrnl.exe: 00000000 T _AvGetSavedDataAddress@0 00000000 I .idata$4 00000000 I .idata$5 00000000 I __imp__AvGetSavedDataAddress@0 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSendTVEncoderOption@16 00000000 I .idata$4 00000000 I .idata$5 00000000 I __imp__AvSendTVEncoderOption@16 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSetDisplayMode@24 00000000 I .idata$4 00000000 I .idata$5 00000000 I __imp__AvSetDisplayMode@24 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSetSavedDataAddress@4 00000000 I .idata$4 00000000 I .idata$5 00000000 I __imp__AvSetSavedDataAddress@4 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _DbgBreakPoint@0 00000000 I .idata$4 00000000 I .idata$5 00000000 I __imp__DbgBreakPoint@0 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text [...] ```

llvm-dlltool-10: Version: **1:10~+201911120943210600592dd459242-1~exp1** ``` $ objdump -x main.exe [...] There is an import table in .rdata at 0x1001f034 The Import Tables (interpreted .rdata section contents) vma: Hint Time Forward DLL First Table Stamp Chain Name Thunk 0001f034 0001f05c 00000000 00000000 0001f234 0001f148 DLL Name: xboxkrnl.exe vma: Hint/Ord Member-Name Bound-To 80000036 54 800000fa 250 80000001 1 80000002 2 80000003 3 80000018 24 8000002c 44 8000002e 46 8000002f 47 80000031 49 [...] ``` ``` $ nm lib/xboxkrnl/libxboxkrnl.lib [...] xboxkrnl.exe: 00000000 T _AvGetSavedDataAddress@0 00000000 I .idata$4 00000000 I .idata$5 00000000 I __imp__AvGetSavedDataAddress@0 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSendTVEncoderOption@16 00000000 I .idata$4 00000000 I .idata$5 00000000 I __imp__AvSendTVEncoderOption@16 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSetDisplayMode@24 00000000 I .idata$4 00000000 I .idata$5 00000000 I __imp__AvSetDisplayMode@24 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _AvSetSavedDataAddress@4 00000000 I .idata$4 00000000 I .idata$5 00000000 I __imp__AvSetSavedDataAddress@4 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text xboxkrnl.exe: 00000000 T _DbgBreakPoint@0 00000000 I .idata$4 00000000 I .idata$5 00000000 I __imp__DbgBreakPoint@0 U __IMPORT_DESCRIPTOR_xboxkrnl 00000000 T .text [...] ```

All my tests were run on Debian testing with GNU Binutils 2.33.1.

JayFoxRox commented 4 years ago

Keep tests from MSYS2 and Ubuntu coming please (which version of LLVM do they use by default?)

Also https://reviews.llvm.org/D68352#1700956 mentions that lld-link is a better option to create an import lib. I had speculated this myself, because earlier I noticed that it supported /DEF:; also /LIB to make it behave like lib.exe (which is our actual goal). I'm not sure if it actually respects the /DEF option in /LIB mode, because llvm-lib (which should be lib.exe) does not support it (for reasons I don't understand).