Dynamic linker, gp-relative addressing, ldscripts and more

[spicyjpeg]

I'm working on a large project (not yet finished but it's going to be open source, also I lost access to my old github and had to register again) and these are some changes I made so far to the SDK.

First of all I added support for dynamic linking, i.e. loading sub-executables or libraries from .DLL files at runtime and patching them so they can access functions from the main executable (useful for reducing RAM usage by unloading unused code, or for loading mods/patches). I don't know whether a similar API exists in the Sony SDK but I tried to keep mine similar to the POSIX dlopen() API. The implementation relies on a "symbol map" file listing the address of every symbol in an executable, which can be generated at build time using GCC's nm command. The relevant functions are part of psxetc and are declared in dlfcn.h, along with some quick javadoc comments explaining how to use them. Proper documentation should eventually be written after more testing.

I had to edit pretty much all Makefiles to ensure libpsn00b is compiled with $gp-relative addressing disabled, due to DLL code using $gp internally. Executables can still be built with $gp-relative addressing enabled as long as they don't use the dynamic linker. I moved compiler options for all target types into a single file, psn00bsdk-setup.mk (fixing #18), so they can be quickly edited for all examples. This also allowed me to add proper linker scripts for both executables and DLLs, which makes installation easier as patching GCC's stock ldscripts would no longer be required.

As if all this wasn't enough (lol) I added BIOS function FlushCache(), fixed some missing header declarations, a few minor bugs (#33) and wrote an example on how to build DLLs and use the dynamic linker. The example works on no$psx, pcsx-redux and DuckStation, but I haven't tested on real hardware or with GCC versions other than 11.1.0.

I split all changes into three separate commits for easier review.

[Lameguy64]

Well that's pretty nuts. I never thought dynamic linking would be possible on the PS1 and I don't think it would be important as the official SDK doesn't support dynamic linking either. The closest you can get to externally loaded modules are code overlays but the linking on those are fixed at compile time.

It'll take me awhile to review and merge your changes as I've been pretty busy with many things at the moment. If modules can be loaded from a BIOS device and not a CD-ROM I could test it on real hardware as my n00bROM based setup supports PCDRV for loading files.

[spicyjpeg]

If modules can be loaded from a BIOS device and not a CD-ROM I could test it on real hardware as my n00bROM based setup supports PCDRV for loading files.

I made DL_LoadSymbolMap() and dlopen() depend on the BIOS file API for this exact reason. They should work with custom file drivers, on devkits and possibly even on arcade boards with non-standard CD drives (e.g. Konami System 573). They are just wrappers around the "real" functions that work on buffers in memory, DL_ParseSymbolMap() and dlinit(), which are still accessible for advanced usage such as loading DLLs from compressed archives.

A few other random notes:

• I just noticed I left #define DEBUG uncommented in dl.c. That should be commented prior to merging as debug logging makes DLL loading noticeably slower (due to entire symbol tables being dumped to the console).
• Since the dynamic linker relies heavily on heap allocation, I edited malloc() and free() to be declared as weak so they can be overridden with alternate implementations, or wrapped to track RAM usage.
• The error messages and dlerror() could/should be removed to save space, even though that would break POSIX "compatibility" (who cares about that anyway). IMO the best solution would be to make the ErrorCode enum public and have dlerror() return codes rather than strings.
• Symbol maps could be stored as binary files instead of text, by creating a custom elf2map tool to extract the symbol table from ELF executables at build time. This would allow getting rid of the ugly parsing and hash-table-building code, but the current implementation seems to work fine so it's better not to fix what isn't broken.
• The DL_CALL() macro currently required to call functions in DLLs can probably be removed by telling GCC to always store function addresses into $t9, however there seems to be no simple way to do that.

[Lameguy64]

Just finished evaluating. Going to merge now.