Switching to (or adding support for) CMake

[spicyjpeg]

Would it make sense for PSn00bSDK to move to, or at least to add support for, a CMake-based build system?

In my PR (#36) to add support for dynamic linking I had to edit pretty much all Makefiles, and it became clear that they are not the easiest thing to maintain when there are almost 30 of them. I am already using CMake (modern 3.20, not the ancient 2.8 stuff) in my upcoming PS1 homebrew project and it's very easy to set up for cross-compiling, only requiring two files to be added. With those in place, a simple project's CMakeLists.txt would just be the following:

cmake_minimum_required(VERSION 3.20)

set(CMAKE_TOOLCHAIN_FILE ../psn00bsdk-toolchain.cmake)
project(
    hello_world
    DESCRIPTION "PSn00bSDK CMake example"
    LANGUAGES C CXX ASM
)
include(../psn00bsdk-setup.cmake)

psn00bsdk_add_executable(main.exe main.c)
psn00bsdk_add_cd_image(hello_world.iso iso.xml.template)
# other macros could be defined e.g. for converting TIM images

If CMake support is added it would make sense to use CMake for building the libraries and examples as well, ditching the current Makefiles altogether. It would also help making the SDK easier to use with an IDE and/or under Windows, as CMake integrates with various IDEs and build systems that don't need Cygwin or MSys.

[Lameguy64]

I'll have to take time reevaluating cmake as I've had many issues trying to use it in the past. But you're right, the makefile method is a bit convoluted and messy in places that even I myself am starting to find it rather cumbersome to maintain.

Though another reason I'm sticking to makefiles in the meantime is so that PSn00bSDK projects can be built under Win9x coupled with the original msys, as I've managed to compile GCC 7.4.0 with MinGW GCC 4.7.2 which produces binaries that still work in 9x. This would come in handy once hardware communications with the DTL-H2000 have been figured out, and I have a feeling current versions of cmake would make use of dependencies that cannot be built with older GCC versions or make use of stupid MSVC calls not available in older versions of Windows. Though I suppose this can be avoided by simply writing the custom DTL-H2000 communication tools to work in Linux or one of the BSDs instead.

I propose you could make a fork that uses cmake exclusively so people who'd rather use cmake will use your fork, plus it'd also give me a chance to evaluate how it's all going to work myself and possibly merge it to mainline.

[saxbophone]

I too am interested in the use of a CMake-based build system. As unwieldy as its syntax can sometimes be, I have found it to be highly effective for setting common compiler flags and such in a concise way. I'm not too familiar with this project's Makefiles but at a glance, it looks like a lot is going on in terms of setting custom linker flags and what-not...

In regards to @Lameguy64 's comments about the usefulness of keeping the Makefiles for Win9x, it's possible to generate Makefiles with CMake. It would need testing to verify if the output achieves the same affects as the current Makefiles, though.

[Lameguy64]

It'd be incredibly clunky if I have to run cmake on a separate machine just to build it on 9x though. However, if cmake does not depend on too many needless dependencies that make use of newer WinAPI calls it might be possible to compile it for 9x.

[saxbophone]

Ah yes that's a good point!

Given that Kitware (CMake vendor) orients itself towards cross-platform dev, it wouldn't surprise me if the internals of CMake are very cross-platform. Then again, I'm not familiar with its source. From a quick glance at the current source code, the GUI appears to be Qt-based.

I appreciate this may be quite a large undertaking

[spicyjpeg]

Unfortunately any makefile or script generated by CMake depends heavily on CMake itself, mostly for mundane stuff like printing colored text but also for determining dependencies or building subprojects. That's in fact why makefiles generated by CMake on Windows don't need Cygwin: the CMake binary actually provides busybox-style implementations of several Unix commands (cmake -E prints out a list).

For DTL-H2000 owners I'd propose a completely different solution. Rather than trying to get the entire SDK to work on Windows 9x, I think some sort of server that exposes a remote GDB target for a modern machine to connect to would be more useful. This way modern IDEs and tools can be used with none of the hassle that comes with a 25-year-old PC, while still having the ability to debug on real hardware. Connecting Windows 9x to a network in 2021 might not seem a bright idea, but most consumer routers have options for limiting devices to LAN only (you'd probably want a network connection anyway to transfer files).

All this said, I have just published my CMake branch and opened #38, complete with initial support for CPack so we could eventually offer prebuilt packages and one-click installers. Let me know if anything needs fixing as I haven't yet tested it extensively.

[Lameguy64]

Well, this is coming from a guy who actually tried to do Arduino programming with AVR assembly one time just to be unusual. Since the H2000 boards haven't been cracked yet it's not of major concern for now.

Once the H2000 boards have been cracked I've envisioned writing the tools to interface with it for Linux or NetBSD, though the latter's likely preferred as the operating system appears to be light enough to run snappily on older systems still. I imagine it'd be kinda like the PS2 TOOL development unit once such a thing comes to fruition.

[spicyjpeg]

I pushed a few more bugfixes and the CMake branch is now usable and fully functional, at least on Linux. I haven't yet been able to get the GCC toolchain to work on Windows, not even the precompiled one, however I managed to build the tools with MSVC after some patching (like I did with mkpsxiso).

Other than Windows compatibility, there are still a few more things to do before the branch can be merged:

• Test the SDK on macOS, especially on M1 processors. In theory everything should just work as CMake abstracts away OS-specific compilers and stuff, but I don't own a Mac for testing.
• Proofread all the new documentation I wrote, and maybe add proper tutorials on how to use the CMake macros and the dynamic linker.
• Merge Lameguy64/mkpsxiso#18 and edit the URL in CMakeLists.txt to point to the main mkpsxiso repo rather than my fork.

Regarding the DTL-H2000, I'd say it's probably best to focus on emulators and software-based loaders/debuggers that run on a regular PS1. The number of users owning and actually willing to use a devkit is very small, so getting into massive reverse engineering efforts on unreliable 1990's PC hardware is IMO not worth it. The top priority right now is making the SDK accessible and easy to install, 'cause nobody wants to use something that's even more complicated to set up than the ancient Sony stuff.

[saxbophone]

Is there any possibility of using Github-Actions to test on Linux, macOS and Windows? I've been able to get C++ stuff unit tested just fine on all three. The only tricky bit I can think of is the long amount of time and space required to build the toolchain, although I believe public open-source projects like these get a free CI credit allowance.

[Lameguy64]

I pushed a few more bugfixes and the CMake branch is now usable and fully functional, at least on Linux. I haven't yet been able to get the GCC toolchain to work on Windows, not even the precompiled one, however I managed to build the tools with MSVC after some patching (like I did with mkpsxiso).

Other than Windows compatibility, there are still a few more things to do before the branch can be merged:

• Test the SDK on macOS, especially on M1 processors. In theory everything should just work as CMake abstracts away OS-specific compilers and stuff, but I don't own a Mac for testing.
• Proofread all the new documentation I wrote, and maybe add proper tutorials on how to use the CMake macros and the dynamic linker.
• Merge CMake script rewrite, installation rules, MSVC fix mkpsxiso#18 and edit the URL in CMakeLists.txt to point to the main mkpsxiso repo rather than my fork.

Regarding the DTL-H2000, I'd say it's probably best to focus on emulators and software-based loaders/debuggers that run on a regular PS1. The number of users owning and actually willing to use a devkit is very small, so getting into massive reverse engineering efforts on unreliable 1990's PC hardware is IMO not worth it. The top priority right now is making the SDK accessible and easy to install, 'cause nobody wants to use something that's even more complicated to set up than the ancient Sony stuff.

Well I guess you're right. Though I'm a sort who likes the idea of doing actual work on old, outdated hardware that's over 20 years old (and incidentally, spends a modest amount of cash on parts for said systems especially on a SMP board) while making fun of 'collectorhoarders' who are unwilling to power on their glorified trophies. I used to do a lot of PS1 programming work with my H2000 years ago and is how I got my effective start on MIPS assembly programming, though it soon fell out of use when I just made-do with debugging through emulators and have grown experienced enough with MIPS that I can literally 'debug' assembly code with my brain. I've even done on-and-off work on a much improved successor to my PSn00bDEBUG debugger, so once that's been paired with homebrewn hardware that provides a high speed communication link to a host system coupled with a retail console modified with 8MB of RAM, you'd effectively end up with a 'Home Edition' DTL-H2000.

But anyway, I'll look into merging your changes shortly. Been feeling really tired with anything programming related lately. I'm also going to include the "indev" directory from my redundant SVN repo containing work-in-progress libraries (including a failed attempt at figuring out the MDEC) as I'm considering retiring the SVN repo.

[Lameguy64]

Well the changes seem to have wrecked my ability to use the SDK I'm afraid. The CMakeList demands a too new of a version of CMake that is not yet available in MSys2 and most likely Debian stable as far as I'm aware, and when I try to override it I get a load of undefined references trying to compile tinyxml2, though this seems to be more of an issue with MSys2 MinGW w64 toolchain.

[spicyjpeg]

Well the changes seem to have wrecked my ability to use the SDK I'm afraid. The CMakeList demands a too new of a version of CMake that is not yet available in MSys2 and most likely Debian stable as far as I'm aware, and when I try to override it I get a load of undefined references trying to compile tinyxml2, though this seems to be more of an issue with MSys2 MinGW w64 toolchain.

There were a few critical bugs in the CMake scripts, which I fixed in #41. I also added a SKIP_TINYXML2 option to force the build script to search for an existing tinyxml2 installation instead of building it from source. That might help if building tinyxml2 fails for whatever reason.

I finally managed to get the SDK working on Windows by the way. MSys2 is no longer required (even though it is the easiest way to obtain Ninja and the MinGW compiler) so you can just grab the latest CMake Windows installer from here and run CMake from a regular command prompt, without having to fiddle with package managers and simulated Unix environments. As long as the toolchain is listed in PATH no additional setup is required to build the SDK.

The precompiled toolchains gave me lots of issues, however I was eventually able to build a fully functional copy of GCC 11.2.0 for Windows using MinGW on Ubuntu. Since it took me a lot of trial and error, I documented the whole process in TOOLCHAIN.md.

Is there any possibility of using Github-Actions to test on Linux, macOS and Windows? I've been able to get C++ stuff unit tested just fine on all three. The only tricky bit I can think of is the long amount of time and space required to build the toolchain, although I believe public open-source projects like these get a free CI credit allowance.

Automated unit testing would be difficult as I don't really know how to run libpsn00b tests in a PS1 emulator, or what kind of tests would make sense for small command-line tools. But GitHub Actions might still be useful for building binary releases now that I added full CPack support. It is already possible to build a Debian/RPM/NSIS package that includes the toolchain, libraries, tools and everything needed except CMake itself (see the updated INSTALL.md), so configuring Actions to run the build automatically and publish releases shouldn't be too hard.

[saxbophone]

Automated unit testing would be difficult as I don't really know how to run libpsn00b tests in a PS1 emulator, or what kind of tests would make sense for small command-line tools.

I forgot to mention that I meant for testing only that the build succeeds, not unit testing as you correctly mention that that doesn't make much sense for a command-line tool.

(I am working on something for very basic unit testing on the PS1, but that's another story!)

[saxbophone]

so configuring Actions to run the build automatically and publish releases shouldn't be too hard.

Agreed. I've barely used CPack myself, but I have some sample Github-actions scripts for doing cross-platform CMake builds which might be helpful as a good start

You might find this useful: https://github.com/saxbophone/CPP20-Cross-Platform-Template/blob/842e1d49fec20b14bf55fedc5ba76d705dfc216e/.github/workflows/build-release.yml

[Lameguy64]

Well the cmake updates have fixed a few things a bit. I have to compile using MSYS Makefiles as the generator because Ninja returns a command not found error from attempting to call the compiler as C:mipsel-unknown-elfbinmipsel-unknown-elf-gcc.exe for some reason, probably because the version of Ninja available in MSys2 might be buggy.

Building with tinyxml2 and mkpsxiso is busted for me as the former uses type definitions that don't exist in the version of MinGW-w64 provided in MSys2 and the latter complains about a missing libtinyxml2.dll, as tinyxml2 is usually linked statically.

[spicyjpeg]

I think the MSys version of Ninja expects Unix-style paths (/c/mipsel-unknown-elf/...) while the MinGW version (the only one supported by CMake on Windows) uses standard native Windows paths. It is confusing since both can be obtained from the MSys2 package manager, but their binaries end up in different directories which may or may not be listed in PATH. Anyway, there is another CMake-compatible standalone version of the Ninja binary available here, it just needs to be copied somewhere and added to PATH.

Building with tinyxml2 and mkpsxiso is busted for me as the former uses type definitions that don't exist in the version of MinGW-w64 provided in MSys2 and the latter complains about a missing libtinyxml2.dll, as tinyxml2 is usually linked statically.

The whole tinyxml2 statically vs. dynamically linked mess is something I initially ran into when I added MSVC support and never managed to fix afterwards. There is an option to force CMake to build static libraries instead of DLLs, which is also honored by the tinyxml2 CMake scripts, but in practice I never got it to work consistently across different compilers and environments.

The easiest fix would be to add tinyxml2 as a submodule within the PSn00bSDK/mkpsxiso repos and brutally compile its sources as part of the tools themselves, instead of bothering with building it as a separate library first. That would solve pretty much all issues with auto-downloads failing and libraries needing manual installation. It would also allow me to remove the ugly install(RUNTIME_DEPENDENCIES) commands and improve compatibility with older CMake versions.

I guess it's time for another round of PRs that will hopefully fix everything while also breaking entirely unrelated parts of the SDK.

I've even done on-and-off work on a much improved successor to my PSn00bDEBUG debugger, so once that's been paired with homebrewn hardware that provides a high speed communication link to a host system coupled with a retail console modified with 8MB of RAM, you'd effectively end up with a 'Home Edition' DTL-H2000.

I actually thought about something similar a few days ago. The communication link thing wouldn't be that complicated from a hardware standpoint. Cypress FX2LP microcontrollers are dirt cheap, can do USB DMA transfers from/to a 16-bit bus at up to 30 MHz and can be wired almost directly into the PS1's expansion port. Throw in some RAM, glue logic and a boot ROM and you'd have a plug-and-play solution for turning any PS1 into a debug unit. With some minor mods to the motherboard (i.e. hijacking /IRQ2 and /CS as the PSIO does) it could even become an inexpensive "tethered" drive emulator, allowing CD images to be booted instantly without having to transfer them to a separate device beforehand.

Of course the biggest hurdle would be sourcing whatever proprietary connector Sony used for the expansion port. AFAIK the PSIO guy got the connector custom made, which is obviously impossible for an open-source project with zero funding. Harvesting it from old cheat dongles would be too difficult for most people and unsustainable in the long term, and 3D printing isn't feasible either due to the accuracy required to produce a reliable connector. Perhaps some of the retrogaming shops that already manufacture custom video cables and PCBs can help here.

[Lameguy64]

Urgh, this is seriously starting to look like a mess just to build PSn00bSDK. This is why I was rather reluctant to accept your PR sooner because of breakages like this. I should have just tested your fork first before accepting it. On the subject of broken MSys64 seems to be busted as mkpsxiso throws a segfault during the directory tree phase when built with it, when I had to implement a rather simple fix to it, and this is with a fresh MSys2 install.

[spicyjpeg]

After almost a week of struggling I finally decided to get rid of all the auto-download and library finding stuff, which is clearly problematic on machines other than mine, and just include mkpsxiso and tinyxml2 as git submodules instead. The CMake script in the tinyxml2 repo is no longer used, instead the sources are simply pulled from the submodule directory and compiled as part of the tools. I regret not doing this earlier.

With these changes in place I tried compiling PSn00bSDK on a fresh Windows 10 VM with only MSys2 installed (GCC, CMake, Ninja and git installed through MSys2, which now ships CMake 3.21 by the way). I encountered no issues other the dynlink example randomly crashing on DuckStation (which seems to have something to do with compiler options). Building packages and installing them in another VM worked fine as well, and mkpsxiso did not crash when building the examples.

There isn't much I can test beyond this. I'm not submitting PRs yet, however I published all changes in the cmake branch so you can test them.

[saxbophone]

@spicyjpeg RE auto-downloading dependencies, have you heard of CPM? (https://github.com/cpm-cmake/CPM.cmake)

It works like a better version of find_package() and FetchContent() that can download dependencies through git and sets them up in the project tree like a CMake subproject.

[spicyjpeg]

@saxbophone: I started working on GitHub Actions CI now that PSn00bSDK is no longer broken (Lameguy64 merged my changes a few days ago, after I fixed some bugs that had been reported on the PSXDEV Discord server). The GCC toolchain is cached after being built: according to GitHub docs caches only expire after 7 days of inactivity, so this should prevent the toolchain from being recompiled unnecessarily most of the time.

Currently both the toolchain and SDK build successfully on both Windows and Linux. All builds are packaged up, bundled with the toolchain and uploaded as artifacts, so they can be downloaded from a commit's workflow run page (e.g. this one) and installed locally for testing. I also added some code to publish these builds as releases when a tag is created, even though PSn00bSDK is still far from being stable enough for a proper release.

I am going to open another PR once I finish working on a few additional examples, mostly related to audio streaming and support for the Konami System 573 (PS1-based arcade platform).

[saxbophone]

@spicyjpeg That's wonderful, how impressive that the toolchain builds for three OSes in such a short amount of time in your CI config!

[spicyjpeg]

I'd say this issue can be closed now that both stable CMake support and GitHub Actions CI are in place. I'm going to delete my actions branch as usual and create a fresh one for the things I'm currently working on (the psxspi library and experimental audio transcoding tools).