WINDOWS: Windows host OS support improvements in Xemu

lgblgblgb commented 1 year ago

As written in "Windows OS support improvements in Xemu" (discussion #384), the goal is to improve Windows host OS support, and source clean-up for better maintainability.

Non-US-ASCII characters in file names and paths

Traditionally, Windows has POSIX/C-like filesystem API in many cases, however for any file names having "special" (ie, non US-ASCII) characters, needs "wide character set support" which involves another set of functions (and their structures) called "wide functions" (and "narrow functions" for the former). Wide functions uses the wchar_t (w for wide) type, involving UTF-16 encoding.

However Xemu being cross-platform, using byte-oriented file names/paths, which means UTF-8 nowadays. SDL2 itself provides a filesystem abstraction layer which would hide the difference, however:

Xemu typically uses lots of code using "naked" POSIX/C functions (also native Windows ones and mingw provided ones)
SDL2 does not provide some of the functions I need (like directory handling, etc), and having SDL abstraction, mixed with own-converted things would create a real mess
The complexity created by Windows introduces cleanness and maintainability problems for non-Windows hosts as well, because of the shared code base

The situation is even worse, consider the case when the user has "special" character(s) in their user name. The preference directory is essential for Xemu to store its data files, and chosen by SDL to be portable. However it's usually contains the user's "home" directory and thus the user name as well, which would fail in this case, so Xemu is totally unusable in this case. It's really unfair especially with users, using non-latin script in their language which means this is guaranteed to happen.

Fortunately Microsoft in their infinite wisdom finally introduced UTF-8 support with the "standard" POSIX/C API somewhere within the Windows 10 line. It is a bit cumbersome to activate (needs an XML based manifest file linked from the resource definitions), it can be used at least. This however introduces the problem that support for older Windows OSes cannot be done using this technique. I think, it's fair enough though, I don't have better choice too much. This technique gives several benefits:

Really support all cases of non-US-ASCII characters in file names and paths in Xemu on Windows host OS
Allow some very nice clean-up in the Xemu code base, where half finished conversions and other tricks had to maintain which tried (badly!) to mitigate the original issue of Windows OSes

Security concerns and running as Administrator

Xemu can easily raise eyebrows of many average Windows users. Surely, it's a good thing that Microsoft would like to have "trusted" executables digitally signed, and so on. However a hobby + open-source project finds itself in a hard situation because of this. Downloading/installing Xemu involves dismissing Windows warning and selecting "Run anyway", also it's often the case that many virus protection software have false alarms on Xemu to be virus, trojan, or things like that. Since the initial suspect of users on Xemu (because of the need to run unsigned executable, warned by Windows itself), it deeps the problem even further, not so surprisingly.

So, in this situation at least I should try to mitigate any possible harm Xemu can do. Xemu itself surely not a trojan or anything like that, but it's also not a software which is written with strict security audits and such keeping in mind. Thus running it as Administrator can be dangerous. Xemu has checks on UNIX-like systems, if it tried to be run as root, I would do the same for Windows (Administrator privileges). See the discussion linked above for further info and possible problems.

Windows console handling

Xemu has some "black magic" code dealing with the "Windows console", which is - again - a very unique special Windows madness compared to other host OSes. Basically it "works", however there are problems:

The code is really ugly
Even I - the author - don't know how it works (I used several examples found on the Net, experimenting, modifying, etc)
Sometimes - for unknown reasons - other opened files are corrupted, see #262, though it's unclear if this issue is already really fixed or not

String format compile time warnings

This is - in theory - is more a "cosmetic" issue than something really bad for an average user (who uses only the provided executables), but still. Unfortunately the seemingly innocent task to print 32 and 64 bits long signed and unsigned values are quite hard when it should be portable. My approach so far:

1.Cast everything "problematic" one of the following: Uint64, Sint64. This is needed as things like long int and long long (etc) seems to mean different things on different architectures/OSes. Good example is the size_t type which can decode to different "basic" types on different cases.

Then, trying to use some formatting primitives which supposed to work in all cases, like:

printf("value=" PRINTF_U64, (Uint64)some_var_which_can_be_64bit_unsigned_on_some_OSes_at_least);

Surely, PRINTF_U64 must be defined, a random example (for real, the exact definition is based on host OS/architecture/etc): #define PRINTF_U64 "%llu"

This is the hard part, as it seems there is no universal formatting which works in every architectures/OSes and mingw versions :( So I get tons of compiler warnings all the time, which is ugly, needless to say.

Current test which does not work in every cases:

#define PRINTF_S64 "%" SDL_PRIs64
#define PRINTF_U64 "%" SDL_PRIu64

Very interestingly, even just with mingw, some version are happy some of them are not :( I really struggle here to find a solution which always works, not even mentioning that other than Windows (32 and 64 bit) cross compilation with different versions of mingw, the source should be compiled cleanly with gcc on Linux, and clang (branded as "gcc") on MacOS, just to stick with the minimum for now as host OS support spectrum.

It is OK to have those defines for PRINTF_U64 and PRINTF_S64 per host OS (ie, win32, win64, "UNIX" separately etc), but it seems even with this, I cannot find a solution when the source base compiles cleanly, even just win32 and win64 with different mingw as cross compilers sometimes drops tons of warnings, sometimes everything is happy. I feel really lost at this point :(

Upgrade mingw on Travis (with newer OS?) to build Windows binaries

On Travis, Ubuntu 20.04 LTS (code name "focal") is used with the in-repository mingw to build Windows binaries. However focal is about 3 years old for now, and the latest LTS (22.04, code name "jammy") is one year old as well. But it seems Travis has problems with using 22.04 at github-pages deploying process, what I use to distribute Xemu. Certainly it can be a solution in theory to stick with Ubuntu 20.04 and use a backported mingw but it's quite complicated and time consuming for a temporary VM like a Travis solution. However, I opened a support ticket at Travis. It seems not to be an openly available ticket I can link here, but at least my travis-community post can be linked: https://travis-ci.community/t/ubuntu-jammy-installing-deploy-dependencies-failed-to-deploy/13947

lgblgblgb commented 1 year ago

Note, that the "check running as admin" things was part of the commit on the UTF8 FS stuff in commit d7d3799a2a7cec51c1c277e45d4c074b91591531

lgblgblgb commented 1 year ago

It turned out some users really want to avoid to upgrade to Windows 10. Before windows 10, the new UTF8 drops warnings. So I introduced registry keys to mute that warning so they can still use Xemu without too much "noise" all the time. Note, that it does not cure the core problem, that any files/paths with non US-ASCII characters won't work.

Registry keys:

HKEY_LOCAL_MACHINE\Software\X-Emulators\AllowAdministratorRun set to DWORD 1 not to give admin warnings
HKEY_LOCAL_MACHINE\Software\X-Emulators\NoUTF8Warning set to DWORD 1 not to give UTF8 warnings

It's in commit: 9a1a079e14d6e079054e19967b41d830e960d84b

lgblgblgb / xemu