Windows support?

[rasky]

I've received a few PRs by @networkfusion (opened in my fork) to compile libdragon tools with MSVC: https://github.com/rasky/libdragon/pulls Before reviewing / merging them, I'd like to agree on a general plan of how Windows support in libdragon would look like, so that it's easier to drive contributions in the right direction. Moreover, I'm resisting using POSIX/GNU specific extensions in tools

I'm not really up to date with Windows development nowadays, so forgive me in advance. Focusing on tools only for now, my understanding is that there are three main avenues:

1) Use WSL (Windows Subsystem for Linux). This is (I simplify) a Linux VM within Windows. My understanding is that libdragon should just work as-is on WSL, no changes required. 2) Use MinGW/MSYS (with GNU make as build system). This would require very little modifications to the source code of tools (possibly none?) and it would still allow to use things like getopt or asprintf. It might require some changes to Makefiles (different extensions, etc.). It might even be possible to cross-compile the tools for Windows target from Linux CI. 3) Use MSVC. This is where I'm more confused about. I'm sure this would require some modification to the source code of tools, it would prevent using POSIX-ism / GNU-ism. It would also require changing the build system as MSVC doesn't ship with a GNU make-compatible tool, so we would have to switch to CMake or similar I guess.

As for compiling the actual game/ROM, I would say that we should stick to a cross-compiled GCC / newline / binutils anyway, so on this regard the effort would probably be setting up some script to be able to cross-compile a Windows version of the MIPS GCC toolchain, and then package everything for distribution.

Any comment on the above from some Windows developer?

[networkfusion]

Hey @rasky .

1. Yes WSL will work fine for build, but will not work with deploy tools due to no support for USB (in WSL V2 which is basically a docker container).

2. I have never (personally) liked the dependency on MSYS or CYGWIN for windows, it adds a big barrier to new users and "should" not be required...

3. It could use Makefiles with careful support for NMake (https://docs.microsoft.com/en-us/cpp/build/reference/nmake-reference?view=msvc-160) to keep the existing MakeFiles (happy). Although there will still be some issues with regards to specific gcc extensions which could easily be handled by an ifdef e.g. If necessary, a dependency on Ninja is an option (it might improve build speed on linux as well). It also needs to take into account environment paths.

[networkfusion]

linking with #30 the user got fedup and closed it...

[meeq]

https://github.com/DragonMinded/libdragon/pull/100 was the relevant PR for #30 but it was never merged. There was a missingdirent.h when compiling for MSVC.

[networkfusion]

unfortuately that is a postix lib... but does work if you copy in something like https://raw.githubusercontent.com/tronkko/dirent/master/include/dirent.h Further information: https://www.programmersought.com/article/14763314327/

[anacierdem]

My take on this one as a Windows user at home (this is where I develop for N64);

1. I once built a toolchain on wsl and it was working as expected (not libdragon though). WSL 1 supports usb devices but with WSL 2 they have changed the infrastucture (based on Hyper-V) and usb support does not work indeed. Also actually docker runs on wsl2 on Windows and is also subject to its limitations. On the other hand it is possible to invoke non-wsl executables from a wsl shell with a Windows interop With that setup it should be possible to invoke a Windows executable from within wsl.
2. I never liked it either although it works pretty well once configured properly.
3. I have hardly used MSVC but IMO, we should concentrate our efforts towards other things rather than maintaining a different set of tools. Do not forget about maintaining tests and other stuff for both etc.

There is also a fourth option, which I still prefer using to this day and that is dockerizing the toolchain. Once we had the docker container in this repository but later I have decided to separate it away as it was causing too much overhead (I did not have merge access back then). I suspect trying to do a similar thing by maintaining different build processes and compatbility will increase the efforts required here. As the project folder is mounted on the container it is possible to just use the native USB tools and docker commands for anything else including the tools.

There are a few problems I have faced with the docker setup;

• Line endings can be an issue if you want to use shell scripts on a docker mount if your defaults ar CRLF. This is just a minor annoyance once you set up them specifically for your docker based projects.
• There is no platform independent way of accessing the host machine (mac and windows have host.docker.internal - see here, but I'm not sure if linux have something similar that we can set once and use). This is something that I want to have b/c I was planning to also have the cross-compiled gdb on the docker machine as well which will connect to a simple socket server running on the host thus talking to the flash cart.

Apart from these caveats I really like this approach because it takes a few minutes to make everything work as it should whatever your working environment is. Even if you completely forget even what to install/build etc. Still I'm a little hesitant on introducing these solutions into the main libdragon repository. WSL can be a good candidate as well but docker has also the advantag of being out of the box compatible with macOS. If anyone had built the toolchain on mac I'd be happy to hear their experience as well. AFAIR it wants the binaries to be signed. You can self sign them but it is a chore and there are potentially other third party dependencies that are not installed (or maybe built) by default. In any case someone has to maintain those solutions (including docker) so we should aim for the least resistance path, which is docker for me ATM, which I already happen to maintain for myself. @rasky suggested removing the npm dependency by compiling native versions of the helper, which would be a good improvement.

[rasky]

Thanks. I'm using docker on MacOS for libdragon development and I'm very happy about it. If that can be made to work on Windows, maybe it's a good way to maintain just one platform.

I agree that the NPM wrapper is a nuisance on Windows. We could work on a new tool that's natively compiled, that'd make the docker way more palatable for Windows users I guess.

[fraser125]

I haven't been able to do a full build of libdragon or a ROM yet so my experience is "incomplete". The issue that I had was the use of 'mkdir' in the makefile, the equivalent on Windows doesn't support making nested directories, so it errors out.

My Environment: Compiler: n64chain version of GCC for Windows in a Windows Command Prompt Editor: Notepad++ I'd rather use VSCode or similar but that is probably easy with the right plugins, once the makefile works. No WSL,, No Docker, No MinGW/MSYS (which is how I'd like to keep it)

When I need GCC for Windows EXE's I download from here: equation.com

Personal Opinion: MinGW/MSYS and Docker are hacky ways to run Linux inside of Windows. If I wanted to learn Linux, I'll learn Linux in a real system or a dedicated VM. When I just want to compile a ROM, I shouldn't have to learn Linux.

[networkfusion]

I agree, docker is a very good option (it does work well for me in a CI environment)... but... against that it also adds a mainainance burden on users (given that you have to clean up the containers regularly)... Also, I personally dont have space for docker or WSL...

The tools should not be a maintainance burden (or at least a very limited one) if considered correctly.

[anacierdem]

@fraser125 is learning make (and a few well known commands) is equivalent to learning linux as well? If not, with a docker container and an appropriate wrapper, you can build a rom with no additional linux specific knowledge IMO. Why do you think those are hacky solutions? Especially docker is used in production to a great extent. A considerable amount of online services are utilizing containerization. I use vscode to develop n64 as well.

[rasky]

Personal Opinion: MinGW/MSYS and Docker are hacky ways to run Linux inside of Windows. If I wanted to learn Linux, I'll learn Linux in a real system or a dedicated VM. When I just want to compile a ROM, I shouldn't have to learn Linux.

Docker is many things. One thing it is is a tool to run prepackaged software in pre-built images using Linux just as a common runtime interface.

The final experience would be that you install Docker on your system, then download a tool called libdragon.exe, cd into a directory containing your project and run "libdragon make". There is no Linux knowledge required.

The only point where this abstraction leaks is that if you try to do complex things in your Makefile, you need to use bash for that. But if you don't like that, you can still write your own build.cmd script in Batch or PowerShell that invokes "libdragon make" as one of the steps.

I think this very well crosses the dot of "no Linux knowledge".

[networkfusion]

As long as you are competent with using commands like docker image prune and/or the toolchain handles it e.g. docker container prune https://docs.docker.com/config/pruning/ (as well as the issues mentioned about diskspace being an issue...

IMHO, I would still really like to see the ability to build it (at this moment). If it becomes a maintainance burden, it can be marked as deprecated, with a message "go back to X point in time where it was supported, and add the fixes!"

[anacierdem]

Docker can consume considerable disk space though I admit that. Still the benefits might overweigh the cons and we can abstract away container maintenance. There is still nothing wrong with maintaining lightweight executables somewhere else that can be downloaded and used but doing and maintaining it here is a little bit too much IMO. I think docker should not enter this repo and should be maintained elsewhere as well. At the end of the day, it boils down to the tools then, should we support MSVC out of the box so that someone can compile it for Windows and maintain it?

[fraser125]

It sounds easy when you do it all the time, for a person new to N64 development it's another frustration and the possible loss of a new contributor. The last time I was trying to setup a C development environment for N64. I finally found it easier to learn MIPS assembly language from scratch.

How about looking at the rest of my post where I got most of the environment working without ANY major changes to the codebase.

[rasky]

Pruning images is only required to reclaim disk space for users that heavily build new containers. In case of libdragon users, they would use a single image that does not inflate over time while building ROMs, and we can make sure the libdragon.exe tool always deletes the old image when it updates it to a new one. So really there should not be so much space required.

[anacierdem]

@fraser125 how do you compile the tools with little change? If that is possible, I think we should consider making those changes here.

[fraser125]

When I need GCC for Windows EXE's I download from here: http://www.equation.com/servlet/equation.cmd?fa=fortran

It would be nice have a Tools Release for for Windows so the user doesn't have to download the GCC toolchain for Windows.

[anacierdem]

To be able to build the tools for Windows (at least for MSVC) we need changes here for compatibility. Still non MSVC builds can be done and uploaded somewhere, there is nothing preventing that.

[networkfusion]

One other point to consider... generally when building a ROM, you have done it against a specific version of libdragon (given changes to the lib are likely going to break things). If docker is the main point of entry, that needs to be carefully considered (especially in a CI environment)... but that is probably a different conversation... and probably just as valid in all environments, just more easily attribuded to the user doing something...

[networkfusion]

To be able to build the tools for Windows (at least for MSVC) we need changes here for compatibility. Still non MSVC builds can be done and uploaded somewhere, there is nothing preventing that.

Only if the tools still rely on GCC extensions... Most of the issues are due to -Wall not being on. And those that do, should be carefully considered... (with the exception of dirent.h for a single tool.

[anacierdem]

@rasky on Windows docker can be configured to use wsl2 and that results in an ever growing vhd image even if you clean the docker images etc. I'm not sure if this was addressed on docker yet or is there a similar issue for a macOS/linux host. So disk usage might be a real concern indeed. It should get fixed/improved at some point though.

[networkfusion]

Prune is very necessary with me (on the W10 WSL2 computer that I use CrashOverrides lib on!)

[anacierdem]

I am OK to reduce GCC dependency for tools and improving compatibility without introducing too many define guards. For the library itself, GCC extensions are necessary IMO.

[networkfusion]

With the tools, these are the current issues, rather than warnings (I remember) and possible solutions:

1. The use of libpng (either change to using something like stb_image.h or warn that MVSC users will need to find a windows compiled one).
2. Byteswap flags using GCC extensions (really should be independent, but it is maintainable to just abstract it).
3. use of dirent.h (really should be made independent, but if not just warn that the file needs to be found and added to the appropriate location).
4. packing a struct using GCC extensions (really should be independent, but it is maintainable to just abstract it).

Most of the -Wall errors/warnings are related to incorrect integer types or use of non safe functions, which should be fixed regardless of environment...

Even with all of this... a CI that just attempts to compile them and warn the users (against a warning that it is an unsupported environment) would help the community to keep it working?!

[rasky]

I am OK to reduce GCC dependency for tools and improving compatibility without introducing too many define guards

I'm afraid the two goals are at conflict, generally speaking. For instance, today's review by Meeq on my PR suggested to use asprintf to clean up the code, but that is not available on MSVC. I also in the past resisted the temptation to use getopt_long in tools to greatly simplify option parsing just because a possible future MSVC port might get into trouble.

It's unfortunate that MSVC is stuck in the '90s with its C compatibility, but I'm wondering if it makes sense to restrain our capability to develop those tools, when 99% of libdragon users just need to be able to run n64tool.exe. So I can't see why using MinGW to compile n64tool.exe is a problem, given that end users won't be required to install MinGW, just to use it.

[networkfusion]

I would not say MSVC is stuck in the 90's persay, but more GCC is bringing the C language into the 21st century ( 17 vs https://docs.microsoft.com/en-us/cpp/build/reference/std-specify-language-standard-version?view=msvc-160... C2x...) portability wise == https://stackoverflow.com/questions/40159892/using-asprintf-on-windows).. however I am definitly not against MinGW to get passed it...

[meeq]

I am also in favor of making Windows builds as uncomplicated as possible. If we have CI doing builds, we can tell beginners to just use the binaries. Advanced users who want to do a build can figure out how to get MinGW installed.

[anacierdem]

I dont know the state of MSVC vs GCC if you think it will be difficult @rasky its not something I would want either. To keep builds clean we will also have to maintain its build pipeline here, which I think is not feasible in the long run. It will rot and get removed at some point to make the build pass. Even if I owned this repository that wouldn't be something I can promise to protect. Even with this relaxed setup PRs can take days to merge.

Dont get me wrong I'm 100% into adding those pipelines and checks making this a perfect environment but realistically I don't think we can make it. Still if we decide to commit and establish strict rules to merge something, like enforcing tests, cross platform support for tools, automated emulator tests, code style restrictions etc. I would not object although it might harm the library more than it improves it ATM. We can re-consider this after we decide we got enough traction though. What do you think?

[rasky]

I think we're converging on:

• Tools: compile with GCC for Windows to have zero hassle with ifdefs. Build can be done on CI. Distribute as binaries for Windows users.
• Toolchain + libdragon: try the docker option, with a new driver that's compiled native on Windows (libdragon.exe) so that no Docker/Linux knowledge is required, but at the same time we can reuse the build system based on make+bash. Try keeping images small (eg: alpine).

[DragonMinded]

Compiling with or under MSVC has never been a priority IMO. Its a nice ecosystem if you are developing Windows software but outside of that? Nah. The original toolchain for windows release was compiled using msys in a canadian cross compiler (compile on platform A, or linux, to generate executables on platform B, or windows, that targets platform C, or mips) setup and also statically linked so there was no dependency on an installed version. That ran on windows, used standard make and allowed the same code to work anywhere.

Docker is definitely an option, but the original long-outdated windows release at https://dragonminded.com/n64dev/ was almost standalone (just needed make for windows) and there's no problem bundling that for a one-stop download. We could even build that using CI so that there's just a download for windows users that doesn't need to be installed. This is how I've always done windows development on various homebrew platforms and it's been nice having no cygwin/msys dependencies and being able to extract and run out of the box.

Maybe this isn't how development gets done anymore though?

[anacierdem]

I think thats an option too... We will probably not support MSVC then. The next important question is in my opinion; should we maintain the build pipelines (whether it is docker or msys/mingw) here on this repository or not?

[rasky]

I think it makes sense to have them here as GitHub pipelines or similar CI, as there's easier too trigger for every new change.

I'm not opposed in building the toolchain as canadian cross. My only issue with that is that we should probably also build GNU make and then make sure that the default example Makefiles don't use any bashism (so not even a if), as otherwise they won't work on a native Windows terminal; I guess the CI can help with this as well.

Moreover, this solution isn't easy to use for Mac where distributing native binaries is hard nowadays.

I'm experimenting with a natively-compiled CLI tool: https://github.com/rasky/libdragon-cli that builds upon the experience of the NPM tool by @anacierdem. I think if we can make the docker flow work without people even knowing we are using docker, it might solve the cross platform issue while constraining us less while developing.

[anacierdem]

Also remember that we depend on dd which we may also need to distribute if we go that route. Should we also include the docker wrapper here as well then? b/c we will also need to make sure it is built and published somewhere. Then it might be a good idea to have an additional repository collaborator as well so that we don't get blocked if we need changes.

[meeq]

It looks like we could replace usage of dd for byte-swapping with objcopy: https://stackoverflow.com/a/19288235

[werkn]

Just throwing it out there but I did a full setup for Windows with Docker here: N64 Windows Setup. It'll give a good sense of the setup under Windows at the moment. In my opinion the biggest frustration with setup on Windows is the lack of a good package manager for automating installs (ie: apt, brew). I've reached for Chocolatey here to do automatic setup but its still quite cumbersome.

tldr In my opinion the suggestion to:

Toolchain + libdragon: try the docker option, with a new driver that's compiled native on Windows (libdragon.exe), so that no Docker/Linux knowledge is required

Would greatly simplify setup. I've setup Libdragon for friends across MacOS, Linux and Windows and I think if it could be done as a one-click installer that would be ideal and remove what I think is the biggest barrier to entry into N64 homebrew using Libdragon and that is setup headaches.

[werkn]

Also should note that I do all my development inside the Docker container using VSCode Remote extension which is very stable (despite I think being in early access preview). This means compat. with existing Libdragon setup/scripts and its just gcc with makefiles and no requirement for MSVC/Cygwin.

[anacierdem]

@werkn There is already a wrapper for docker which should be much easier than your experience in the blogpost. It uses NPM and docker desktop, both of which having easy to reach installers on Windows. With a standalone executable, we will be able to get rid of NPM but in turn we will be loosing the ability to automatically register it to the path and dependency installation capabilities (we may achieve it via docker though) + it may also require git. @rasky is working on a version of it and I highly suggest trying it out when it is stable. We may try to provide an installer executable as well.

[rasky]

The new version is here: [REDACTED]

Here you should find a Windows binary: [REDACTED] (Everything is super WIP)

As mentioned by @anacierdem, this version requires manual installation on PATH and a separate installation of Git for Windows.

The first requirement could be dropped by wrapping the binary into a simple installer like those made by InnoSetup or WiX.

The second requirement is coming from a different design choice I'm experimenting with. Basically, @anacierdem docker images contain the full toolchain plus a specific version of libdragon. Whenever somebody commits on libdragon, the container image must be updated (it can be automated with CI). This also means that users must update Docker images to update libdragon.

In the new tool, the approach I'm thinking is using a docking container just for the toolchain which changes very rarely, and then vendoring libdragon within the ROM project. The tool supports vendoring via git submodule and git subtree, and hence the git requirement; this is done when the user runs "libdragon init" to initialize a new ROM project (which is also nice to get in one command a running main plus a working Makefile). When the user runs "libdragon update" the tool then updates libdragon by using the correct git incantation.

I think that in general the ecosystem will be better served with a vendoring approach, given how we already manages to break the API a few times (see how Flappy Bird was broken for many months). We don't want new developers to have troubles to recompile games released years or months before just because they don't know the correct libdragon/toolchain to use.

Another good effect of vendoring libdragon is that libdragon init would create a Makefile that builds both the vendored libdragon and the ROM project. This means that users can easily patch libdragon if they need, without changing their workflow.

A third good effect is that this approach rarely updates the docker image, which means that there will be fewer images in general and less disk space used (this was one of the concerns mentioned in this issue).

Now that I think of it, we might even drop the Git for Windows requirement if the tool supported a mode to vendor libdragon by downloading zip archives via HTTP and recording the version in some dot file. But is Git for Windows really such a hassle?

[anacierdem]

I don't think git is a hassle but I don't think NPM is one either. I'm totally ok with rebuilding libdragon locally rather than including it in the container as well. It is a thing we can achieve with npm as well, we already have versioning. It will be a decision of the community in the end 😊

[rasky]

Updates to this issue.

After some discussions with @anacierdem, we decided to improve the existing libdragon-cli tool rather than starting a new one, borrowing the new design ideas. @anacierdem did the development and the tool is now available:

https://github.com/anacierdem/libdragon-docker/releases/tag/v10.2.1

You can use this tool to do extremely easy and fast lib dragon development, using the Docker toolchain. Download a prebuilt binary and just run "libdragon init" to end up with a skeleton project, and "libdragon make" to compile it. It requires Docker (obviously) and command-line git in the PATH (Git for Windows for instance).

Using this CLI tool, you can run libdragon tools simply with "libdragon exec n64tool " or similar. To clarify, this is how I do all my development on macOS and it works very well, transparently and very fast. Nonetheless, I've also mailed a PR to do a native Windows build of the tools, using MinGW as agreed here, because there surely are situations when it's important to run the tools without having the whole toolchain configured (eg: asset preparation made by a non developer).

If somebody wants to do a full native Windows build of the toolchain, the starting point is the CI and tools/build-toolchain.sh that would have to be modified to support doing a "canadian cross" (building a Windows->MIPS compiler on Linux). It shouldn't be impossible but it requires some effort. If a Windows toolchain arises and it works flawlessly, we might evaluate integrating it with the CLI tool as alternative to Docker. The tool interface would stay the same ("libdragon init", etc.) but it would download and use the toolchain rather than download and using a Docker image.

This implements most of the plan described here: https://github.com/DragonMinded/libdragon/issues/161#issuecomment-892654312

[anacierdem]

One thing to note: git is not a hard dependency for the cli if the user does not care about source control, but it is recommended nevertheless.