Introduce MONO_IOMAP-like compatibility for Unix platforms

[flibitijibibo]

Prior to 6.0, Mono had a feature known as IOMAP, which allowed users to enable an extra code path that would check for filesystem incompatibilities commonly found when running Windows .NET applications on Unix platforms (Linux in particular). With the introduction of CoreFX to Mono's System.IO implementation, the feature no longer works and the Mono team deemed the feature simultaneously "removed" and "deprecated", though this almost appears to be in name only as 100% of the original code is still present and the feature is still listed in the README as of writing (plus a little bonus issue that we'll get to at the end of this report).

There are lots of reasons that developers need IOMAP, many of them centered around backward compatibility and preservation, but another important reason to have this is for current applications with Windows-centric content pipelines as well as user-generated content. For example, a game built on Windows may have files referenced as plain text paths in their content blobs, and if the cases don't match, Unix builds would need to rebuild all of the content with case sensitivity fixes every time they make a content change, which could mean going as far as porting your content build system to a platform that has a case-sensitive filesystem, if that's even possible to begin with (keeping in mind that this only works if you don't have issues with data mismatches for cross-platform online interactions, unless you take the Unix content and drag it back to the Windows build). If this all sounds annoying and fragile, that's because it is. This is true for both developers and users who make custom content for a game.

As for an implementation, the feature is not as large as you might think; the extra code paths were contained almost entirely in w32file-unix.c, and as it turns out, this file does almost the exact same thing as pal_io.c in System.Native! Plugging the old code into the new code was surprisingly easy to do:

https://github.com/flibitijibibo/corefx/commit/2902d5856985830db1e23f78ed8b83d4c502a261

Almost 100% of the code is a direct, unmodified copypaste of the original, with the exception of exactly one function that was just a wrapper which served no purpose on non-Win32 platforms. In my local testing, it works great! There are some caveats, however:

• This is obviously a nasty copypaste and should probably be cleaned up to match the CoreFX style. A real implementation of this in upstream CoreFX would function the same, but would not do all the sloppy header inlining I did, nor would it do the weird things Mono is doing in their find_file implementation (use of glib is the obvious one).
• Not every pal_io function has a mono-io-portability equivalent. Implementing these should be trivial, but for this proposal I wanted to emphasize how much was already done with absolutely no new code written whatsoever.
• Mono, as of the current master revision, uses both the old MonoIO code and the new Interop.Sys code simultaneously. I originally wanted to propose this as a totally new DOTNET_IOMAP feature, but in testing this it turns out you need the variables to match so that it's as consistent as it was before Mono 6. (So yes, in vanilla Mono it's actually possible to have partial MONO_IOMAP support that happens to work if you get lucky.)

[ghost]

Tagging subscribers to this area: @jozkee Notify danmosemsft if you want to be subscribed.

[danmoseley]

Is the scenario entirely a developer machine scenario? It seems potentially a bit risky, to switch on this mode and then have your app accidentally delete a file that differed in casing.

@marek-safar @akoeplinger do you have a read on how widely this was used? I see some discussion here https://github.com/mono/mono/issues/9300#issuecomment-406614665

[flibitijibibo]

This feature is currently used in the retail versions of Terraria, Wizorb, A Virus Named TOM, and Unexplored. That’s my catalog of examples based on the FNA game list at least. It is also usable for mod support in other games like Celeste (Everest tries to enforce case sensitivity though), Stardew Valley, and Terraria (tModLoader for example). There are also tools like Rhys which need it to run Windows XNA games on Linux via FNA’s XNA ABI compatibility files.

[danmoseley]

Any idea whether Unity includes it in their Mono builds? Just since you mention games.

[flibitijibibo]

It appears it is available in their runtime, an example of it being used to fix a plugin: https://forum.unity.com/threads/darkrift-server-plugins-on-linux-machines-errors-and-how-to-fix-it.444328/

EDIT: Seems this is a separate companion application rather than a Unity export, but still... the Google results go on forever.

[marek-safar]

@danmosemsft I believe the primary use-case was for non-development scenarios. The use case is to run libraries which have bugs in the casing on filesystems which distinguish between different cases.

[rfht]

Without MONO_IOMAP, it can be impossible to run assemblies generated on Windows on a Unix machine if the developer didn't pay close attention to match cases in filenames and the paths in the assemblies. Of course, it would be nice to have the developers pay attention to it, but it's easy to not even notice the issue. Without a replacement for MONO_IOMAP=all as an option, cross-platform compatibility will be impaired in my opinion.

It seems potentially a bit risky, to switch on this mode and then have your app accidentally delete a file that differed in casing.

The switch should only be used if necessary with appropriate warnings IMO.

[danmoseley]

@jkotas has this come up before?

[jkotas]

This is the first time I am hearing about this. It looks like a partial Windows emulator. I do not think it is something we would want to include in the product by default.

it can be impossible to run assemblies generated on Windows on a Unix machine if the developer didn't pay close attention

There are many other reasons why code written for Windows may not work on Unix.

[flibitijibibo]

IOMAP is not something that Windows users and developers will be intimately familiar with, but on other platforms it's essential for everyday use. It's enough to where I would 100% have to make a directly competing repository to ensure that C# development is still possible on Linux should this not be integrated in any form.

By default the feature is not enabled at runtime, and the build is configurable in such a way that IOMAP can be completely disabled and removed from the final binary via --disable-portability. If the default behavior in .NET's official runtime was to disable this, I would be entirely okay with that, I have to self-build everything for MonoKickstart anyhow so one added flag will be no problem for us.

[TheSpydog]

There are many other reasons why code written for Windows may not work on Unix.

Isn't one of the main selling points of .NET that the same assembly can work across multiple platforms? The use case mentioned in this thread (cross platform games using the FNA framework) is a great example of what that can look like.

FNA is entirely built around the idea of single assembly portability -- the same .exe used on Windows also works on macOS and Linux. All platform-specific code is handled by a small set of native libraries. In fact, it's not just limited to desktop platforms; FNA also supports iOS, Xbox One, and Nintendo Switch, all from the exact same C# project with no platform-specific branches.

For situations such as these, where portability has been carefully considered, I see no reason why having an optional environment variable to fix up hardcoded path strings would be harmful in any way. Especially for preservation projects like Rhys, since it's quite possible the source code for these games has been lost to time, making a "proper" fix of the file paths impossible.

[jkotas]

the same assembly can work across multiple platforms?

can != will. You can make the same assembly work accross multiple platforms if you follow the rules for writing cross-platform code, like not assuming a case-insensitive file system.

[TheSpydog]

But they do work already, just with a slightly older version of Mono. Perhaps you missed the last part of my comment, about how this would especially benefit preservation projects. For games that were originally made targeting XNA and Windows only, going back and fixing up the source isn't even possible sometimes. Does that mean they should be abandoned entirely, even if they're already functional on Linux with the use of this environment variable?

[flibitijibibo]

can != can't, if we aren't given the resources to fix a bug then products will just be broken forever and the people who made the choice to take those resources away will be the ones to blame. Again, I would like to make it as clear as possible that this feature is going to get written either way, it just seems impractical to maintain an entire runtime to patch in code that an official runtime already had a decade ago.

[cybik]

I believe that what's being said is that the feature which is requested here, would be a good "opt-in" addition to the framework that should also never be active by default. I don't think that anyone here would want this feature active at all times for all .net payloads.

While I do understand the argument that "good programming practices are always the best" (and believe me, I advocate for it), at the same time it is unfortunately wrong to assume that everyone would follow them, or would have either the inclination or the time to do so if their original target (and the platform on which they did their good work) did not have such restrictions.

Also, as @flibitijibibo mentioned, sometimes they can't even get to the code, which directly hampers any efforts to fix the casing issues.

[danmoseley]

If we wanted to do something here (not suggesting we do) would another possible approach to expose a hook such that a 3rd party library could be discovered that contained the "mangling" code. That still has significant design/build cost but there would be no maintenance of the "mangling".

[flibitijibibo]

I'd be up for evaluating that as an alternative - whatever lets me plug into pal_io.c+pal_time.c will likely be good enough for me (it all gets statically linked at the end anyway).

[jkotas]

a hook such that a 3rd party library could be discovered that contained the "mangling" code.

That is not that different from telling people who really wants this to build their own libSystem.Native.so.

[cybik]

That is not that different from telling people who really wants this to build their own libSystem.Native.so.

Maybe so, but I'd argue that rewriting/bundling a whole libSystem.Native.so just to introduce filepath case auto-mangling/demangling is a smidge overkill.

[danmoseley]

I'd recommend that you build and share your own as suggested above. This kind of "heuristic" code has a maintenance and testing burden and a hook would require significant design and implementation (eg., to discover the library, to avoid perf issues) and some maintenance as well. It seems that sharing your own binary would work well enough for this scenario.

[0x0ade]

This kind of "heuristic" code has a maintenance and testing burden

Isn't this burden already mostly lifted by the fact that it was maintained and has existed as part of mono? Please correct me if I'm wrong, but as far as I can tell, the point of this issue is to "reintroduce" MONO_IOMAP into the codepath that is now shared between core and mono, right?

[flibitijibibo]

I think we have the resources to test (and write tests, assuming Mono's unit tests aren't sufficient), I'm not so sure about maintaining a patchset repo. MonoKickstart will always be okay since that's for static bundles, but distributions with system runtimes will struggle with this, and telling users to shove an arbitrary file from my public FTP into /usr/lib64/ seems a little... unsavory.

[TheSpydog]

@danmosemsft @jkotas I'm a bit confused what the resolution was here, if there was any. In an attempt to clarify this issue, let me ask the following question:

If a community member (read: likely me and/or @flibitijibibo) fully implemented a MONO_IOMAP-equivalent feature that was... 1) Disabled by default at build time, guarded by a compile flag 2) Disabled by default at runtime, and only enabled via an environment variable 3) Compliant with CoreFX style 4) Compatible with all relevant pal_io functions

...would that be a candidate for inclusion in the product? If not, why? What else would be necessary for the inclusion of this feature?

I totally understand not wanting to increase the maintenance burden of the .NET runtime with niche features, but this is a unique situation in that A) it's of vital importance for preservation projects, B) it was already present in prior versions of the Mono runtime, and C) it doesn't have any feasible workarounds, short of forking the whole runtime.

[jkotas]

I am sorry - I do not see this as a good feature to have in this repo, even if the code is not building by default.

What else would be necessary for the inclusion of this feature?

Convince me or some other maintainer of dotnet/runtime that it is a good feature to have. E.g. Do other runtimes (Java, Go, Rust, ...) have a feature like this?

It is a non-goal for us to keep all 10 or 20 year apps working on latest .NET Core versions without any changes. There are number of features present in .NET Framework and Mono runtimes that are stripped in .NET Core. Even between major .NET Core versions, we do occasionally make breaking changes to move the .NET platform forward to stay relevant that may require adjustments of the user code. To run old code without any changes, it is best to use the old runtime that the code was developed against.

[cybik]

If I recall correctly, and thanks to some work I did for $dayjob last week, with Gradle & Java, it is possible to apply interesting compiled classpath transforms at build time. One could presumably write a File.class intercept of some sort that would be comprehensive enough to replicate an automatic different-case resolution behaviour close to what's being suggested here. [edit: There might also be some Kotlin shenaniganery making such extensions to FileIO possible.]

But that would apply to source-accessible approaches, which may not be available in the case of preservation-type projects that may have lost access to such sources.

For the sake of argument though: going down the classmod path fully and altering assembled executable Java payloads via such transforms would also mean altering the "executable" (JAR files) pre-run (or creating an alternate executable just before runtime, or using class modification packages with a java "launcher" of sorts), which as a build engineer I am very much not fond of - altering runtimes / build workspaces is a gigantic /redacted/ no, as one can't guarantee anything when modifications like that start happening.

Honestly, if I had to say, the interest of the IOMAP feature in discussion is that the mangling/demangling happens at the framework level, and the additional components in the library can focus on their own features. Altering the "origin" executable is literally the antithesis of the objective here, and in some cases could actually have less desirable implications beyond the scope of simple programming/adaptation.

[jwaffe75]

Hello,

I understand your point of view on this, and you don't have to support every bizarre hack out there, but I wish you would reconsider your decision on this.

Paths:

• Yes, it's true that Path.Combine and other methods are without a doubt superior to concatenation with \. That is the right thing to do, and I always did it and never shut up about it for years (I was a riot at parties).
• But, so far every single Windows C# project I've inherited or looked at, without fail, has been stuffed with string concatenations and \ by previous devs.
• Sadly people just don't read the docs and design software with this in mind, instead they seem to just copy/paste paths from Windows Explorer, it "just works", they move on.

Linux/Windows compatibility:

There are many other reasons why code written for Windows may not work on Unix.

• Yes, it's true that making an emulator to work around \ paths will not automatically make every Windows application run on Linux
• Sadly, Windows is tolerant of \ or /, even weird stuff like C:\/\/\/\/\/\/\\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/ but Linux supports only /
• There are programs that run "well enough" on Linux, but are broken by the \ problem

Respectfully I have to disagree that it's as simple as "programs that expect \ are broken anyway".

Backwards compatibility with older versions of Mono

• It is a nontrivial amount of work to fix every single occurrence of \\ and replace it with something more portable.
• Mono used to support this, now it suddenly doesn't.
• If you were a small developer, suddenly people can't use your software. What would you be more likely to do? A) put in a lot of time to support a new OS, or B) ignore the problem? A bunch of projects just lost "unofficial hacky Linux support".
• If you were a user that got a "hack" working on Mono < 6 that no longer works now, what would you be more likely to do? A) volunteer to edit large amounts of code, or B) stick with Mono < 6?
• I realize that there's problems with emulation, that's why it should be opt-in, but I though that was covered already with the env var. If a user were trying to run something that was designed to support .NET Core from day 1, couldn't they just not set the var and everything would be fine?
• People who use Wine are used to hacks and emulation, amazing things can be done, even very old Direct2D programs can work. We're OK with compromises and half baked substitutions if the alternative is nothing at all.
• I think it might be important to realize the cultural differences here, people trying to run high reliability software vs people just trying to get some silly game working. I've been in both situations. I get that it makes sense to prioritize the former, but I'm not sure why this feature was removed completely.
• I'm a big advocate for clean code, but if you take a slightly broader viewpoint of this, the practical alternatives are A) use an older version of Mono, or B) stop using mono and emulate dotnet in Wine, using the Windows installer. In my opinion that is far less "clean". I'd far rather run native mono with some light path emulation than a whole Windows drive_c Wineprefix, if I could.

If this feature isn't in here, as far as I'm concerned it's a huge loss in compatibility and it makes Mono less useful for me personally.

[kg]

Since this thread was revived,

Convince me or some other maintainer of dotnet/runtime that it is a good feature to have. E.g. Do other runtimes (Java, Go, Rust, ...) have a feature like this?

Win32 itself contains a feature like this - When performing file operations like open, rename etc via Win32 APIs, various things happen to your paths before they are passed on to the underlying filesystem (NTFS, etc) because this both provides backwards compatibility and improves ease of use for both developers and end-users. Generally speaking, NTFS is case-sensitive and Win32 is not, and this usually works out in the end user's favor because the average westerner does not see a meaningful difference between 'orange.txt' and 'Orange.txt'. Windows is not the only environment that made this decision.

So when it comes to preserving an existing feature like IOMAP, IMO the debate becomes: Do we want to prioritize all the existing developers and end users who rely on this feature over an ongoing maintenance cost? How large is the maintenance cost? Do we believe the maintenance cost will increase significantly in the future? Do we want to provide this feature to new users, as well, or only offer it to people who already depend on it? Does this feature merely make things slightly easier for the developer, or does it enable use cases that don't exist without it?

This thread already contains examples of use cases that aren't possible without IOMAP, like sharing game data files across platforms. A modern video game's data files can measure in the tens to hundreds of gigabytes, and requiring game data files to change means having to re-download all of that. Many software users are on limited bandwidth plans, so the necessary patch to swap out a bunch of backslashes in game archives is competing for a slice of that quota. For independent software companies and open source developers, distributing these updates off of their own servers may literally cost them additional cloud transfer and storage fees.

It is a non-goal for us to keep all 10 or 20 year apps working on latest .NET Core versions without any changes. There are number of features present in .NET Framework and Mono runtimes that are stripped in .NET Core. Even between major .NET Core versions, we do occasionally make breaking changes to move the .NET platform forward to stay relevant that may require adjustments of the user code. To run old code without any changes, it is best to use the old runtime that the code was developed against.

I'm not sure this helps clarify the issue at all. It seems obvious that not every 10 or 20 year old app is a priority for compatibility, but in this case we're talking about a relatively old and established runtime feature with an obvious use case and a very low implementation cost, being used in production by a sizable amount of end-user facing software - people buying games on Steam etc, not just anonymous data-entry clerks inside one or two specific companies that could update their internal tools.

"Use the old runtime that the code was developed against" is not a realistic answer if we care about end users. Apple just rolled out an entirely new CPU+GPU architecture pair, eventually Old Runtimes will simply not work on that hardware. They already demonstrated a willingness to kill decades of 32-bit software in order to reduce their own maintenance burden and make their OS smaller (by not shipping 32-bit binaries anymore). Old Runtimes also contain security vulnerabilities and other defects, things that can't be addressed without an upgrade or very hazardous attempts to backport fixes. If we drop things like IOMAP, we're telling users of software originally developed for Mono - often before netcore was even an idea, let alone a usable product - that because of a decision they made a long time ago, they have to stay on older hardware and/or older versions of OS X.

As mentioned earlier in the thread this also goes beyond "running old code without any changes", because user-authored content may contain platform-specific paths that previously worked with IOMAP enabled and now will not work unless every bit of I/O in the entire application and its dependencies is rewritten to filter paths. This is certainly a thing you can do, but it's far more expensive than having <1000 lines of code in the PAL do it simply and robustly instead when an optional flag is turned on. Someone consuming nugets has no way to dig into those binaries and modify all of their System.IO calls.

IMO the use cases provided in this issue thread so far make a compelling case for this feature, and the fact that Ethan's draft PR is tiny and mostly composed of existing vetted, production-tested Mono code further supports the case for it. If there are specific concrete reasons why we view it as a significant maintenance burden or ongoing hazard, we should identify and enumerate them so we can settle the discussion for good. Otherwise between things like this and the removal-without-replacement of dllmap, we create the false impression that Mono developers and their end users don't matter to us, when that is absolutely not the case. Enumerating the hazards and reasoning behind our decision will also be valuable to the wider community because they can take those hazards into account when implementing their own workarounds.