dotnet / sdk

Core functionality needed to create .NET Core projects, that is shared between Visual Studio and CLI
https://dot.net/core
MIT License
2.7k stars 1.06k forks source link

RFC: dotnet [Arm64 | x64] coexistence #16896

Closed richlander closed 3 years ago

richlander commented 3 years ago

Microsoft and Apple are both producing new OSes for Arm64 that support x64 emulation. For .NET, we need to determine how to support .NET Arm64 and x64 builds on those OSes in a way that offers a reasonable user experience and that has reasonable cost.

Problem to solve

As stated, we need to offer x64 and Arm64 builds of .NET within one 64-bit namespace (without much help from the OS on how to manage it). That leads to three issues:

One could reasonable ask why we need to support coexistence once native Arm64 build are available.

Context

The Windows and macOS plans and implementations are similar but have key differences, per our understanding.

- Same: No WoW64 subsystem or Program Files (x64) style experience. For example, there is no x64 command prompt.
- Same: An executable can be universal.
- Different: macOS universal executables are transparently restartable, while Windows ones are not.
- Different: Windows x64 emulation is here to stay, while macOS x64 emulation will probably be removed in 3-5 years.

The following are various high-level solutions that we could employ, with pros and cons.

Status quo

On Windows x64, .NET is installed to two locations (depending on architecture):

- C:\Program Files\dotnet
- C:\Program Files (x86)\dotnet

Today, customers need one of those two directories in the path in order to get the "dotnet" experience they want, like if they want a 32-bit or 64-bit "dotnet build".

Note: There will not be a C:\Program Files (x64) directory on Windows Arm64. We are expected to install 64-bit products (Intel and/or Arm based) in C:\Program Files.

On macOS, .NET is installed to one location:

- /usr/local/share/dotnet 

Going forward

There will be one dotnet in the path, just like today. Much like the 32- and 64-bit support we offer on Windows, we'll offer both Arm64 and x64 builds of .NET and customers can install both or either, and can control which is in the PATH. That will determine if they get an Arm64 or x64 dotnet build. We are not planning on building a .NET version manager that enables switching which architecture you get. We intend to rely on the PATH.

The question is what structure we offer for the two 64-bit products, how intuitive that is (now and later) and how expensive that is for us to offer.

Native dotnet

Premise: there is a "dotnet" directory on every OS, and it is the native architecture.

We'd end up with the following, on Windows and macOS, respectively:

This is the usual "who gets the good name" problem.

Pros:

Cons:

Archify dotnet

Premise: fully embrace multi-arch support, with arch-specific directories.

We'd end up with the following:

On Arm64, we'd have these arch-specific directories. This is the "no one gets the good name option; everybody loses" option.

Pros:

Cons

Hide architecture differences

Premise: These differences don't need to be so apparent. We already have version folders under dotnet. We can add arch folders (or something similar). This option has a lot of sub-options, too.

Option 1 -- Insert a new folder, with discrete .NET hives underneath:

Option 2 -- Intermix architectures in one structure (here, just shown with Windows, for simplicity):

Put "x64" and "arm64" somewhere in the folder hierarchy. Where it is, is an implementation decision. It is the same (in spirit) as C:\Windows\Microsoft.NET\Framework and C:\Windows\Microsoft.NET\Framework64.

Option 3 -- Arm64 is native architecture with x64 intermixed in:

Pros:

Cons

Hide architectural differences with universal binaries

Both Windows and macOS offer a form of universal binaries, which we could use as a multiplexer between the two 64bit products. There are two problems with that.

We're rejecting this option for now. We could decide to adopt this solution on macOS and not Windows, should feedback push us in that direction.

Compatibility Wins

Premise: People have already installed .NET x64 on Apple Silicon machines. We need to respect that. This is the opposite of "Native dotnet"

We'd end up with the following:

Pros:

Cons:

Hard design questions

Other considerations

dotnet-issue-labeler[bot] commented 3 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

richlander commented 3 years ago

This issue is a kind of RFC.

I'm currently leaning to "Hide architectural differences" option 3. It offers much of the same experience as "Native dotnet" while providing a little more wiggle room than some of the other options.

jkotas commented 3 years ago

"Hide architectural differences" option 3

+1

C:\Program Files\dotnet\shared_x64\Microsoft.NETCore.App\6.0.0-preview.3.21181.6\

Why not park the frameworks, sdks and everything else under the x64 directory? ie make this C:\Program Files\dotnet\x64\shared\Microsoft.NETCore.App\6.0.0-preview.3.21181.6\, C:\Program Files\dotnet\x64\sdk\..., etc. The advantage of doing it this way is that everything in x64 environment can assume the same directory structure relative to dotnet.exe.

richlander commented 3 years ago

That makes total sense. My main point was that the x64 can go anywhere, but your proposal seems to be the best one for the reasons you've shared. Nice.

I'm not going to update the document now. Let's wait for a first round of review and then produce a new doc with the final plan.

snickler commented 3 years ago

I really like Hide Architectural Differences option 3.

Arm64 native with x64 mixed in, keeps everything together with a less confusing folder structure and would likely solve the issue of the hostfxr.dll being loaded for the incorrect architecture due to the incorrect x64 installation path - one of the big pains with arm64 and x64 coexistence.

I would absolutely LOVE to never see this again:

Message: Failed to load the dll from [C:\Program Files\dotnet\host\fxr\6.0.0-preview.3.21201.4\hostfxr.dll], HRESULT: 0x800700C1
The library hostfxr.dll was found, but loading it from C:\Program Files\dotnet\host\fxr\6.0.0-preview.3.21201.4\hostfxr.dll failed
snickler commented 3 years ago

Another nice suggestion would be to allow an option to include both arm64/x64 hostfxr.dlls in with framework-dependent deployments. Procmon lead me to discover that upon the load of a .NET executable, it first checks the directory the application exists for a hostfxr.dll.

By copying an x64 hostfxr.dll to the same directory of an x64 targeted framework-dependent app and running the executable, it properly loaded (In reference to https://github.com/microsoft/PowerToys/issues/10658)

This is more of a short-term workaround than an actual solution, though.

richlander commented 3 years ago

As you suggest, this option is more of a short term workaround. We will get this working so that this suggestion isn't needed.

Also, it wouldn't work for cross platform apps on one hand (they would need a lot of hostfxr binaries) and it wouldn't match with apps that were R2R compiled.

Marv51 commented 3 years ago

Can someone catch me up on why this is needed? Why would I want x64 dotnet on my arm64 computer, once a native version is available?

I'm sure this completely outs me as a noob, but why would installing to the same path be a bad idea? So that the two architectures replace each other?

snickler commented 3 years ago

Can someone catch me up on why this is needed? Why would I want x64 dotnet on my arm64 computer, once a native version is available?

I'm sure this completely outs me as a noob, but why would installing to the same path be a bad idea? So that the two architectures replace each other?

x64 on Arm64 emulation exists currently for the Windows Insiders Dev Channel builds, so we're able to run both Arm64 and x64 applications side by side. The current structure of the .NET installation throws a wrench in the mix for being able to execute .NET Core/.NET 5+ applications that are either of those architectures - since they both install to C:\Program Files\dotnet by default.

Since the same directory is being used, you're likely to have multiple SDK versions and packs installed with no way of telling whether they're x64 or Arm64. When dotnet loads the native hostfxr.dll (.NET Host Resolver) of one of the runtimes installed and it doesn't match the architecture of the calling executable (either dotnet.exe or the compiled-app.exe), you'll immediately encounter the error I listed above.

Basically, everything being loaded by either the dotnet executable or the executable/dll produced after building your project, and the .NET runtime dlls need to be built in the same architecture.

It's the same as if you had an x64 executable on an x64 build of Windows trying to load an ARM64 dll.

richlander commented 3 years ago

Thanks @snickler. Great answer.

I updated the Problem to Solve section. Does that help @Marv51?

jkotas commented 3 years ago

If we decided not to solve this problem,

This can be in the list of going forward options for the sake of completeness. Something like:

Ignore the problem

Premise: The non-native execution is point-in-time problem and it is not worth complicating the product for it. We will recomend to use dotnet-install script and DOTNET_ROOT environment variable as workaround for running x64 framework dependent apps on arm64 devices.

Option 1: Maintain status quo Pros: Zero cost to implement Cons: Breaks .NET Core SxS compatibility promise. Installing a new version of .NET Core on a machine should never break existing apps.

Option 2: Block installation of non-native .NET Runtime Pros: Avoids cons of option 1. Cons: Major breaking change. It is not possible to install .NET apps that come with x64 runtime on arm64 machines anymore. This includes Visual Studio.

snickler commented 3 years ago

Premise: The non-native execution is point-in-time problem and it is not worth complicating the product for it. We will recomend to use dotnet-install script and DOTNET_ROOT environment variable as workaround for running x64 framework dependent apps on arm64 devices.

I was thinking of doing this as another workaround, since the current SDK/Runtime installers don't properly respect the DOTNETHOST_X64 parameter (which prevented me from giving an easy workaround).

Instructions running the dotnet-install scripts to install the x64 runtimes to a common base path, and for setting that path in the InstallLocation REG_SZ value under HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\dotnet\Setup\InstalledVersions\x64 would also help.

EDIT: Oh. It actually worked.

  1. Download dotnet-install.ps1 from https://dot.net/v1/dotnet-install.ps1.
  2. Choose/Create a directory. For my Pro X, I'm using c:\dotnetx64.
  3. Open an elevated PowerShell/Pwsh and run the following
    $dotnetInstallPath = "HKLM:\Software\WOW6432Node\dotnet\Setup\InstalledVersions\x64"
    if(!(Test-Path $dotnetInstallPath))
    {
       New-Item -Path $dotnetInstallPath -Force;
    }
    New-ItemProperty -Path $dotnetInstallPath -Name "InstallLocation" -Value "c:\dotnetx64" -PropertyType String -Force;
  4. Change to the directory you downloaded the dotnet-install.ps1 to, and run .\dotnet-install.ps1 -Architecture x64 -InstallDir C:\dotnetx64\ for the LTS SDK build. .\dotnet-install.ps1 -Channel main -Version latest -Architecture x64 -InstallDir C:\dotnetx64\ for the latest SDK build from the main branch

By this point, you should be able to run an x64-compiled .NET executable without trouble.

riverar commented 3 years ago

Hide architecture differences Premise: These differences don't need to be so apparent. We already have version folders under dotnet. We can add arch folders (or something similar). This option has a lot of sub-options, too.

Option 1 -- Insert a new folder, with discrete .NET hives underneath: C:\Program Files\dotnet/arm64 C:\Program Files\dotnet/x64 /usr/local/share/dotnet/arm64 /usr/local/share/dotnet/x64

I like this approach, as it's explicit and makes the most sense to me. I would also potentially explore the idea of renaming the x64 folder to amd64 for additional clarity.

I firmly reject any idea that repurposes the "x64" moniker as a bucket for "64-bit" software. "x64", in the Windows community, has strong ties to amd64, and any other definition would introduce a lot of confusion.

I'm not a fan of universal binaries either, as it introduces a sort of non-determinism. "Did I flip the right switches to make this universal thing go to amd64 and not arm64 for my cross-compile project?"

richlander commented 3 years ago

Option 1 definitely has its merits, as you call out @riverar. The reason I like this option less than option 3 is that it is less future oriented. It says "x64 and Arm64 will be at parity forever." In Apple land, we know that's not true. We guess that Apple will stop supporting x64 emulation relatively soon, almost certainly before 2025. Windows is a different story. We've decided that we're going to make the same choice on Windows and macOS, at the very least for our sanity. Option 3 pulls ahead as the winner because the "arm64" folder quickly becomes unnecessary, at least on macOS.

I understand the amd64 vs x64 naming issue. I had an AMD64 test machine in my office before that AMD product was even public. I saw the naming discussions at Microsoft on the topic in 2003-2005. I think that's a historical thing and I disagree. Most people in the Windows community (outside of perhaps MVPs) are not familiar with that term, not care about it. We're very likely to go with x64. If the Windows team was to create a new Program Files folder for this purpose, I can tell you with confidence that it would be Program Files (x64). We talked about it, and that was the name they offered. The other name was never discussed.

riverar commented 3 years ago

The reason I like this option less than option 3 is that it is less future oriented. It says "x64 and Arm64 will be at parity forever."

Not sure what you mean by "will be at parity forever". Is the hypothetical here that developers will see these folders and somehow get the impression that the x64 runtime will work on their machines after a hypothetical update to macOS removes Rosetta 2 support? Or something else? Struggling to understand the problem you're trying to solve there and what makes option 3 "future oriented". Apologies.

vitek-karas commented 3 years ago

"Hide architectural differences" option 3

+1 and +100 for @jkotas suggestion to keep the same folder structure for both architectures - so everything x64 goes under the x64 directory.

Not sure what you mean by "will be at parity forever".

As mentioned we expect x64 emulation to disappear at least on macOS relatively soon. Once that happens we would like to get back to the simple world of having one dotnet installation on the machine - and that one should live in the default location - dotnet. For our own sanity we prefer options which keep things as consistent as possible across architectures/platforms. Going with dotnet/x64 and dotnet/arm64 would definitely work, but does it mean we should also move to dotnet/x86 in x64 Windows, or even dotnet/x64 on x64 Linux, even if it's the only architecture supported on those platforms? Note: On x64 Windows we now install to C:\Program Files\dotnet and C:\Program Files (x86)\dotnet.

(Bit of a sidetrack but I wanted to clear this up) In answer to @snickler comment about putting hostfxr into a framework dependent app.

In answer to @snickler comment about ability to manually install and register via registry.

riverar commented 3 years ago

For our own sanity we prefer options which keep things as consistent as possible across architectures/platforms.

Isn't the most consistent option then to explicitly create the architecture folders per Option 1? The "platform native" folder then stops being this thing that flaps around, requiring you to think "hm, what is platform native here?". I don't see any benefits to having this "symlink" for platform native. Just confusion. (Good luck with xplat documentation too!)

snickler commented 3 years ago

In answer to @snickler comment about ability to manually install and register via registry.

  • This is by design and it's basically what installers do. See this design doc for all the details on this.
  • Note that this only fixes the problem for applications with executable (apphost.exe), it doesn't have really any effect on the CLI itself (dotnet.exe).

I agree. Since most of the issues are around the apphost executables, that could be solved in this manner.

For the dotnet.exes, I don't see any easy way of getting around this other than possibly making it an ARM64EC-compiled binary? Not sure it's worth the effort. If anything, the apphost executable workaround is feasible.

richlander commented 3 years ago

@riverar -- the architecture folders approach is also consistent, but again, it's oriented on here and now. We don't want architectural folders in the future as Arm64 becomes the prevalent option. We have a strong philosophy that the native arch will have the good name. We expect that most other dev platforms and apps will do the same thing if they support x64 emulation.

In short, this boils down to "what do we want the platform to look like in 5 and 10 year". We want it to look the same as today, simply a "dotnet" folder in the place where you install dev platforms on a given OS.

After .NET Core 3.1 goes EOL, I expect we'll see x64 downloads from Arm64 machines drop significantly, below 10%. If that's true, do you hold the same view (which option to pick)? Or do you believe that the ratio will be different than the 9:1 one I've suggested?

pb5050 commented 3 years ago

im trying to do what they are saying but its not detailed enough for me i ranh the script and it didnt ask where to install? it just auto installed somewhere now im trying to figure that part out

ssimek commented 3 years ago

I can't see how anything other than a universal binary is the right approach for macOS. The OS support for UB is on a completely different level than in Windows and any attempt to create two sets of binaries and switch via PATH is bound to create more problems than it solves. Also, while it is unlikely Windows will ever reach even a majority of machines running ARM, it is for all practical purposes a certainty with macOS, since all new machines sold will be ARM soon.

Another point - native code constitutes only about 5% of the entire SDK package (26M out of 454M for .NET 6 preview 3, ARM64). While the remaining files differ for some reason between architecture builds, I hope this is due to some build process difference.

As a quick PoC, I tried stitching the arm64 and x86_64 SDKs using lipo - the result works only on the architecture used as a base with the other one failing with Failed to create CoreCLR, HRESULT: 0x80004005, but I guess it has to do with some files being architecture specific even though they are not Mach executables.

jkotas commented 3 years ago

native code constitutes only about 5% of the entire SDK package

That's not accurate. Nearly all managed .dlls in the SDK are AOT compiled and have architecture specific code in them as well.

ssimek commented 3 years ago

native code constitutes only about 5% of the entire SDK package

That's not accurate. Nearly all managed .dlls in the SDK are AOT compiled and have architecture specific code in them as well.

In that case, I have to apologize. I must have lived under a rock for a long time as I assumed the same technique is still applied as in FX 2.0 days when the .dlls got compiled only after installation. @jkotas Can you please provide some link to how this works? I'm really curious and a quick google turned out nothing relevant, just tons of R2R, CoreRT and similar stuff.

jkotas commented 3 years ago

AOT compilation is done during app build/publish in .NET Core. https://devblogs.microsoft.com/dotnet/conversation-about-crossgen2/ has some good context for this.

richlander commented 3 years ago

FYI: This doc is very incomplete. It has just enough detail in it to get feedback on high-level choices. I am writing another doc that goes into much more detail on the chosen option. I'll link to it from here.

Zooming out, I'd like to see a system where using x64 emulation was natural and didn't rely on the PATH. The linchpin to that is relying on the native architecture SDK to target both native architecture and emulated runtimes. There are some challenges to delivering that, so I cannot yet say that we are delivering that plan.

omajid commented 3 years ago

(Sorry, I haven't read the conversation here, jumped down from reading the initial issue)

Have you folks looked at how Debian suggests where architecture-specific packages (eg, dotnet) should be located?

https://www.debian.org/doc/debian-policy/ch-opersys.html:

... files to instead be installed to /lib/triplet and /usr/lib/triplet, where triplet is the value returned by dpkg-architecture -qDEB_HOST_MULTIARCH for the architecture of the package. Packages may not install files to any triplet path other than the one matching the architecture of that package; for instance, an Architecture: amd64 package containing 32-bit x86 libraries may not install these libraries to /usr/lib/i386-linux-gnu.

That seems to suggest another option in addition to the ones suggested in the first post:

Perhaps we can re-use an existing policy instead of inventing our own?

Btw, the FHS and Debian policy guidelines both suggest that dotnet belongs under /usr/lib/ (or /usr/local/lib...), not /usr/share/.

richlander commented 3 years ago

Thanks @omajid. That makes sense in terms of a Linux option. It doesn't seem directly applicable for macOS and even less so for Windows. I think the big difference is that Linux has a strong separation between install location and the user UX in many cases, with a heavy use of symlinks. Fair?

vitek-karas commented 3 years ago

@richlander @omajid Thinking about this some more, I actually think we didn't design anything which would tie us to a specific location. The /usr/shared/dotnet and /usr/shared/dotnet/x64 can be almost seen as an example. We will very likely use it on some OSes, but I don't see anything which would make us use that everywhere. We already have scenarios on Linux where the install location is different. That's what the /etc/dotnet/install_location is for. That combined with the right PATH settings should be enough to install into any location. Is it correct to assume that different Linux distros have different approach to this?

Side note: Proposal for updating the format of the /etc/dotnet/install_location to support multiple architectures on the same machine.

rseanhall commented 3 years ago

"Hide architectural differences" option 3 doesn't seem like a good solution to me. It requires the installer to detect that it is running under emulation and install to a different location in that scenario. This is not something that you can fix retroactively. For example, what if Windows decides to add Arm64 emulation to x64 OS in 5-10 years? You'll run into this same problem again, all of the Arm64 installers previously released will start installing into the native location.

richlander commented 3 years ago

what if Windows decides to add Arm64 emulation to x64 OS in 5-10 years

Highly unlikely scenario, but ignoring that. This same model would apply. We'd create an arm64 directory within the dotnet directory. We'd updated Arm64 installers to respect that.

What's the other option that you would endorse that would enable the scenario you've outlined that allows installers to work retroactively?

rseanhall commented 3 years ago

We'd update Arm64 installers to respect that.

That's my point. You can't retroactively update the Arm64 installers. Maybe it could work with your option if you can make all installers, today, detect when they are running under emulation and install to the architecture specific folder underneath the dotnet directory. Otherwise, it should probably always install into the architecture specific folder underneath the dotnet directory. I think that's Option 1?

richlander commented 3 years ago

We're not going to change the x64 model on x64. That's not warranted. We've been installing the dotnet directory since .NET Core 1.0. There is no good reason to change that. That would be over-pivoting to a theoretical problem.

jkotas commented 3 years ago

It is not unusual that we have to update the shipped .NET versions in servicing to account for OS changes. We have done that number of times. One example from many: https://github.com/dotnet/corefx/pull/34443 . Updating the installers for this (theoretical) case would be one of those changes.

richlander commented 3 years ago

This issue has been superseded by https://github.com/dotnet/sdk/issues/17464.

Design conversation is hosted at https://github.com/dotnet/designs/pull/217