Open richlander opened 1 year ago
Tagging subscribers to this area: @vitek-karas, @agocke, @vsadov See info in area-owners.md if you want to be subscribed.
Author: | richlander |
---|---|
Assignees: | - |
Labels: | `area-Host` |
Milestone: | - |
Depends where you want to do this. In a script something like which dotnet
is probably a good answer.
From code you could use nethost
library, it finds hostfxr
for a given application and from there you can go up 2 levels where dotnet
should live. nethost
will look at the same things as apphost
(if used correctly): https://github.com/dotnet/runtime/blob/main/docs/design/features/native-hosting.md#new-host-binary-for-finding-hostfxr
This comes up frequently in the C# compiler source code as there are a number of cases where we need to launch a managed process. Effectively we have csc.dll
and need to launch it as a process but that requires us first finding a dotnet
to use. The pattern for finding dotnet
hasn't been documented and as a result our implementation has been derived by looking at ENV variables set by the runtime, reviewing patterns in other parts of the .NET SDK, etc ... Basically trying to draw from other examples.
There are generally two scenarios I think about in this area:
dotnet build
? dotnet
that is used when running tools in the SDK. For example the runtime
repo uses this to effectively dogfood the latest runtime against SDK tools. This is very valuable.After thinking this through this morning I think the right approach going forward is to only consult $PATH
. Basically find the first dotnet
instance and use that. This mostly closely mirrors how a customer would run our own tooling and is most likely to line up with their expectations. Basically dotnet build
uses the same host as dotnet exec csc
. Further I think this should apply to any tool in the SDK, not just the compiler.
That would have the following impact on our product and testing:
$DOTNET_ROOT
or $DOTNET_HOST_PATH
for the purpose of finding dotnet
should cease doing so. Instead consult $PATH
only. dotnet
for testing / dogfood purposes by changing values like $DOTNET_ROOT
should cease doing so. Instead that specific infra run should prepend the directory that contains the recently built dotnet
to $PATH
If you're asking what is $DOTNET_HOST_PATH
I had the same question. In researching through our code I believe that it is effectively an arcade invention. It's a mechanism for us to control where arcade puts dotnet
into our enlistments. Unfortunately this seems to have filtered into our actual production code as well. That feels wrong.
Perhaps I have the history / intent of this variable wrong and if so please correct me 😄
I think DOTNET_HOST_PATH
is something SDK uses - there are many use cases in that repo: https://github.com/search?q=repo%3Adotnet%2Fsdk%20DOTNET_HOST_PATH&type=code
I think it uses it to pass the location of the host from the parent to the child processes, so that child processes use the same SDK/runtime install which was used to run the top level command. It's possible Arcade uses it as well. But I could be wrong.
For the usage inside SDK though we might want to keep looking at DOTNET_HOST_PATH
. The problematic scenario is for example:
/my/private/install/dotnet build
This should spawn the child nodes for msbuild, csc and so on using the runtime/SDK from /my/private/install
and not look for it on the PATH
. For example if I have only installed 7.0 globally (so it's on PATH) but then I run /my/private/8.0-preview.7/dotnet
then things should just work.
Unless we also change SDK to modify PATH for its child processes.
Outside of SDK I agree that PATH
is probably the best approach. Installers, scripts and users will set PATH
to point to dotnet.exe
because we tell them to and because they need it anyway (so that they can run dotnet build
and similar). It will also be set in CI systems to point to the dotnet from the SDK they want used and so on.
DOTNET_ROOT
in itself is not enough and there's much more involved algorithm how to find the runtime if that variable is not set. That's basically what nethost
implements.
I think DOTNET_HOST_PATH is something SDK uses -
I 100% agree it uses it, I'm questioning why it does that. The $DOTNET_HOST_PATH
does not appear to be a variable that we support as a product. The only places I can find where it is set is through arcade, not the customer facing product. As such I'm questioning whether or not those uses in the SDK are correct and at the moment I don't believe they are.
For the usage inside SDK though we might want to keep looking at DOTNET_HOST_PATH.
Why though? I'm struggling to see what value it provides vs. always using $PATH
and just modifying that to use it.
If we do want a variable that special cases the SDK though then
DOTNET_
feels wrong. This is for effectively our internal testing purposes. Giving it a name like DOTNTE_
feels like we're elevating it to an actual .NET feature. For the usage inside SDK though we might want to keep looking at DOTNET_HOST_PATH.
Why though? I'm struggling to see what value it provides vs. always using $PATH and just modifying that to use it.
I realized this as well after I sent the response. It would be simpler if SDK just modified PATH
for its child processes I assume and then everything could use `PATH.
Unfortunately PATH
is not great to use - especially on Windows. Historically our guidance (given to us by other teams) was to try to not parse PATH
and make sense out of it. But maybe it's OK for just spawning processes, since we can let the OS do it for us I assume.
Historically our guidance (given to us by other teams) was to try to not parse PATH and make sense out of it
Yeah I'm not really thrilled about parsing $PATH
either. Ended up just stealing the code from msbuild that does it. This is essentially how our entire build tool stack works today so it seems like a good source to draw some code from.
Would be nice to have an API for FindOnPath
but that seems unlikely. Also I'd still have to polyfill that for downlevel targets.
@rainersigwald, @baronfel: have you all ever considered exposing a FindToolPath
on ToolTask
or MSBuild APIs in general that we could use to find dotnet
? That way the logic for parsing $PATH
is centralized.
It's come up before but I couldn't find an issue so https://github.com/dotnet/msbuild/issues/9018.
@vitek-karas is there a managed API for finding the location of the muxer? Is this something we want to provide?
No - we don't have an API like that. There's something similar in SDK: https://github.com/dotnet/sdk/blob/a4b77d4fd5e878fd42aef22d86c0668c90888da6/src/Resolvers/Microsoft.DotNet.MSBuildSdkResolver/MSBuildSdkResolver.cs. And there seem to be other versions of this elsewhere, for example: https://github.com/microsoft/MSBuildLocator/blob/master/src/MSBuildLocator/DotNetSdkLocationHelper.cs (this one even mentions that maybe it should use nethost but doesn't).
dotnet test
also has something similar, where it is actually looking for dotnet.exe
in some cases.
I know we've had several discussions with various SDK teams about this in the past, but no specific result. The problem is that there are like 5 versions of this code each with its own set of quirks and it's too late to change that.
There is one other view to consider which was brought up in https://github.com/dotnet/roslyn/issues/69023. Essentially all the tools that execute as part of an SDK action should use the same dotnet
host.
For example:
> export PATH=/some/dotnet/install:$PATH
> which dotnet
/some/dotnet/install/dotnet
> /some/dotnet2/install/dotnet build
In this view point tools like the compiler would be launched using /some/dotnet2/install/dotnet
even though it is not the first dotnet
on $PATH
. Or possibly it's not on $PATH
at all.
I'm certainly sympathetic to this point of view. Mostly because the opposite view seems quite strange. Basically if I launch msbuild
with one dotnet
host and then suddenly other tools are launching with another dotnet
host that is effectively revealing implementation details of the build. Most of the time they are the same cause the primary customer use case is dotnet build
without explicit qualification of which dotnet
.
If there was an easy way from a process to find the dotnet
host that created it then I'd lean towards this as the primary mechanism for finding the dotnet
host. That cannot be the sole mechanism for finding dotnet
though as we still need to fall back to $PATH
lookup in cases where our .NET core tools are loaded from .NET Framework processes.
I think an API called something like AppContext.ExecutingMuxerPath
makes sense -- the idea is that we would internally use the native hosting APIs to grab the location of hostfxr, and then from there we would try to find dotnet.exe
(the muxer) in the expected location.
The problem being that there are a number of uncommon configurations where this will not work, including self-contained deployments where there just isn't a muxer to find. So this API would probably return null
in a variety of cases where it's not possible to return a useful path.
@jaredpar Would this address all scenarios you're thinking of? I think the basic idea is that this would fix the problem, "I would like to start a new dotnet process using the same runtime/host that I'm running under." It wouldn't try to solve the problem of launching the "first" .NET process (either from native code or from the desktop framework). In those cases you would have to rely on PATH being appropriately configured, or use the native hosting APIs.
Another limitation would be that the muxer serves double-duty as the entry point to both the runtime and to the SDK. There's no guarantee that an SDK will actually be present in any given dotnet process though. So people have to be careful to only use this to execute DLLs, not run SDK commands.
For the usage inside SDK though we might want to keep looking at DOTNET_HOST_PATH.
Why though? I'm struggling to see what value it provides vs. always using $PATH and just modifying that to use it.
I'm also not a fan of overriding PATH and looking into it, because it means that we can't use a dotnet exe without wrapping it with a batch/pwsh to setup this variable. That's really not great.
Even if DOTNET_HOST_PATH
is not official, it seems that it is still considered and used all over the places as the primary location to look before going to the path, so why not making it official? Could we then always enforce it from dotnet.exe? (when run for the first time without a DOTNET_HOST_PATH
?)
I think an API called something like AppContext.ExecutingMuxerPath makes sense -- the idea is that we would internally use the native hosting APIs to grab the location of hostfxr, and then from there we would try to find dotnet.exe (the muxer) in the expected location.
Having an API would be nice, but I would think that making DOTNET_HOST_PATH
first class would be simpler and would even make it working with launching intermediate sub-child process shells that inherit from this variable and can resolve to dotnet.exe without having to explicitly propagate this variable around. It makes it more consistent to stick with an environment variable to propagate such things between managed and native processes.
"I would like to start a new dotnet process using the same runtime/host that I'm running under.
For me at least that is the primary problem to solve. Basically I have a suite of tools that launch each other and I want them to use the same host.
Having an API would be nice, but I would think that making DOTNET_HOST_PATH first class would be simpler
This is a proven approach we've used in MsBuild and it functions well for this type of scenario. Perhaps this new API just seeds this value more automatically.
@elinor-fung Any thoughts on the above? Either a managed API, or setting DOTNET_HOST_PATH
environment variable when initializing the runtime?
setting DOTNET_HOST_PATH environment variable when initializing the runtime
I don't really want to get the host itself into the business of setting environment variables. I think it would be reasonable to enable something like AppContext.GetData("DOTNET_HOST_PATH")
.
"I would like to start a new dotnet process using the same runtime/host that I'm running under.
For me at least that is the primary problem to solve. Basically I have a suite of tools that launch each other and I want them to use the same host.
Is it the same host that is desired, or the same runtime? Given https://github.com/dotnet/designs/pull/303, those could be different locations. My impression is that the same runtime would be the desire.
What about versions? Even if we provide API to get to the same runtime location, there might be several runtimes (and SDKs) installed there. Are the scenarios such that we would also want to be able to use the same version of the runtime/SDK as the currently running process? (I must admit I don't know exactly how we would do that, just asking what the scenario calls for).
For build scenarios I think we want "same runtime and SDK as current MSBuild process"--at least for SDK built-in tooling like Roslyn. Tools packaged in NuGet packages may want more flexibility but that feels like a good default.
I don't really want to get the host itself into the business of setting environment variables
On that topic, I have been having a difficult time to track down which process/main is actually setting DOTNET_HOST_PATH
? When we are referring to DOTNET_HOST_PATH
, is it acknowledged that it is only about the muxer? (dotnet
exe, in the SDK situation)
For build scenarios I think we want "same runtime and SDK as current MSBuild process"--at least for SDK built-in tooling like Roslyn. Tools packaged in NuGet packages may want more flexibility but that feels like a good default.
By curiosity, how do we propagate from process to process the same runtime? (assuming we go through the dotnet
muxer?) Do we point to the dotnet path/sdk/x.y.z/dotnet.dll
in a specific SDK version folder?
I don't really want to get the host itself into the business of setting environment variables. I think it would be reasonable to enable something like AppContext.GetData("DOTNET_HOST_PATH").
I understand, and agree with, the reluctance for the runtime to be setting environment variables. At the same time though it would seem odd if the runtime is giving an API to find the host executable but then our SDK tool chain is effectively inventing another mechanism via $DOTNET_HOST_PATH
environment variable. If we go with a new API then I think we need to consider moving the SDK to using that mechanism as the primary mode.
Is it the same host that is desired, or the same runtime?
Agree it's the same runtime. Basically the location we can dotnet exec
with to get the same runtime executing the current process.
Is it the same host that is desired, or the same runtime? Given https://github.com/dotnet/designs/pull/303, those could be different locations. My impression is that the same runtime would be the desire.
Actually, I don't think my statement was well-defined. What DOTNET_HOST_PATH is intended to do is point to a muxer that can run the target binary. There's no guarantee that binary will end up actually using the same runtime, due to all the decisions that happen when loading the binary (roll-forward, etc).
I think my proposal is: find the muxer associated with the current hostfxr, if one exists. Run under that muxer. No guarantees about SDK or runtime.
I understand, and agree with, the reluctance for the runtime to be setting environment variables. At the same time though it would seem odd if the runtime is giving an API to find the host executable but then our SDK tool chain is effectively inventing another mechanism via
$DOTNET_HOST_PATH
environment variable. If we go with a new API then I think we need to consider moving the SDK to using that mechanism as the primary mode.
I think there are potentially two different scenarios here: do you want tools launched under your process to know that this is a .NET process and where it was launched from? Or do you simply want managed processes to be able to find a compatible muxer?
I think of this as the divide between native children and managed children. If you want native children to know the location of the muxer, you probably want to use an environment variable. But also, I don't think this is an appropriate contract to use for the .NET runtime -- we wouldn't want to automatically pollute the environment variable space of child processes.
Conversely, a managed API is effectively an entirely internal contract to a .NET binary. That's definitely the kind of thing that makes sense to build into every .NET app.
So I think an AppContext switch makes a lot of sense as a universal host feature, and environment variable less so. But, I would also find it reasonable if the dotnet
CLI/SDK, in particular, wanted to have all child processes inherit a .NET-specific variable. That would be something unique to running tools under the SDK and we wouldn't expect, say, a random WinForms app to start passing its muxer path to native child processes.
There are scenarios where apps need to be launched with
dotnet
host. The big question is how to find the host, with the two obvious places to look being:PATH
DOTNET_ROOT
,/etc/dotnet/install_location
, ...)I don't believe we've documented this.
This question may be similar to how one locates
node
from code given the use ofnvm
. I haven't looked at that pattern/guidance, but it seems similar.