FreeBSD 13 jail: Failed to create CoreCLR: HRESULT: 0x8007054f

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.

https://docs.microsoft.com/dotnet/core/

MIT License

15.28k stars 4.73k forks source link

FreeBSD 13 jail: Failed to create CoreCLR: HRESULT: 0x8007054f #99857

Closed alucryd closed 7 months ago

alucryd commented 8 months ago

Description

It is impossible to run dotnet 8 inside a FreeBSD 13 jail, where it was previously possible to run dotnet 6.

Reproduction Steps

Create a fresh jail on TrueNAS Core (with allow_mlock, which was required for dotnet 6)
Install dotnet from the repo: pkg install dotnet
Try running dotnet --help

Expected behavior

Dotnet 8 should run, as did dotnet 6.

Actual behavior

Dotnet fails to run with the following error: Failed to create CoreCLR: HRESULT: 0x8007054f.

Regression?

Was working fine with dotnet 6.

Known Workarounds

No response

Configuration

Dotnet 8.0 FreeBSD 13 jail on TrueNAS Core 13.0-U6.1

Other information

No response

dotnet-policy-service[bot] commented 8 months ago

Tagging subscribers to this area: @mangod9 See info in area-owners.md if you want to be subscribed.

am11 commented 7 months ago

root@f13:~ # truss -f dotnet --help 2>&1 | grep ERR

75727: open("/usr/local/etc/libmap.d",O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,0165) ERR#2 'No such file or directory'
75727: open("/lib/casper/libthr.so.3",O_RDONLY|O_CLOEXEC|O_VERIFY,032266000) ERR#2 'No such file or directory'
75727: open("/lib/casper/libc++.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
75727: open("/lib/libc++.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
75727: open("/lib/casper/libcxxrt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,013026041400) ERR#2 'No such file or directory'
75727: open("/lib/casper/libm.so.5",O_RDONLY|O_CLOEXEC|O_VERIFY,00) ERR#2 'No such file or directory'
75727: open("/lib/casper/libgcc_s.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,010320643400) ERR#2 'No such file or directory'
75727: open("/lib/casper/libc.so.7",O_RDONLY|O_CLOEXEC|O_VERIFY,020443000) ERR#2 'No such file or directory'
75727: readlink("/etc/malloc.conf",0x7fffffffcc10,1024) ERR#2 'No such file or directory'
75727: access("/usr/local/dotnet/libcoreclr.so",F_OK) ERR#2 'No such file or directory'
75727: access("/usr/local/dotnet/dotnet.dll",F_OK) ERR#2 'No such file or directory'
75727: access("/root/global.json",F_OK)      ERR#2 'No such file or directory'
75727: access("/global.json",F_OK)       ERR#2 'No such file or directory'
75727: __realpathat(AT_FDCWD,"/usr/local/dotnet/sdk/8.0.100/dotnet.runtimeconfig.dev.json",0x801a1e000,1024,0) ERR#2 'No such file or directory'
75727: fstatat(AT_FDCWD,"/usr/local/dotnet/sdk/8.0.100/dotnet.runtimeconfig.dev.json",0x7fffffffd340,AT_SYMLINK_NOFOLLOW) ERR#2 'No such file or directory'
75727: __realpathat(AT_FDCWD,"/usr/local/dotnet/shared/Microsoft.NETCore.App/8.0.0/Microsoft.NETCore.App.runtimeconfig.dev.json",0x801a1e000,1024,0) ERR#2 'No such file or directory'
75727: fstatat(AT_FDCWD,"/usr/local/dotnet/shared/Microsoft.NETCore.App/8.0.0/Microsoft.NETCore.App.runtimeconfig.dev.json",0x7fffffffce50,AT_SYMLINK_NOFOLLOW) ERR#2 'No such file or directory'
75727: access("",F_OK)               ERR#2 'No such file or directory'
75727: access("opt/coreservicing",F_OK)      ERR#2 'No such file or directory'
75727: access("",F_OK)               ERR#2 'No such file or directory'
75727: access("/usr/local/dotnet/store/x64/net8.0",F_OK) ERR#2 'No such file or directory'
75727: open("/usr/local/lib/librt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
75727: open("/lib/casper/librt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
75727: open("/lib/librt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
75727: open("/usr/local/lib/liblzma.so.5",O_RDONLY|O_CLOEXEC|O_VERIFY,026454000) ERR#2 'No such file or directory'
75727: open("/lib/casper/liblzma.so.5",O_RDONLY|O_CLOEXEC|O_VERIFY,026454000) ERR#2 'No such file or directory'
75727: open("/lib/liblzma.so.5",O_RDONLY|O_CLOEXEC|O_VERIFY,026454000) ERR#2 'No such file or directory'
75727: open("/usr/local/lib/libz.so.6",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
75727: open("/lib/casper/libz.so.6",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
75727: open("/lib/casper/libmd.so.6",O_RDONLY|O_CLOEXEC|O_VERIFY,00) ERR#2 'No such file or directory'
75727: statfs("/sys/fs/cgroup",0x7fffffffc958)   ERR#2 'No such file or directory'
75727: open("/proc/self/mountinfo",O_RDONLY,0666) ERR#2 'No such file or directory'
75727: open("/proc/self/mountinfo",O_RDONLY,0666) ERR#2 'No such file or directory'
75727: mlock(0x883200000,4096)           ERR#1 'Operation not permitted'

Apparently, it requires mlock permission. In your jail conf file, add allow.mlock and restart the jail. (I use cbsd, so there it was about adding this line allow_mlock="1" for jcreate after https://github.com/am11/node-sass/blob/freebsd-ci/scripts/configure_freebsd_ci_jail.sh#L32)

I haven't tried .NET 6, but it would be interesting to know when this mlock dependency was introduced and if it was intentional. @janvorli, @Thefrank any ideas?

MichalPetryka commented 7 months ago

(with allow_mlock, which was required for dotnet 6)

@am11 FYI, also I think that executable pages might be implicitly locked?

am11 commented 7 months ago

Maybe, but with allow_mlock, the webapi started for me successfully in freebsd 13.3 jail:

(virtualizing freebsd-amd64 on osx-arm64 UTM and jail OS is also amd64)

alucryd commented 7 months ago

Mlock is already enabled, it was already a requirement for DotNet 6. I have one user who can run it successfully though, might be limited to my particular setup, trying to know more. I run TrueNAS Core as a VM inside TrueNAS scale, never had any issue with DotNet 6 under the same condition though. DotNet 8 also works fine on FreeBSD 13 and 14 also running as VMs on Scale. Could also be that my jail is broken, I will try to start fresh later today.

am11 commented 7 months ago

My jail setup was failing and recreating with allow_mlock fixed it. Can you share the output of truss -f dotnet --help 2>&1 | grep ERR? I'm on freebsd 14 host and 13.3 jail.

alucryd commented 7 months ago

New information, I'm told it works in a 13.3 jail, but not in a 13.2 jail, didn't realize I didn't have the latest version since I did update it before testing. Will need to create a fresh one to get the latest version. I'm sure I'll find that it works there, will keep you posted. Thanks for the help.

Thefrank commented 7 months ago

@am11 the mlock requirement for jails was there since I started building around .netcore 3.1 ~ .net5. As far as I know, this is a FreeBSD jail behavior and not something that was added intentionally into the runtime.

@alucryd the pkg builds (pkg install dotnet) are non-portable and will likely only work under the version they were built. The major issue being which OpenSSL lib it is linked against.

alucryd commented 7 months ago

@TheFrank Yeah learned that the hard way, wasn't able to have a single package for both 13 and 14. I can confirm that dotnet 8 does work fine in a 13.3 jail though, so this particular issue can be closed. Thank you all for helping with my issue.

am11 commented 7 months ago

I'm told it works in a 13.3 jail, but not in a 13.2 jail

I created 13.2 jail with cbsd and dotnet 8 works there as well. If you are able to repro it again, the truss output will help understanding the cause. You can also run it under lldb inside the jail (with settings set target.process.follow-fork-mode child).

alucryd commented 7 months ago

@am11 There you go, this outcome has been confirmed by 2 other TrueNAS Core users:

[root@dotnet ~]# truss -f dotnet --help 2>&1 | grep ERR
 9864: open("/usr/local/etc/libmap.d",O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,0165) ERR#2 'No such file or directory'
 9864: open("/lib/libc++.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
 9864: readlink("/etc/malloc.conf",0x7fffffffd5a0,1024) ERR#2 'No such file or directory'
 9864: open("/dev/pvclock",O_RDONLY|O_CLOEXEC,04) ERR#2 'No such file or directory'
 9864: access("/usr/local/dotnet/libcoreclr.so",F_OK) ERR#2 'No such file or directory'
 9864: access("/usr/local/dotnet/dotnet.dll",F_OK) ERR#2 'No such file or directory'
 9864: access("/root/global.json",F_OK)      ERR#2 'No such file or directory'
 9864: access("/global.json",F_OK)       ERR#2 'No such file or directory'
 9864: __realpathat(AT_FDCWD,"/usr/local/dotnet/sdk/8.0.100/dotnet.runtimeconfig.dev.json",0x801a1e000,1024,0) ERR#2 'No such file or directory'
 9864: fstatat(AT_FDCWD,"/usr/local/dotnet/sdk/8.0.100/dotnet.runtimeconfig.dev.json",0x7fffffffdcd0,AT_SYMLINK_NOFOLLOW) ERR#2 'No such file or directory'
 9864: __realpathat(AT_FDCWD,"/usr/local/dotnet/shared/Microsoft.NETCore.App/8.0.0/Microsoft.NETCore.App.runtimeconfig.dev.json",0x801a1e000,1024,0) ERR#2 'No such file or directory'
 9864: fstatat(AT_FDCWD,"/usr/local/dotnet/shared/Microsoft.NETCore.App/8.0.0/Microsoft.NETCore.App.runtimeconfig.dev.json",0x7fffffffd7e0,AT_SYMLINK_NOFOLLOW) ERR#2 'No such file or directory'
 9864: access("",F_OK)               ERR#2 'No such file or directory'
 9864: access("opt/coreservicing",F_OK)      ERR#2 'No such file or directory'
 9864: access("",F_OK)               ERR#2 'No such file or directory'
 9864: access("/usr/local/dotnet/store/x64/net8.0",F_OK) ERR#2 'No such file or directory'
 9864: open("/usr/local/lib/librt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
 9864: open("/lib/librt.so.1",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
 9864: open("/usr/local/lib/liblzma.so.5",O_RDONLY|O_CLOEXEC|O_VERIFY,026454000) ERR#2 'No such file or directory'
 9864: open("/lib/liblzma.so.5",O_RDONLY|O_CLOEXEC|O_VERIFY,026454000) ERR#2 'No such file or directory'
 9864: open("/usr/local/lib/libz.so.6",O_RDONLY|O_CLOEXEC|O_VERIFY,014232266000) ERR#2 'No such file or directory'
 9864: statfs("/sys/fs/cgroup",0x7fffffffd2e8)   ERR#2 'No such file or directory'
 9864: open("/proc/self/mountinfo",O_RDONLY,0666) ERR#2 'No such file or directory'
 9864: open("/proc/self/mountinfo",O_RDONLY,0666) ERR#2 'No such file or directory'
 9864: cpuset_getaffinity(0x3,0x9,0x2688,0x20,0x7fffdfffdf70) ERR#22 'Invalid argument'

am11 commented 7 months ago

9864: cpuset_getaffinity(0x3,0x9,0x2688,0x20,0x7fffdfffdf70) ERR#22 'Invalid argument'

ERR 22 is EINVAL, and per https://man.freebsd.org/cgi/man.cgi?query=cpuset_getaffinity&sektion=2&n=1, it happens when arg1 (level), arg2 (which) or arg5 (mask) is invalid.

Ballpark: I guess the cpuset={a CPU ID} (or cpuset={min}-{max} or cpuset={comma-separated CPU IDs}) option in jail conf is invalid? or 13.2 had a bug when that option is specified which 13.3 fixed.

Thefrank commented 7 months ago

uname -a from TrueNAS-13.0-U6.1

13.1-RELEASE-p9 FreeBSD 13.1-RELEASE-p9 n245429-296d095698e TRUENAS amd64

The jail is running a newer version of FreeBSD than the host? While that is something that you can do, it is not advised. ~My guess here is that it was a change from 13.0/13.1->13.2 that was not picked up by iXSystems. AFAIK they track STABLE not RELENG so it might not have even been in the branch due to ABI breakage.~ edit: I am not seeing anything immediate in terms of changes here but stock FreeBSD 13.2 works and FreeBSD 13.2 under a FreeBSD ~13.1 kernel does not.

This version of dotnet8 (sdk 8.0.100) was built under FreeBSD 12.4 and works under TrueNAS CORE 13.1 u6.1 jails

root@dtest:~/test # ./dotnet --info
.NET SDK:
 Version:           8.0.100
 Commit:            57efcf1350
 Workload version:  8.0.100-manifests.6c33ef20

Runtime Environment:
 OS Name:     FreeBSD
 OS Version:  13
 OS Platform: FreeBSD
 RID:         freebsd-x64
 Base Path:   /root/test/sdk/8.0.100/

.NET workloads installed:
 Workload version: 8.0.100-manifests.6c33ef20
There are no installed workloads to display.

Host:
  Version:      8.0.0
  Architecture: x64
  Commit:       5535e31a71

.NET SDKs installed:
  8.0.100 [/root/test/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.0 [/root/test/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.0 [/root/test/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

root@dtest:~/test # uname -a
FreeBSD dtest 13.1-RELEASE-p9 FreeBSD 13.1-RELEASE-p9 n245429-296d095698e TRUENAS amd64

dotnet8 (sdk 8.0.101) that was built under FreeBSD 13.2 does not work in either a FreeBSD 13.1 or FreeBSD 13.2 jail under TrueNAS CORE. The result is the same when using pkg install dotnet instead of my prebuilt.

root@dtest2:~/test # ./dotnet --help
Failed to create CoreCLR, HRESULT: 0x8007054F

Atleast this lets me know I need to build THREE versions of my packages now :(

am11 commented 7 months ago

Atleast this lets me know I need to build THREE versions of my packages now :(

Was that a bug in 13.2 jail or does building for 13.2 specifically fix the problem under TrueNAS Core? It sounds to me the former case that 13.2 jail (created with cpuset=# etc.) was buggy.

Thefrank commented 7 months ago

TrueNAS CORE uses iocage to manage jails in the GUI. Both iocage and the built-in jail/jls/jexec` are available in the console.

The config when using iocage is rather sparse iocage create -n newjail vnet="on" dhcp="on" bpf="on" allow_mlock="1" defaultrouter=192.168.0.1/24 -r 13.2-RELEASE gives us:

{
    "allow_mlock": 1,
    "bpf": 1,
    "cloned_release": "13.2-RELEASE",
    "defaultrouter": "192.168.0.1/24",
    "dhcp": 1,
    "host_hostname": "newjail",
    "host_hostuuid": "newjail",
    "jail_zfs_dataset": "iocage/jails/newjail/data",
    "release": "13.2-RELEASE-p8",
    "vnet": 1
}

AFAIK cpuset does not work on the current version of iocage but it did on older versions.

Here is what I have been able to determine via testing on AMD64 systems with dotnet8

Host	Jailed	.NET8 working "built for"
FreeBSD 13.2-RELEASE-p9	no	12.4, 13.2, pkg
FreeBSD 13.2-RELEASE-p9	13.2-p9	12.4, 13.2, pkg
FreeBSD 13.3-RELEASE	no	untested
FreeBSD 13.3-RELEASE	13.3	untested
TrueNAS CORE - 13.0-U6.1 (FreeBSD 13.1-RELEASE-p9 n245429-296d095698e)	no	N/A
TrueNAS CORE - 13.0-U6.1 (FreeBSD 13.1-RELEASE-p9 n245429-296d095698e)	13.1-p9	12.4
TrueNAS CORE - 13.0-U6.1 (FreeBSD 13.1-RELEASE-p9 n245429-296d095698e)	13.2-p8	12.4
FreeBSD 14.0-RELEASE	no	all
FreeBSD 14.0-RELEASE	14.0	all

all=pkg install dotnet AND prebuilts for ANY version of FreeBSD work (currently includes: 12.4 until 8.0.101 when EOL'd, 13.2, and 14.0)
VERSION=ABI targeted/built for.
pkg=installed via pkg install dotnet
N/A=something outside of user control prevents this from working (i.e. design decision)

A cursory check of net7 and net6 yield similar results. Only versions built for 12.4 will work under TrueNAS CORE.

I guess I need to build my packages for 13.1(?)/whatever TrueNAS is, 13.2, and 14.0. I might be able to get away with dropping 14.0 as 13.2 binaries seem to work fine post-OpenSSL patch

am11 commented 7 months ago

The error 9864: cpuset_getaffinity(0x3,0x9,0x2688,0x20,0x7fffdfffdf70) ERR#22 'Invalid argument' in getaffinity related to cpuset might be the one fixed by https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=260487 (as we use sched_getaffinity in runtime which calls the underlying cpuset_getaffinity).

I guess I need to build my packages for

I think we should try to keep the runtime code robust against the point-in-time kernel/base-system bugs, as we do on linux; rather than expecting that many variant builds by the package maintainers.

The next step would be to repro it with plain C and find out the issue with cpuset_getaffinity syscall:

$ cat > repro.c <<EOF
#include <sched.h>
#include <unistd.h>

int main() {
    cpu_set_t cpuSet;
    return sched_getaffinity(getpid(), sizeof(cpu_set_t), &cpuSet);
}
EOF

$ cc repro.c
$ truss -f ./a.out

Unfortunately, I couldn't reproduce the error in my jail environment so far.

Thefrank commented 7 months ago

first test: freebsd-version ; uname -a

13.1-RELEASE-p9
FreeBSD 13.1-RELEASE-p9 FreeBSD 13.1-RELEASE-p9 n245429-296d095698e TRUENAS amd64

result:

48930: mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34361978880 (0x800223000)
48930: mprotect(0x800220000,4096,PROT_READ)      = 0 (0x0)
48930: issetugid()                               = 0 (0x0)
48930: sigfastblock(0x1,0x8002228f0)             = 0 (0x0)
48930: open("/etc/libmap.conf",O_RDONLY|O_CLOEXEC,010460030) = 3 (0x3)
48930: fstat(3,{ mode=-rw-r--r-- ,inode=436415,size=47,blksize=4096 }) = 0 (0x0)
48930: read(3,"# $FreeBSD$\nincludedir /usr/loc"...,47) = 47 (0x2f)
48930: close(3)                                  = 0 (0x0)
48930: open("/usr/local/etc/libmap.d",O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,0165) ERR#2 'No such file or directory'
48930: open("/var/run/ld-elf.so.hints",O_RDONLY|O_CLOEXEC,010002363) = 3 (0x3)
48930: read(3,"Ehnt\^A\0\0\0\M^@\0\0\0\M^W\0\0"...,128) = 128 (0x80)
48930: fstat(3,{ mode=-r--r--r-- ,inode=983680,size=279,blksize=4096 }) = 0 (0x0)
48930: pread(3,"/lib:/usr/lib:/usr/lib/compat:/u"...,151,0x80) = 151 (0x97)
48930: close(3)                                  = 0 (0x0)
48930: open("/lib/libc.so.7",O_RDONLY|O_CLOEXEC|O_VERIFY,012320443000) = 3 (0x3)
48930: fstat(3,{ mode=-r--r--r-- ,inode=64561,size=1955744,blksize=131072 }) = 0 (0x0)
48930: mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362114048 (0x800244000)
48930: mmap(0x0,4235264,PROT_NONE,MAP_GUARD,-1,0x0) = 34362118144 (0x800245000)
48930: mmap(0x800245000,540672,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0) = 34362118144 (0x800245000)
48930: mmap(0x8002c9000,1359872,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x83000) = 34362658816 (0x8002c9000)
48930: mmap(0x800415000,36864,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1ce000) = 34364018688 (0x800415000)
48930: mmap(0x80041e000,28672,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1d6000) = 34364055552 (0x80041e000)
48930: mmap(0x800425000,2269184,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0) = 34364084224 (0x800425000)
48930: munmap(0x800244000,4096)                  = 0 (0x0)
48930: close(3)                                  = 0 (0x0)
48930: mprotect(0x800415000,32768,PROT_READ)     = 0 (0x0)
48930: sysarch(AMD64_SET_FSBASE,0x7fffffffdf80)  = 0 (0x0)
48930: mprotect(0x800415000,32768,PROT_READ|PROT_WRITE) = 0 (0x0)
48930: mprotect(0x800415000,32768,PROT_READ)     = 0 (0x0)
48930: readlink("/etc/malloc.conf",0x7fffffffd670,1024) ERR#2 'No such file or directory'
48930: issetugid()                               = 0 (0x0)
48930: __sysctl("vm.overcommit",2,0x7fffffffbbfc,0x7fffffffbbf0,0x0,0) = 0 (0x0)
48930: mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 34368126976 (0x800800000)
48930: mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34370224128 (0x800a00000)
48930: mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 34372321280 (0x800c00000)
48930: mprotect(0x202000,4096,PROT_READ)         = 0 (0x0)
48930: getpid()                                  = 48930 (0xbf22)
48930: cpuset_getaffinity(0x3,0x2,0xbf22,0x20,0x7fffffffea98) = 0 (0x0)
48930: exit(0x0)
48930: process exit, rval = 0

second test: freebsd-version ; uname -a

13.2-RELEASE-p9
FreeBSD 13.1-RELEASE-p9 FreeBSD 13.1-RELEASE-p9 n245429-296d095698e TRUENAS amd64

result:

49363: mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34361978880 (0x800223000)
49363: mprotect(0x800220000,4096,PROT_READ)      = 0 (0x0)
49363: issetugid()                               = 0 (0x0)
49363: sigfastblock(0x1,0x800222990)             = 0 (0x0)
49363: open("/etc/libmap.conf",O_RDONLY|O_CLOEXEC,010460030) = 3 (0x3)
49363: fstat(3,{ mode=-rw-r--r-- ,inode=27136,size=47,blksize=4096 }) = 0 (0x0)
49363: read(3,"# $FreeBSD$\nincludedir /usr/loc"...,47) = 47 (0x2f)
49363: close(3)                                  = 0 (0x0)
49363: open("/usr/local/etc/libmap.d",O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,0165) ERR#2 'No such file or directory'
49363: open("/var/run/ld-elf.so.hints",O_RDONLY|O_CLOEXEC,010002363) = 3 (0x3)
49363: read(3,"Ehnt\^A\0\0\0\M^@\0\0\0G\0\0\0\0"...,128) = 128 (0x80)
49363: fstat(3,{ mode=-r--r--r-- ,inode=93655,size=199,blksize=4096 }) = 0 (0x0)
49363: pread(3,"/lib:/usr/lib:/usr/lib/compat:/u"...,71,0x80) = 71 (0x47)
49363: close(3)                                  = 0 (0x0)
49363: open("/lib/libc.so.7",O_RDONLY|O_CLOEXEC|O_VERIFY,012320443000) = 3 (0x3)
49363: fstat(3,{ mode=-r--r--r-- ,inode=287404,size=1958576,blksize=131072 }) = 0 (0x0)
49363: mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362114048 (0x800244000)
49363: mmap(0x0,4206592,PROT_NONE,MAP_GUARD,-1,0x0) = 34362118144 (0x800245000)
49363: mmap(0x800245000,540672,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0) = 34362118144 (0x800245000)
49363: mmap(0x8002c9000,1359872,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x83000) = 34362658816 (0x8002c9000)
49363: mmap(0x800415000,40960,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1ce000) = 34364018688 (0x800415000)
49363: mmap(0x80041f000,28672,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1d7000) = 34364059648 (0x80041f000)
49363: mmap(0x800426000,2236416,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0) = 34364088320 (0x800426000)
49363: munmap(0x800244000,4096)                  = 0 (0x0)
49363: close(3)                                  = 0 (0x0)
49363: mprotect(0x800415000,36864,PROT_READ)     = 0 (0x0)
49363: sysarch(AMD64_SET_FSBASE,0x7fffffffdf70)  = 0 (0x0)
49363: mprotect(0x800415000,36864,PROT_READ|PROT_WRITE) = 0 (0x0)
49363: mprotect(0x800415000,36864,PROT_READ)     = 0 (0x0)
49363: readlink("/etc/malloc.conf",0x7fffffffd660,1024) ERR#2 'No such file or directory'
49363: issetugid()                               = 0 (0x0)
49363: mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 34368126976 (0x800800000)
49363: mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34370224128 (0x800a00000)
49363: mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 34372321280 (0x800c00000)
49363: mprotect(0x202000,4096,PROT_READ)         = 0 (0x0)
49363: getpid()                                  = 49363 (0xc0d3)
49363: cpuset_getaffinity(0x3,0x9,0xc0d3,0x20,0x7fffffffea98) ERR#22 'Invalid argument'
49363: exit(0xffffffff)
49363: process exit, rval = 4294967295

edit: full output

am11 commented 7 months ago

I see. In my case it works fine with 13.2-RELEASE jain (sitting on 14.0 host):

$ uname -a
FreeBSD f132.my.domain 13.2-RELEASE FreeBSD 14.0-RELEASE #0 releng/14.0-n265380-f9716eee8ab4: Fri Nov 10 05:57:23 UTC 2023     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

$  freebsd-version
13.2-RELEASE

$ truss -f ./a.out
26962: mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 53112616255488 (0x304e3ea00000)
26962: mprotect(0x19e25b164000,4096,PROT_READ)   = 0 (0x0)
26962: issetugid()               = 0 (0x0)
26962: sigfastblock(0x1,0x19e25b166990)      = 0 (0x0)
26962: open("/etc/libmap.conf",O_RDONLY|O_CLOEXEC,07650030030) = 3 (0x3)
26962: fstat(3,{ mode=-rw-r--r-- ,inode=50,size=47,blksize=4096 }) = 0 (0x0)
26962: read(3,"# $FreeBSD$\nincludedir /usr/loc"...,47) = 47 (0x2f)
26962: close(3)                  = 0 (0x0)
26962: open("/usr/local/etc/libmap.d",O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,0165) ERR#2 'No such file or directory'
26962: open("/var/run/ld-elf.so.hints",O_RDONLY|O_CLOEXEC,010002363) = 3 (0x3)
26962: read(3,"Ehnt\^A\0\0\0\M^@\0\0\0G\0\0\0\0"...,128) = 128 (0x80)
26962: fstat(3,{ mode=-r--r--r-- ,inode=385077,size=199,blksize=4096 }) = 0 (0x0)
26962: pread(3,"/lib:/usr/lib:/usr/lib/compat:/u"...,71,0x80) = 71 (0x47)
26962: close(3)                  = 0 (0x0)
26962: open("/lib/libc.so.7",O_RDONLY|O_CLOEXEC|O_VERIFY,012320443000) = 3 (0x3)
26962: fstat(3,{ mode=-r--r--r-- ,inode=26910,size=1958208,blksize=131072 }) = 0 (0x0)
26962: mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34941075456 (0x822a68000)
26962: mmap(0x0,4206592,PROT_NONE,MAP_GUARD,-1,0x0) = 34923323392 (0x82197a000)
26962: mmap(0x82197a000,540672,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0) = 34923323392 (0x82197a000)
26962: mmap(0x8219fe000,1359872,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x83000) = 34923864064 (0x8219fe000)
26962: mmap(0x821b4a000,40960,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1ce000) = 34925223936 (0x821b4a000)
26962: mmap(0x821b54000,28672,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1d7000) = 34925264896 (0x821b54000)
26962: mmap(0x821b5b000,2236416,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0) = 34925293568 (0x821b5b000)
26962: munmap(0x822a68000,4096)          = 0 (0x0)
26962: close(3)                  = 0 (0x0)
26962: mprotect(0x821b4a000,36864,PROT_READ)     = 0 (0x0)
26962: sysarch(AMD64_SET_FSBASE,0x8210fdfb0)     = 0 (0x0)
26962: mprotect(0x821b4a000,36864,PROT_READ|PROT_WRITE) = 0 (0x0)
26962: mprotect(0x821b4a000,36864,PROT_READ)     = 0 (0x0)
26962: readlink("/etc/malloc.conf",0x8210fd6a0,1024) ERR#2 'No such file or directory'
26962: issetugid()               = 0 (0x0)
26962: mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 53112618352640 (0x304e3ec00000)
26962: clock_gettime(4,{ 146856.753862790 })     = 0 (0x0)
26962: clock_gettime(4,{ 146856.754531870 })     = 0 (0x0)
26962: clock_gettime(4,{ 146856.755200949 })     = 0 (0x0)
26962: mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 53112620449792 (0x304e3ee00000)
26962: mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 53112622546944 (0x304e3f000000)
26962: mprotect(0x202000,4096,PROT_READ)     = 0 (0x0)
26962: getpid()                  = 26962 (0x6952)
26962: cpuset_getaffinity(0x3,0x9,0x6952,0x20,0x8210fead8) = 0 (0x0)
26962: exit(0x0)                
26962: process exit, rval = 0

Could you give this a try:

--- repro.c
+++ repro.c

int main() {
    cpu_set_t cpuSet;
+    pid_t pid = getpid();
-    return sched_getaffinity(getpid(), sizeof(cpu_set_t), &cpuSet);
+    return cpuset_getaffinity(CPU_LEVEL_WHICH, CPU_WHICH_TIDPID, pid == 0 ? -1 : pid, sizeof(cpu_set_t), &cpuSet);
}

if it works, we can add it as a fallback in PAL.

Thefrank commented 7 months ago

Running under TrueNAS CORE host. Jail images were pulled from FreeBSD mirrors.

first setup as earlier (13.1 jail on 13.1 host)

# cc repro.c
repro.c:7:48: error: use of undeclared identifier 'CPU_WHICH_TIDPID'
    return cpuset_getaffinity(CPU_LEVEL_WHICH, CPU_WHICH_TIDPID, pid == 0 ? -1 : pid, sizeof(cpu_set_t), &cpuSet);
                                               ^
1 error generated.

second setup as earlier (13.2 jail on 13.1 host)

# cc repro.c
# truss -f ./a.out 
66563: mmap(0x0,135168,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34361978880 (0x800223000)
66563: mprotect(0x800220000,4096,PROT_READ)      = 0 (0x0)
66563: issetugid()                               = 0 (0x0)
66563: sigfastblock(0x1,0x800222990)             = 0 (0x0)
66563: open("/etc/libmap.conf",O_RDONLY|O_CLOEXEC,010460030) = 3 (0x3)
66563: fstat(3,{ mode=-rw-r--r-- ,inode=27136,size=47,blksize=4096 }) = 0 (0x0)
66563: read(3,"# $FreeBSD$\nincludedir /usr/loc"...,47) = 47 (0x2f)
66563: close(3)                                  = 0 (0x0)
66563: open("/usr/local/etc/libmap.d",O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC,0165) ERR#2 'No such file or directory'
66563: open("/var/run/ld-elf.so.hints",O_RDONLY|O_CLOEXEC,010002364) = 3 (0x3)
66563: read(3,"Ehnt\^A\0\0\0\M^@\0\0\0G\0\0\0\0"...,128) = 128 (0x80)
66563: fstat(3,{ mode=-r--r--r-- ,inode=93655,size=199,blksize=4096 }) = 0 (0x0)
66563: pread(3,"/lib:/usr/lib:/usr/lib/compat:/u"...,71,0x80) = 71 (0x47)
66563: close(3)                                  = 0 (0x0)
66563: open("/lib/libc.so.7",O_RDONLY|O_CLOEXEC|O_VERIFY,020443000) = 3 (0x3)
66563: fstat(3,{ mode=-r--r--r-- ,inode=287404,size=1958576,blksize=131072 }) = 0 (0x0)
66563: mmap(0x0,4096,PROT_READ,MAP_PRIVATE|MAP_PREFAULT_READ,3,0x0) = 34362114048 (0x800244000)
66563: mmap(0x0,4206592,PROT_NONE,MAP_GUARD,-1,0x0) = 34362118144 (0x800245000)
66563: mmap(0x800245000,540672,PROT_READ,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x0) = 34362118144 (0x800245000)
66563: mmap(0x8002c9000,1359872,PROT_READ|PROT_EXEC,MAP_PRIVATE|MAP_FIXED|MAP_NOCORE|MAP_PREFAULT_READ,3,0x83000) = 34362658816 (0x8002c9000)
66563: mmap(0x800415000,40960,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1ce000) = 34364018688 (0x800415000)
66563: mmap(0x80041f000,28672,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_PREFAULT_READ,3,0x1d7000) = 34364059648 (0x80041f000)
66563: mmap(0x800426000,2236416,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_FIXED|MAP_ANON,-1,0x0) = 34364088320 (0x800426000)
66563: munmap(0x800244000,4096)                  = 0 (0x0)
66563: close(3)                                  = 0 (0x0)
66563: mprotect(0x800415000,36864,PROT_READ)     = 0 (0x0)
66563: sysarch(AMD64_SET_FSBASE,0x7fffffffdf70)  = 0 (0x0)
66563: mprotect(0x800415000,36864,PROT_READ|PROT_WRITE) = 0 (0x0)
66563: mprotect(0x800415000,36864,PROT_READ)     = 0 (0x0)
66563: readlink("/etc/malloc.conf",0x7fffffffd660,1024) ERR#2 'No such file or directory'
66563: issetugid()                               = 0 (0x0)
66563: mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 34368126976 (0x800800000)
66563: mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34370224128 (0x800a00000)
66563: mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 34372321280 (0x800c00000)
66563: mprotect(0x202000,4096,PROT_READ)         = 0 (0x0)
66563: getpid()                                  = 66563 (0x10403)
66563: cpuset_getaffinity(0x3,0x9,0x10403,0x20,0x7fffffffea98) ERR#22 'Invalid argument'
66563: exit(0xffffffff)
66563: process exit, rval = 4294967295

From what I remember sched_getaffinity just ends up at cpuset_getaffinity eventually as it is mostly for linux or posix compatibility. The former is still missing documentation so I am not 100% sure.

This appears to be a case where the jail is expecting a behavior different than what the host provides. The jail uses the host's kernel. FreeBSD's symbol versioning can work around this most of the time but that only works for older jail on newer host not the other way around. The symbols either don't exist on the older host or they do but are not marked as needing versioning.

This is made a bit more complex as TrueNAS CORE is not patch-for-patch the same version as FreeBSD. I am unsure which upstream branch TrueNAS CORE tracks.

uname from TrueNAS Core then FreeBSD 13.3-RELENG

FreeBSD truenas.local 13.1-RELEASE-p9 FreeBSD 13.1-RELEASE-p9 n245429-296d095698e TRUENAS amd64
FreeBSD freebsd13.local 13.3-RELEASE FreeBSD 13.3-RELEASE releng/13.3-n257428-80d2b634ddf0 GENERIC amd64

This is starting to look more like behavior from running newer os on older kernel but I am also willing to keep looking into this. I should have more time later this week ~to try and build dotnet8 directly on the two jail configurations.~ not necessary(?)

am11 commented 7 months ago

CPU_WHICH_TIDPID -> CPU_WHICH_PID for older versions. (for me TIDPID compiles on 13.2)

Thefrank commented 7 months ago

CPU_WHICH_TIDPID -> CPU_WHICH_PID for older versions. (for me TIDPID compiles on 13.2)

Resolves the failure on the older one

33705: cpuset_getaffinity(0x3,0x2,0x83a9,0x20,0x7fffffffea98) = 0 (0x0)
33705: exit(0x0)
33705: process exit, rval = 0

am11 commented 7 months ago

@Thefrank, that looks promising. If you have time, could you try applying this patch https://github.com/dotnet/runtime/compare/main...am11:feature/freebsd-port/getaffinity and test the runtime build? Otherwise, I'll try it over the weekend. Assuming this was the only issue, we can put up the PR.

Thefrank commented 7 months ago

I am hitting this on native builds both with and without the patch when building from HEAD (for me it was: https://github.com/dotnet/runtime/commit/ffea258d50ed9f8c545c5e7393013ebfe282cf9c)

./build.sh -c Release -os freebsd -arch x64 -ci --restore --build /p:OfficialBuildId=20240321.99

  crossgen2_publish -> /root/runtime/artifacts/bin/crossgen2_publish/x64/Release/crossgen2.dll
  Generating native code
  Segmentation fault (core dumped)
/root/runtime/artifacts/bin/coreclr/freebsd.x64.Release/build/Microsoft.NETCore.Native.targets(310,5): error MSB3073: The command ""/root/runtime/artifacts/bin/coreclr/freebsd.x64.Release/ilc-published/ilc" @"/root/runtime/artifacts/obj/coreclr/crossgen2_publish/freebsd.x64.Release/native/crossgen2.ilc.rsp"" exited with code 139. [/root/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
##vso[task.logissue type=error;sourcepath=/root/runtime/artifacts/bin/coreclr/freebsd.x64.Release/build/Microsoft.NETCore.Native.targets;linenumber=310;columnnumber=5;code=MSB3073;](NETCORE_ENGINEERING_TELEMETRY=Build) The command ""/root/runtime/artifacts/bin/coreclr/freebsd.x64.Release/ilc-published/ilc" @"/root/runtime/artifacts/obj/coreclr/crossgen2_publish/freebsd.x64.Release/native/crossgen2.ilc.rsp"" exited with code 139.

Build FAILED.

/root/runtime/artifacts/bin/coreclr/freebsd.x64.Release/build/Microsoft.NETCore.Native.targets(310,5): error MSB3073: The command ""/root/runtime/artifacts/bin/coreclr/freebsd.x64.Release/ilc-published/ilc" @"/root/runtime/artifacts/obj/coreclr/crossgen2_publish/freebsd.x64.Release/native/crossgen2.ilc.rsp"" exited with code 139. [/root/runtime/src/coreclr/tools/aot/crossgen2/crossgen2_publish.csproj]
    0 Warning(s)
    1 Error(s)

Will try again both from crossbuild from linux and from net9 preview2 tag and edit this post

edit: crossbuild finishes but I don't recall if uses crossgen. Is cross-os supported? As a side note, the generated runtime.freebsd-x64.Microsoft.DotNet.ILCompiler......nupkg contains linux ELFs instead of FreeBSD ELFs making crossbuilds less useful for future native builds

edit2: net9p2 runs into the same(?) issue at crossgen2

edit3: crossbuild taken from net9p2 + https://github.com/dotnet/runtime/compare/main...am11:feature/freebsd-port/getaffinity under 3 different conditions

root@fbsd:~ # freebsd-version;uname -a
13.2-RELEASE-p9
FreeBSD fbsd 13.1-RELEASE-p9 FreeBSD 13.1-RELEASE-p9 n245429-296d095698e TRUENAS amd64
root@fbsd:~ # elfctl -e +noaslr test/dotnet 
root@fbsd:~ # ./test/dotnet --info
Failed to create CoreCLR, HRESULT: 0x8007054F

Host:
  Version:      9.0.0-preview.2.24128.5
  Architecture: x64
  Commit:       8e5e748c5c
  RID:          freebsd-x64

.NET SDKs installed:
  9.0.100-preview.2.24157.14 [/root/test/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 9.0.0-preview.2.24128.4 [/root/test/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 9.0.0-preview.2.24128.5 [/root/test/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

  --------------------------------
root@freebsd13:~ # elfctl -e +noaslr test/dotnet 
root@freebsd13:~ # freebsd-version;uname -a
13.1-RELEASE-p5
FreeBSD freebsd13 13.1-RELEASE-p9 FreeBSD 13.1-RELEASE-p9 n245429-296d095698e TRUENAS amd64
root@freebsd13:~ # ./test/dotnet --info
.NET SDK:
 Version:           9.0.100-preview.2.24157.14
 Commit:            f7466905f9
 Workload version:  9.0.100-manifests.04914b26
 MSBuild version:   17.10.0-preview-24127-03+6f44380e4

Runtime Environment:
 OS Name:     FreeBSD
 OS Version:  13
 OS Platform: FreeBSD
 RID:         freebsd-x64
 Base Path:   /root/test/sdk/9.0.100-preview.2.24157.14/

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      9.0.0-preview.2.24128.5
  Architecture: x64
  Commit:       8e5e748c5c

.NET SDKs installed:
  9.0.100-preview.2.24157.14 [/root/test/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 9.0.0-preview.2.24128.4 [/root/test/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 9.0.0-preview.2.24128.5 [/root/test/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

  -------------------------------
root@freebsd:~ # freebsd-version;uname -a
13.3-RELEASE
FreeBSD freebsd 13.3-RELEASE FreeBSD 13.3-RELEASE releng/13.3-n257428-80d2b634ddf0 GENERIC amd64
root@freebsd:~ # elfctl -e +noaslr test/dotnet 
root@freebsd:~ # ./test/dotnet --info
.NET SDK:
 Version:           9.0.100-preview.2.24157.14
 Commit:            f7466905f9
 Workload version:  9.0.100-manifests.04914b26
 MSBuild version:   17.10.0-preview-24127-03+6f44380e4

Runtime Environment:
 OS Name:     FreeBSD
 OS Version:  13
 OS Platform: FreeBSD
 RID:         freebsd-x64
 Base Path:   /root/test/sdk/9.0.100-preview.2.24157.14/

.NET workloads installed:
There are no installed workloads to display.

Host:
  Version:      9.0.0-preview.2.24128.5
  Architecture: x64
  Commit:       8e5e748c5c

.NET SDKs installed:
  9.0.100-preview.2.24157.14 [/root/test/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 9.0.0-preview.2.24128.4 [/root/test/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 9.0.0-preview.2.24128.5 [/root/test/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download