Failed to build tests for libs on Riscv64

[ashaurtaev]

After https://github.com/dotnet/runtime/pull/95980 we get next output during build of CoreCLR

  Determining projects to restore...
  Tool 'coverlet.console' (version '6.0.0') was restored. Available commands: coverlet
  Tool 'dotnet-reportgenerator-globaltool' (version '5.0.2') was restored. Available commands: reportgenerator
  Tool 'microsoft.dotnet.xharness.cli' (version '9.0.0-prerelease.24077.1') was restored. Available commands: xharness
  Tool 'microsoft.visualstudio.slngen.tool' (version '11.1.0') was restored. Available commands: slngen

  Restore was successful.
  All projects are up-to-date for restore.
  Determining projects to restore...
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
  Nothing to do. None of the projects specified contain packages to restore.

Build FAILED.

Build commands:

ROOTFS_DIR=/home/runtime/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset host.native+clr.native+clr.corelib+clr.tools+clr.nativecorelib+clr.packages+clr.nativeaotlibs+clr.crossarchtools+libs.native+libs.sfx+libs.oob /p:EnableSourceLink=false

After the first command the build completes successfully. The error occurs in the following command:

ROOTFS_DIR=/home/runtime/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset libs.pretest+libs.tests --testscope all /p:EnableSourceLink=false /p:UseLocalAppHostPack=true

Please help to fix it.

cc @gbalykov

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure See info in area-owners.md if you want to be subscribed.

Issue Details

---

After https://github.com/dotnet/runtime/pull/95980 we get next output during build of CoreCLR ``` Determining projects to restore... Tool 'coverlet.console' (version '6.0.0') was restored. Available commands: coverlet Tool 'dotnet-reportgenerator-globaltool' (version '5.0.2') was restored. Available commands: reportgenerator Tool 'microsoft.dotnet.xharness.cli' (version '9.0.0-prerelease.24077.1') was restored. Available commands: xharness Tool 'microsoft.visualstudio.slngen.tool' (version '11.1.0') was restored. Available commands: slngen Restore was successful. All projects are up-to-date for restore. Determining projects to restore... /home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj] /home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj] /home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj] /home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj] Nothing to do. None of the projects specified contain packages to restore. Build FAILED. ``` Build commands: ``` ROOTFS_DIR=/home/runtime/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset host.native+clr.native+clr.corelib+clr.tools+clr.nativecorelib+clr.packages+clr.nativeaotlibs+clr.crossarchtools+libs.native+libs.sfx+libs.oob /p:EnableSourceLink=false ``` After the first command the build completes successfully. The error occurs in the following command: ``` ROOTFS_DIR=/home/runtime/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset libs.pretest+libs.tests --testscope all /p:EnableSourceLink=false /p:UseLocalAppHostPack=true ``` Please help to fix it. cc @gbalykov

Author:	ashaurtaev
Assignees:	-
Labels:	`area-Infrastructure`, `untriaged`, `arch-riscv`
Milestone:	-

[ashaurtaev]

@ViktorHofer please help, what should I do?

[ViktorHofer]

@dotnet/runtime-infrastructure can you please help? Seems to be host and prebuilt related. Possibly a component that tries to use a live built host?

[tomeksowi]

Possibly a component that tries to use a live built host?

The one that I know of is src/libraries/System.Runtime.InteropServices/tests/TestAssets/NativeExports/NativeExports.csproj, at least that's what the /p:UseLocalAppHostPack=true was meant to override; not doing so resulted in ComInterfaceGenerator.Tests and LibraryImportGenerator.Tests try to use .so's for x64.

But I don't see it anywhere in build log with --verbosity diag and it also fails without /p:UseLocalAppHostPack=true, so not sure if it's related to this failure.

[ashaurtaev]

If option /p:UseLocalAppHostPack=true is not added, then the error still occurs

ROOTFS_DIR=/home/runtime/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset libs.pretest+libs.tests --testscope all /p:EnableSourceLink=false

  Determining projects to restore...
  Tool 'coverlet.console' (version '6.0.0') was restored. Available commands: coverlet
  Tool 'dotnet-reportgenerator-globaltool' (version '5.0.2') was restored. Available commands: reportgenerator
  Tool 'microsoft.dotnet.xharness.cli' (version '9.0.0-prerelease.24077.1') was restored. Available commands: xharness
  Tool 'microsoft.visualstudio.slngen.tool' (version '11.1.0') was restored. Available commands: slngen

  Restore was successful.
  All projects are up-to-date for restore.
  Determining projects to restore...
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
/home/runtime/.dotnet/sdk/9.0.100-alpha.1.23615.4/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(157,5): error NETSDK1084: There is no application host available for the specified RuntimeIdentifier 'linux-riscv64'. [/home/runtime/Build.proj]
  Nothing to do. None of the projects specified contain packages to restore.

Build FAILED.

[ashaurtaev]

@dotnet/area-infrastructure-libraries Could you take a look, please?

[am11]

@ViktorHofer, this is an variant of https://github.com/dotnet/runtime/issues/58109. The build flow is twisted, and we need to build it like this:

# using mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-22.04-cross-riscv64

$ export ROOTFS_DIR=/crossrootfs/riscv64

# clr cross tools depend on live apphsot and host managed depends on something from sfxproj..
# so we build host.native first
$ ./build.sh host.native --cross --arch riscv64 -c Release

# now build clr+libs+tests
$ ./build.sh clr+libs --cross --arch riscv64 --runtimeConfiguration Release \
    --librariesConfiguration Release --hostConfiguration Release \
    --subset libs.pretest+libs.tests --testscope all -p:UseLocalAppHostPack=true

# now build packs 
$ ./build.sh packs --cross --arch riscv64 -c Release

# finally the host
$ ./build.sh host --cross --arch riscv64 -c Release

I think linux-riscv64 at this point (while it's still a community-supported platform) provides a good opportunity to streamline the subset dependencies, because on other archs apphost gets pulled from SDK (accidentally?) in places where we don't expect.

[clamp03]

cc @dotnet/samsung

[gbalykov]

Original issue with libs tests build was fixed at some point (not sure about exact commit), but before that similar problem with coreclr tests started to happen as I mentioned in https://github.com/dotnet/runtime/pull/96941#discussion_r1507143464.

Now clr, libs and libs tests build works with next commands without any issues:

ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset host.native+clr.native+clr.corelib+clr.tools+clr.nativecorelib+clr.packages+clr.nativeaotlibs+clr.crossarchtools+libs.native+libs.sfx+libs.oob /p:EnableSourceLink=false

ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset libs.pretest+libs.tests --testscope all /p:EnableSourceLink=false /p:UseLocalAppHostPack=true

However, trying to build clr tests with next command fails:

ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 BuildAsStandalone=true ./src/tests/build.sh -Release -cross -riscv64 -clang -priority1

error (same as @clamp03 mentioned in https://github.com/dotnet/runtime/pull/98476#issuecomment-1978047139):

    Determining projects to restore...
/home/runtime/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj : error NU1101: Unable to find package Microsoft.NETCore.App.Runtime.linux-riscv64. No packages exist with this id in source(s): /home/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/FSharp/library-packs, dotnet-eng, dotnet-libraries, dotnet-libraries-transport, dotnet-public, dotnet-tools, dotnet9, dotnet9-transport [/home/runtime/src/tests/build.proj]
/home/runtime/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj : error NU1101: Unable to find package Microsoft.NETCore.App.Host.linux-riscv64. No packages exist with this id in source(s): /home/runtime/.dotnet/sdk/9.0.100-preview.1.24101.2/FSharp/library-packs, dotnet-eng, dotnet-libraries, dotnet-libraries-transport, dotnet-public, dotnet-tools, dotnet9, dotnet9-transport [/home/runtime/src/tests/build.proj]
    Failed to restore /home/runtime/src/tests/Common/test_dependencies_fs/test_dependencies.fsproj (in 19.4 sec).
/home/runtime/src/tests/build.proj(548,5): error MSB3073: The command ""/home/runtime/.dotnet/dotnet" restore Common/test_dependencies_fs/test_dependencies.fsproj  /p:SetTFMForRestore=true /p:TargetOS=linux /p:TargetArchitecture=riscv64 /p:Configuration=Release /p:CrossBuild=true" exited with code 1.

@am11 do you know what can help with this one?

[ViktorHofer]

@am11 do you know what can help with this one?

@jkoritzinsky @hoyosjs @trylek might be able to help

[am11]

@gbalykov, please try applying:

--- a/src/tests/build.proj
+++ b/src/tests/build.proj
@@ -543,6 +543,7 @@
   <Target Name="RestorePackage">
     <PropertyGroup>
       <_ConfigurationProperties>/p:TargetOS=$(TargetOS) /p:TargetArchitecture=$(TargetArchitecture) /p:Configuration=$(Configuration) /p:CrossBuild=$(CrossBuild)</_ConfigurationProperties>
+      <_ConfigurationProperties Condition="'$(UseLocalAppHostPack)' == 'true'">$(_ConfigurationProperties) -p:EnableAppHostPackDownload=false -p:EnableTargetingPackDownload=false -p:EnableRuntimePackDownload=false</_ConfigurationProperties>
       <DotnetRestoreCommand>"$(DotNetTool)" restore $(RestoreProj) $(PackageVersionArg) /p:SetTFMForRestore=true $(_ConfigurationProperties)</DotnetRestoreCommand>
     </PropertyGroup>
     <Exec Command="$(DotnetRestoreCommand)"/>

and add -p:UseLocalAppHostPack=true at the end of your call:

ROOTFS_DIR=pwd/.tools/rootfs/riscv64 BuildAsStandalone=true ./src/tests/build.sh -Release -cross -riscv64 -clang -priority1

[gbalykov]

@am11 Thanks for quick response, your fix did the trick! Now clr and libs tests can be built for riscv64 without any issues (I checked on f84d33c7b8313d36f31f686920c7e5e9f038b60e). Will you open PR for your patch?

For reference, he's a set of commands to build coreclr, libs, coreclr tests and libs tests for riscv64:

# Build coreclr + libs
ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset host.native+clr.native+clr.corelib+clr.tools+clr.nativecorelib+clr.packages+clr.nativeaotlibs+clr.crossarchtools+libs.native+libs.sfx+libs.oob /p:EnableSourceLink=false

# Build libs tests
ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --runtimeConfiguration Release --librariesConfiguration Release --hostConfiguration Release --subset libs.pretest+libs.tests --testscope all /p:EnableSourceLink=false /p:UseLocalAppHostPack=true

# Build coreclr tests
ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 BuildAsStandalone=true ./src/tests/build.sh -Release -cross -riscv64 -clang -priority1 -p:UseLocalAppHostPack=true

[ViktorHofer]

I wonder if this simpler command also works. Would you mind trying?

# Build coreclr + libs
ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --configuration Release --subset host.native+clr.native+clr.corelib+clr.tools+clr.nativecorelib+clr.packages+clr.nativeaotlibs+clr.crossarchtools+libs /p:EnableSourceLink=false

# Build libs tests
ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 ./build.sh --cross --clang --arch riscv64 --configuration Release --subset libs.tests --testscope all /p:EnableSourceLink=false /p:UseLocalAppHostPack=true

# Build coreclr tests
ROOTFS_DIR=`pwd`/.tools/rootfs/riscv64 BuildAsStandalone=true ./src/tests/build.sh -Release -cross -riscv64 -clang -priority1 -p:UseLocalAppHostPack=true

[am11]

Great! Yes, I will send a PR shortly. This seems like a stopgap solution. I think the proper solution would be to integrate eng/targetingpacks.targets in src/tests and adjust the subset dependencies, so we can utilize apphost live built in various subsets without going through the sdk overhead. Also that "$(DotNetTool)" restore part can probably be replaced by <MSBuild Targets="Restore" to avoid shelling out the restore task.

[gbalykov]

I wonder if this simpler command also works. Would you mind trying?

This also works

[am11]

@gbalykov side question, on riscv64 machine, looks like we need the SDK to execute tests using src/tests/run.sh script. Is there any way to skip SDK requirement to run all tests? I can individually run e.g. MathRoundDouble_ro.sh described here https://github.com/dotnet/runtime/blob/c723f067fddf39528384d65cd79e8a86197555e2/docs/workflow/testing/coreclr/testing.md#running-individual-tests.

I downloaded the three artifacts_part.. files from today's build: https://github.com/am11/CrossRepoCITesting/releases/tag/linux-riscv64_10307028129, joined them cat ~/Downloads/artifacts_part* > artifacts.tar.gz, extracted it in runtime git repo directory.

[tomeksowi]

on riscv64 machine, looks like we need the SDK to execute tests using src/tests/run.sh script.

Yes. However, @SzpejnaDawid recently made some progress with RISC-V SDK (and even a successful build).

Is there any way to skip SDK requirement to run all tests?

We used src/tests/Common/scripts/bringup_runtest.sh during early development, it's pure bash, no SDK necessary. However, it doesn't respect exclusions in src/tests/issues.targets so the runs have false positives. For CI we rolled out our own runner scripts at least until RISC-V SDK is in good enough shape to use src/tests/run.sh.

[am11]

Thanks! I was thinking about doing a full source build (dotnet/dotnet) but wanted to get early results on the device with runtime build only. Running it with find -exec command atm, will grep the stats and share the logs once it’s done.

[gbalykov]

@am11 as @tomeksowi mentioned you can use corerun host to run clr tests using src/tests/Common/scripts/bringup_runtest.sh. We also use custom version of this script for coreclr testing, but approach is same using .sh files for each test. Same for libraries tests, we use custom version of testing script, but on lower level it either uses corerun or dotnet host (you can manually pack artifacts of host.native, clr and libs) with xunit.

By the way, this issue is resolved, I think it can be closed.

[am11]

Yup, I added https://github.com/am11/CrossRepoCITesting/blob/2d01552673ca8f3f2fe8699076e85d441fb8c437/.github/workflows/linux-riscv64-runtime-build.yml#L31, lets see if that satisfies all the requirements for bringup script. GHA has size cap, so it requires us to be a bit mindful to not let obj dir grow too large.

I used a rudimentary find artifacts/tests -name '*.sh' -exec bash -c 'pushd $(dirname {}); ./$(basename {}) -coreroot=$HOME/projects/runtime/artifacts/tests/coreclr/linux.riscv64.Release/Tests/Core_Root; popd' \; | tee results.txt but it got stuck when running a large array test (which is excluded in issues.targets). I'll try to run with bringup script and see if I can figure out a way to ignore tests for this scenario. Maybe the issues.targets can create a pre-rendered issues.txt flat list in artifacts/tests on build, which this .sh could parse? I will see if I can figure out something to help other community platforms.

By the way, this issue is resolved, I think it can be closed.

I was actually investigating the TODO: integrate eng/targetingpacks.targets in src/tests/build.proj, but it's more involved than I thought. That TODO is related to this issue. I will get back to it once I get some things off my table. Meanwhile if someone else want to give it a go, here is the wip branch: https://github.com/dotnet/runtime/compare/main...am11:runtime:feature/cleanups/targetingpacks (will update tonight when I am on my computer; alpine build fixes were merged upstream).