search

dotnet / source-build

A repository to track efforts to produce a source tarball of the .NET Core SDK and all its components
MIT License
265 stars 130 forks source link

Running --minimal smoke-tests in Ubuntu 22.04 #2824

Closed mirespace closed 2 years ago

mirespace commented 2 years ago

Hi, I'm trying to use the smoke tests in https://github.com/dotnet/installer/tree/release/6.0.1xx/src/SourceBuild/tarball/content/test/Microsoft.DotNet.SourceBuild.SmokeTests as building tests for dotnet packaging in Ubuntu 22.04 using tag 6.0.103-source.build. I know this is kind of legacy as per https://github.com/dotnet/source-build/issues/1654, but by now is what I'm executing.

I got some problems executing it from the building. First was the smoke-test.sh file, I had to correct the artifacts folder:

--- a/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh
+++ b/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh
@@ -7,7 +7,7 @@ VERSION_PREFIX=6.0
 # See https://github.com/dotnet/source-build/issues/579, this version
 # needs to be compatible with the runtime produced from source-build
 DEV_CERTS_VERSION_DEFAULT=6.0.0-preview.6.21355.2
-ARTIFACTS_DIR="$SCRIPT_ROOT/../../../../../../../artifacts/"
+ARTIFACTS_DIR="$SCRIPT_ROOT/../../../../artifacts/"
 executingUserHome=${HOME:-}

 export DOTNET_CLI_TELEMETRY_OPTOUT=1

After that, If I execute manually it (./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64) is all OK -log attached all-tests.log - (except for me because of the part where it's downloading stuff to be able to run the tests).

So, let's go to the problems:

  1. Trying the options minimal, excluding omnisharptest and onlinetest into smoke-test.sh to see if it can only test local stuff... It doesn't. Executing ./assets/smoke-tests/smoke-test.sh --dotnetDir {Directory.GetParent(DotNetHelper.DotNetPath)} --projectOutput --archiveRestoredPackages --targetRid {Config.TargetRid} --minimal --excludeOnlineTests --excludeOmniSharpTests all resolved is ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests : 
    
[...]
build/dotnet6-KxJTNR/dotnet6-6.0.103/t

est/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -maxcpucount -property:PublishTrimmed=true -property:PublishReadyToRun=true -property:SelfContained=True -property: _CommandLineDefinedSelfContained=true -property:RuntimeIdentifier=linux-x64 -property:_CommandLineDefinedRuntimeIdentifier=true -restore -target:Publish -verbosity:m /bl:/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/logs/CSharp_classlib_local_nohttps_publish-self-contained-R2R-portable.binlog ./CSharp_classlib.csproj Determining projects to restore... Restored /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_classlib/CSharp_classlib.csproj (in 120 ms). CSharp_classlib -> /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_classlib/bin/Debug/net6.0/linux-x64/CSharp_classlib.dll Optimizing assemblies for size, which may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink CSharp_classlib -> /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_classlib/bin/Debug/net6.0/linux-x64/publish/ finished language C#, type classlib starting language C#, type xunit running new The template "xUnit Test Project" was created successfully. running restore /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -nologo -maxcpucount -target:Restore -verbosity:m /bl:/build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/logs/CSharp_xunit_local_nohttps_restore.binlog ./CSharp_xunit.csproj Determining projects to restore... /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj : error NU1101: Unable to find package Microsoft.NET.Test.Sdk. No packages exist with this id in source(s): source-built-packages /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj : error NU1101: Unable to find package xunit. No packages exist with this id in source(s): source-built-packages /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj : error NU1101: Unable to find package xunit.runner.visualstudio. No packages exist with this id in source(s): source-built-packages /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj : error NU1101: Unable to find package coverlet.collector. No packages exist with this id in source(s): source-built-packages Failed to restore /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0407084134/CSharp_xunit/CSharp_xunit.csproj (in 169 ms). make[1]: [debian/rules:132: override_dh_install] Error 1 make[1]: Leaving directory '/build/dotnet6-KxJTNR/dotnet6-6.0.103' make: [debian/rules:30: binary] Error 2

Therefore I can't use this as a workaround for the next point @MichaelSimons :( (although I'm working on how to download this stuff while creating the tarball... no good progress on this yet).

2. Executing the `VERBOSE=1 ./build.sh --run-smoke-test` allowing to download requisites (as we see it worked manually previously we suppose it has to work... it doesn't):

VERBOSE=1 ./build.sh --run-smoke-test Found bootstrap SDK 6.0.102, bootstrap Arcade 6.0.0-beta.21609.4, bootstrap SourceLink 1.1.0-beta-21480-02 Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-aZycfo/dotnet6-6.0.103/.dotnet/sdk/6.0.102/MSBuild.dll /bl:/build/dotnet6-aZycfo/dotnet6-6.0.103/artifacts/log/Debug/BuildTests_0330135715.binlog /clp:v=m /fileLoggerParameters:LogFile=/build/dotnet6-aZycfo/dotnet6-6.0.103/artifacts/logs/BuildTests_0330135715.log /flp:v=detailed /t:RunSmokeTest /build/dotnet6-aZycfo/dotnet6-6.0.103/build.proj [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4' [SourceBuiltSdkResolver] Looking for SDK Microsoft.DotNet.Arcade.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4' [SourceBuiltSdkResolver] Overriding Microsoft.DotNet.Arcade.Sdk with 'ARCADE' [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4' [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4' [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'

Welcome to .NET 6.0!

SDK Version: 6.0.102

Installed an ASP.NET Core HTTPS development certificate. To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only). Learn about HTTPS: https://aka.ms/dotnet-https

Write your first app: https://aka.ms/dotnet-hello-world Find out what's new: https://aka.ms/dotnet-whats-new Explore documentation: https://aka.ms/dotnet-docs Report issues and find source on GitHub: https://github.com/dotnet/core Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli

[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4' [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4' [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4' [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4' Determining projects to restore... Restored /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.csproj (in 5 sec). [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4' [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4' [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4' [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-aZycfo/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4' Microsoft.DotNet.SourceBuild.SmokeTests -> /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll Test run for /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (.NETCoreApp,Version=v6.0) Microsoft (R) Test Execution Command Line Tool Version 17.0.0 Copyright (c) Microsoft Corporation. All rights reserved.

Starting test execution, please wait... A total of 1 test files matched the specified pattern. [xUnit.net 00:39:36.88] Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [FAIL] Failed Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [39 m 36 s] EXEC : error Message: [/build/dotnet6-aZycfo/dotnet6-6.0.103/build.proj] Assert.Equal() Failure Expected: 0 Actual: 127 Stack Trace: at Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript() in /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs:line 43 Standard Output Messages: Executing: ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64

Results File: /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/TestResults/_Camelot-TITAN_2022-03-30_13_57_28.trx

Failed! - Failed: 1, Passed: 3, Skipped: 0, Total: 4, Duration: 39 m 33 s - /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (net6.0) /build/dotnet6-aZycfo/dotnet6-6.0.103/build.proj(98,5): error MSB3073: The command "/build/dotnet6-aZycfo/dotnet6-6.0.103/.dotnet/dotnet test ./test/Microsoft.DotNet.SourceBuild.SmokeTests --logger:trx -c Release" exited with code 1. make[1]: [debian/rules:129: override_dh_install] Error 1 make[1]: Leaving directory '/build/dotnet6-aZycfo/dotnet6-6.0.103' make: [debian/rules:30: binary] Error 2


How can it be possible if the command stated here ` Executing: ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-aZycfo/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 ` worked manually before ? I put some prints out in the SmokeTest.Cs file for  executeResult.StdOut, executeResult.StdErr and executeResult.ExitCode (to see where the 1 code comes from) and got the following:

testing-smoke-0406114624/CSharp_console/bin/Debug/net6.0/publish/ Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -maxcpucount -property:SelfContained=True -property:_CommandLineDefinedSelfContained=true -property:RuntimeIdentifier=ubuntu.22.04-x64 -property:_CommandLineDefinedRuntimeIdentifier=true -restore -target:Publish -verbosity:m /bl:/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/assets/smoke-tests/testing-smoke-0406114624/logs/CSharp_console_local_nohttps_publish-self-contained-ubuntu.22.04-x64.binlog ./CSharp_console.csproj Determining projects to restore... StdOut: /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : The local source '/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/artifacts/obj/x64/Release/blob-feed/packages/' doesn't exist. [/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/assets/smoke-tests/testing-smoke-0406114624/CSharp_console/CSharp_console.csproj] [/build/dotnet6-OIjAuR/dotnet6-6.0.103/build.proj] StdErr: ExitCode: 1 Failed Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [6 s] EXEC : error Message: [/build/dotnet6-OIjAuR/dotnet6-6.0.103/build.proj] Assert.Equal() Failure Expected: 0 Actual: 1 Stack Trace: at Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript() in /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs:line 47 Standard Output Messages: Executing: ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests

Results File: /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/TestResults/_Camelot-TITAN_2022-04-06_11_46_24.trx

Failed! - Failed: 1, Passed: 3, Skipped: 0, Total: 4, Duration: 6 s - /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (net6.0) /build/dotnet6-OIjAuR/dotnet6-6.0.103/build.proj(98,5): error MSB3073: The command "/build/dotnet6-OIjAuR/dotnet6-6.0.103/.dotnet/dotnet test ./test/Microsoft.DotNet.SourceBuild.SmokeTests --logger:trx -c Release" exited with code 1. make[1]: [debian/rules:131: override_dh_install] Error 1 make[1]: Leaving directory '/build/dotnet6-OIjAuR/dotnet6-6.0.103' make: [debian/rules:30: binary] Error 2


It seems another wrong artifact folder set elsewhere : `/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : The local source '/build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/artifacts/obj/x64/Release/blob-feed/packages/' doesn't exist. ` ... could it be a possibility?

Thanks in advance for looking into this.
MichaelSimons commented 2 years ago

@mirespace, I have a couple suggestions for the two different attempts you tried.

  1. Can you provide clarity on what path you are executing ./assets/smoke-tests/smoke-test.sh from? It appears that you maybe invoking the script directly out of the source directory. This isn't support as you are discovering. Try invoking it out of the build output directory /test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/assets/smoke-tests/smoke-test.sh. When doing so, revert your change to the artifacts variable.

    That said, I don't suspect this is the cause of the failure. Can checked to see if the packages being loaded exist in /build/dotnet6-KxJTNR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/../../../../artifacts/obj/x64/Release/blob-feed/packages/? From the log, this is where it is looking for the Microsoft.NET.Test.Sdk, xunit, xunit.runner.visualstudio, and coverlet.collector packages

  2. In the /build/dotnet6-OIjAuR/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/artifacts/obj/x64/Release directory there should be a testing-smoke*/logs directory for each run. Can you share the logs for the a run the demonstrates the problem?

    When attempting to run the tests via the build.sh script, did you revert you changes to modify the artifacts variable.

mirespace commented 2 years ago

Hi @MichaelSimons! I run all the process from the scratch again. To force the parameters in the .sh script, I made this patch:

diff --git a/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs b/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs
index f6a1d96b4..2c732a64b 100644
--- a/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs
+++ b/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs
@@ -24,7 +24,7 @@ public SmokeTests(ITestOutputHelper outputHelper)
     [Fact]
     public void SmokeTestsScript()
     {
-        string smokeTestArgs = $"--dotnetDir {Directory.GetParent(DotNetHelper.DotNetPath)} --projectOutput --archiveRestoredPackages --targetRid {Config.TargetRid}";
+        string smokeTestArgs = $"--dotnetDir {Directory.GetParent(DotNetHelper.DotNetPath)} --projectOutput --archiveRestoredPackages --targetRid {Config.TargetRid} --minimal --excludeOnlineTests --excludeOmniSharpTests";
         if (Config.TargetRid.Contains("osx"))
         {
             smokeTestArgs += " --excludeWebHttpsTests";
  1. The complete command is <<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh --dotnetDir /<<PKGBUILDDIR>>/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests: is executing from the top of the source tree building source (dotnet6-random as temporal folder for building):
(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103# /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests

I didn't realize when open the issue that it fails because the bin folder is not present:

/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/smoke-test.sh: line 485: /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/dotnet: No such file or directory

because it would be done by ./build.sh --run-smoke-test (I guess this is a step before invoking the smoke-test.sh script).

so, directly, it can't be called.

  1. I reverted the change to the artifacts variable this time, and I executed from the top of the source tree the build command:
(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103# VERBOSE=1 ./build.sh --run-smoke-test

It failed a first time for the xUnit.net test:

(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103# VERBOSE=1 ./build.sh --run-smoke-test
Found bootstrap SDK 6.0.102, bootstrap Arcade 6.0.0-beta.21609.4, bootstrap SourceLink 1.1.0-beta-21480-02
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-4woFhD/dotnet6-6.0.103/.dotnet/sdk/6.0.102/MSBuild.dll /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/artifacts/log/Debug/BuildTests_0413105044.binlog /clp:v=m /fileLoggerParameters:LogFile=/build/dotnet6-4woFhD/dotnet6-6.0.103/artifacts/logs/BuildTests_0413105044.log /flp:v=detailed /t:RunSmokeTest /build/dotnet6-4woFhD/dotnet6-6.0.103/build.proj
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.DotNet.Arcade.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Overriding Microsoft.DotNet.Arcade.Sdk with 'ARCADE'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
[SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'

  Welcome to .NET 6.0!
  ---------------------
  SDK Version: 6.0.102

  ----------------
  Installed an ASP.NET Core HTTPS development certificate.
  To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
  Learn about HTTPS: https://aka.ms/dotnet-https
  ----------------
  Write your first app: https://aka.ms/dotnet-hello-world
  Find out what's new: https://aka.ms/dotnet-whats-new
  Explore documentation: https://aka.ms/dotnet-docs
  Report issues and find source on GitHub: https://github.com/dotnet/core
  Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
  --------------------------------------------------------------------------------------
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
    Determining projects to restore...
    Restored /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/Microsoft.DotNet.SourceBuild.SmokeTests.csproj (in 4.2 sec).
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.Sdk. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadAutoImportPropsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.ILLink.Tasks. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
  [SourceBuiltSdkResolver] Looking for SDK Microsoft.NET.SDK.WorkloadManifestTargetsLocator. Detected config(s) in env: 'ARCADE' for 'Microsoft.DotNet.Arcade.Sdk/6.0.0-beta.21609.4' at '/build/dotnet6-4woFhD/dotnet6-6.0.103/packages/restored/ArcadeBootstrapPackage/microsoft.dotnet.arcade.sdk/6.0.0-beta.21609.4'
    Microsoft.DotNet.SourceBuild.SmokeTests -> /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll
  Test run for /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (.NETCoreApp,Version=v6.0)
  Microsoft (R) Test Execution Command Line Tool Version 17.0.0
  Copyright (c) Microsoft Corporation.  All rights reserved.

  Starting test execution, please wait...
  A total of 1 test files matched the specified pattern.
  [xUnit.net 00:00:40.08]     Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [FAIL]
    Failed Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript [39 s]
EXEC : error Message:  [/build/dotnet6-4woFhD/dotnet6-6.0.103/build.proj]
     Assert.Equal() Failure
  Expected: 0
  Actual:   1
    Stack Trace:
       at Microsoft.DotNet.SourceBuild.SmokeTests.SmokeTests.SmokeTestsScript() in /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/SmokeTests.cs:line 43
    Standard Output Messages:
   Executing: ./assets/smoke-tests/smoke-test.sh --dotnetDir /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet --projectOutput --archiveRestoredPackages --targetRid ubuntu.22.04-x64 --minimal --excludeOnlineTests --excludeOmniSharpTests

  Results File: /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/TestResults/_Camelot-TITAN_2022-04-13_10_50_56.trx

  Failed!  - Failed:     1, Passed:     3, Skipped:     0, Total:     4, Duration: 36 s - /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/Microsoft.DotNet.SourceBuild.SmokeTests.dll (net6.0)
/build/dotnet6-4woFhD/dotnet6-6.0.103/build.proj(98,5): error MSB3073: The command "/build/dotnet6-4woFhD/dotnet6-6.0.103/.dotnet/dotnet test ./test/Microsoft.DotNet.SourceBuild.SmokeTests --logger:trx -c Release" exited with code 1.

And, no, the Microsoft.NET.Test.Sdk, xunit, xunit.runner.visualstudio, and coverlet.collector packages are not in /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/assets/smoke-tests/../../../../../../../artifacts/obj/x64/Release/blob-feed/packages/. So, yes, that's the reason of failing, but... if the parameters are --minimal and --excludeOnlineTests everything needed to run the test shouldn't already be there and not need to be downloaded?

  1. The only error I see in that files is related with the artifacts folder location:
(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests# ack error testing-smoke-041*/logs/*.log
/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : The local source '/artifacts/obj/x64/Release/blob-feed/packages/' doesn't exist. [/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/CSharp_console.csproj]
(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests#

complete file is (indeed, only this log has been generated): 

(jammy-amd64-sbuild)root@Camelot-TITAN:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests# cat testing-smoke-0413110747/logs/smoke-test.log
    running new

Welcome to .NET 6.0!
---------------------
SDK Version: 6.0.103

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
The template "Console App" was created successfully.
    new succeeded
    running restore
/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -nologo -maxcpucount -target:Restore -verbosity:m /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/logs/CSharp_console_local_nohttps_restore.binlog ./CSharp_console.csproj
  Determining projects to restore...
  Restored /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/CSharp_console.csproj (in 71 ms).
    restore succeeded
    running build
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -consoleloggerparameters:Summary -maxcpucount -restore -verbosity:m /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/logs/CSharp_console_local_nohttps_build.binlog ./CSharp_console.csproj
  Determining projects to restore...
  All projects are up-to-date for restore.
  CSharp_console -> /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/bin/Debug/net6.0/CSharp_console.dll

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.02
    build succeeded
    running run
Hello, World!
    run succeeded
    running multi-rid-publish
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -maxcpucount -property:SelfContained=False -property:_CommandLineDefinedSelfContained=true -restore -target:Publish -verbosity:m /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/logs/CSharp_console_local_nohttps_publish-fx-dep.binlog ./CSharp_console.csproj
  Determining projects to restore...
  All projects are up-to-date for restore.
  CSharp_console -> /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/bin/Debug/net6.0/CSharp_console.dll
  CSharp_console -> /build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/bin/Debug/net6.0/publish/
Microsoft (R) Build Engine version 17.0.0+c9eb9dd64 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/MSBuild.dll -maxcpucount -property:SelfContained=True -property:_CommandLineDefinedSelfContained=true -property:RuntimeIdentifier=ubuntu.22.04-x64 -property:_CommandLineDefinedRuntimeIdentifier=true -restore -target:Publish -verbosity:m /bl:/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/logs/CSharp_console_local_nohttps_publish-self-contained-ubuntu.22.04-x64.binlog ./CSharp_console.csproj
  Determining projects to restore...
/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/bin/Release/net6.0/.dotnet/sdk/6.0.103/NuGet.targets(130,5): error : The local source '/artifacts/obj/x64/Release/blob-feed/packages/' doesn't exist. [/build/dotnet6-4woFhD/dotnet6-6.0.103/test/Microsoft.DotNet.SourceBuild.SmokeTests/assets/smoke-tests/testing-smoke-0413110747/CSharp_console/CSharp_console.csproj]
MichaelSimons commented 2 years ago

[Triage] Closing as the recently changes to the smoke-tests removed the script. The ability to run tests in an offline mode is tracked with https://github.com/dotnet/source-build/issues/2844.