search

dotnet / performance

This repo contains benchmarks used for testing the performance of all .NET Runtimes
MIT License
701 stars 272 forks source link

MonoAOTLLVM runs are completely failing #2100

Closed LoopedBard3 closed 3 years ago

LoopedBard3 commented 3 years ago

There is a target framework error causing benchmark to not build, in turn preventing the tests from running. The issue here is just to help track dotnet/benchmarkdotnet#1827. Full Error:

MSBUILDDEBUGCOMM=1
USER=helixbot
MSBUILDDEBUGPATH=/home/helixbot/work/B77D09F3/w/ADD6094E/uploads
PERFLAB_BUILDARCH=x64
HELIX_WORKITEM_FRIENDLYNAME=release.Partition6
HELIX_WORKITEM_ARCHIVE_ROOT=/home/helixbot/work/B77D09F3/w/ADD6094E
HELIX_CORRELATION_ID=f2888cfa-7c79-47ce-ab49-80fc37f9a15f
PERFLAB_TARGET_FRAMEWORKS=net7.0
DOTNET_CLI_TELEMETRY_OPTOUT=1
SHLVL=1
HOME=/home/helixbot
HELIX_EVENT_URI=https://helixtelemetry.servicebus.windows.net/telemetry/messages?sr=http%3A%2F%2Fhelixtelemetry.servicebus.windows.net%2Ftelemetry%2Fmessages%2F&sig=7dCdMGmXm1CvKXqeozMn8wvxADzylPUqoJIXPEYNcsE%3D&se=1636461703&skn=WorkItemTelemetryReporting
HELIX_LOG_ROOT=/home/helixbot/dotnetbuild/logs
PERFLAB_UPLOAD_TOKEN=?sv=2020-08-04&ss=bq&srt=sco&sp=rwdlacuptfx&se=2022-10-20T02:07:04Z&st=2021-10-20T18:07:04Z&spr=https&sig=9mVe%2ByoyE8dyfT9%2Frgrsdt93A%2BcOKJtSgGaoXBq3bIY%3D
PERFLAB_REPO=dotnet/runtime
PERFLAB_QUEUE=Ubuntu.1804.Amd64.Tiger.Perf
HELIX_RESULTS_CONTAINER_RSAS=?sv=2019-07-07&se=2022-01-24T12%3A41%3A38Z&sr=c&sp=rl&sig=eDgbkY3uPAnIgkAPt2cXDp0gl8u2WtGBhHL7Ku8qvFw%3D
ORIGPYPATH=/usr/bin/python3:/etc/helix/scripts:/etc/helix/scripts
DOTNET_VERSION=7.0.100-alpha.1.21518.14
PERFLAB_BRANCH=refs/heads/main
HELIX_WORKITEM_DELIVERY_COUNT=1
PERFLAB_LOCALE=en-US
LOGNAME=helixbot
JOURNAL_STREAM=9:39402
_=/usr/bin/python3
HELIX_CORRELATION_ARCHIVE_ROOT=/home/helixbot/work/B77D09F3/w
HELIX_RESULTS_CONTAINER_URI=https://helixri8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-f2888cfa7c7947ceab/release.Partition6
HELIX_RESULTS_CONTAINER_WSAS=?sv=2019-07-07&se=2021-11-05T12%3A41%3A38Z&sr=c&sp=acw&sig=XMJoaB0WyXHCG%2FlTjO7vZXBqAwmUvhlU%2FH4pyhKkjWk%3D
TEMP=/home/helixbot/work/B77D09F3/t
HELIX_DUMP_FOLDER=/home/helixbot/dotnetbuild/dumps
HELIX_SCRIPT_ROOT=/etc/helix/scripts
HELIX_WORKITEM_ROOT=/home/helixbot/work/B77D09F3/w/ADD6094E/e
PERFLAB_INLAB=1
PERFLAB_BUILDTIMESTAMP=2021-10-26T11:11:44Z
UseSharedCompilation=false
PERFLAB_PERFHASH=9eb6fe645313015fbc8853df2a7cc71e167da85c
DOTNET_ROOT=/home/helixbot/work/B77D09F3/p/dotnet
HELIX_WORKITEM_UPLOAD_ROOT=/home/helixbot/work/B77D09F3/w/ADD6094E/uploads
PATH=/home/helixbot/work/B77D09F3/p/dotnet:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
INVOCATION_ID=8eeb9d565f9543198d9ca172a8be22b0
HELIX_WORKITEM_PAYLOAD=/home/helixbot/work/B77D09F3/w/ADD6094E/u
CRYPTOGRAPHY_ALLOW_OPENSSL_102=true
DOTNET_MULTILEVEL_LOOKUP=0
LANG=en_US.UTF-8
HELIX_CONFIG_ROOT=/etc/helix-prep
TMP=/home/helixbot/work/B77D09F3/t
PERFLAB_CONFIGS=CompilationMode=tiered;RunKind=micro_mono;LLVM=false;MonoInterpreter=false;MonoAOT=true
SHELL=/bin/sh
HELIX_WORK_ROOT=/home/helixbot/work
HELIX_PYTHONPATH=/usr/bin/python3
HELIX_WORKITEM_ID=043f5c74-edb8-452e-ba6b-26691e81379b
PERFLAB_HASH=d2c19e1ce68b0784ccbe5e09c7fa3ca5ac32fba7
PWD=/home/helixbot/work/B77D09F3/w/ADD6094E/e
HELIX_CORRELATION_PAYLOAD=/home/helixbot/work/B77D09F3/p
PERFLAB_BUILDNUM=20211026.2
PYTHONPATH=
HELIX_CLIENT_INSTANCE_ID=0
HELIX_CURRENT_LOG=/home/helixbot/work/B77D09F3/w/ADD6094E/console.fdc7508b.log
+ python3 /home/helixbot/work/B77D09F3/p/performance/scripts/benchmarks_ci.py --csproj /home/helixbot/work/B77D09F3/p/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture x64 -f net7.0 --upload-to-perflab-container --dotnet-versions 7.0.100-alpha.1.21518.14 --cli-source-info args --cli-branch refs/heads/main --cli-commit-sha d2c19e1ce68b0784ccbe5e09c7fa3ca5ac32fba7 --cli-repository https://github.com/dotnet/runtime --cli-source-timestamp 2021-10-26T11:11:44Z --bdn-artifacts /home/helixbot/work/B77D09F3/p/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments=--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT --runtimes monoaotllvm --aotcompilerpath /home/helixbot/work/B77D09F3/p/monoaot/sgen/mini/mono-sgen --customruntimepack /home/helixbot/work/B77D09F3/p/monoaot/pack --aotcompilermode llvm  --partition-count 30 --partition-index 6
[2021/10/26 06:32:32][INFO] ----------------------------------------------
[2021/10/26 06:32:32][INFO] Initializing logger 2021-10-26 06:32:32.814763
[2021/10/26 06:32:32][INFO] ----------------------------------------------
[2021/10/26 06:32:32][INFO] Installing tools.
[2021/10/26 06:32:32][INFO] ----------------------
[2021/10/26 06:32:32][INFO] Downloading DotNet Cli
[2021/10/26 06:32:32][INFO] ----------------------
[2021/10/26 06:32:32][INFO] DotNet Install Path: '/home/helixbot/work/B77D09F3/p/performance/tools/dotnet/x64'
[2021/10/26 06:32:32][INFO] Downloading https://dot.net/v1/dotnet-install.sh
[2021/10/26 06:32:32][INFO] $ pushd "/home/helixbot/work/B77D09F3/p/performance"
[2021/10/26 06:32:32][INFO] $ /home/helixbot/work/B77D09F3/p/performance/tools/dotnet/x64/dotnet-install.sh -InstallDir /home/helixbot/work/B77D09F3/p/performance/tools/dotnet/x64 -Architecture x64 -Version 7.0.100-alpha.1.21518.14
[2021/10/26 06:32:32][INFO] dotnet-install: Note that the intended use of this script is for Continuous Integration (CI) scenarios, where:
[2021/10/26 06:32:32][INFO] dotnet-install: - The SDK needs to be installed without user interaction and without admin rights.
[2021/10/26 06:32:32][INFO] dotnet-install: - The SDK installation doesn't need to persist across multiple CI runs.
[2021/10/26 06:32:32][INFO] dotnet-install: To set up a development environment or to run apps, use installers rather than this script. Visit https://dotnet.microsoft.com/download to get the installer.
[2021/10/26 06:32:32][INFO] 
[2021/10/26 06:32:33][INFO] dotnet-install: .NET Core SDK version 7.0.100-alpha.1.21518.14 is already installed.
[2021/10/26 06:32:33][INFO] dotnet-install: Adding to current process PATH: `/home/helixbot/work/B77D09F3/p/performance/tools/dotnet/x64`. Note: This change will be visible only when sourcing script.
[2021/10/26 06:32:33][INFO] dotnet-install: Note that the script does not resolve dependencies during installation.
[2021/10/26 06:32:33][INFO] dotnet-install: To check the list of dependencies, go to https://docs.microsoft.com/dotnet/core/install, select your operating system and check the "Dependencies" section.
[2021/10/26 06:32:33][INFO] dotnet-install: Installation finished successfully.
[2021/10/26 06:32:33][INFO] $ popd
[2021/10/26 06:32:33][INFO] $ dotnet --info
[2021/10/26 06:32:33][INFO] .NET SDK (reflecting any global.json):
[2021/10/26 06:32:33][INFO]  Version:   7.0.100-alpha.1.21518.14
[2021/10/26 06:32:33][INFO]  Commit:    67dee0f1ec
[2021/10/26 06:32:33][INFO] 
[2021/10/26 06:32:33][INFO] Runtime Environment:
[2021/10/26 06:32:33][INFO]  OS Name:     ubuntu
[2021/10/26 06:32:33][INFO]  OS Version:  18.04
[2021/10/26 06:32:33][INFO]  OS Platform: Linux
[2021/10/26 06:32:33][INFO]  RID:         ubuntu.18.04-x64
[2021/10/26 06:32:33][INFO]  Base Path:   /home/helixbot/work/B77D09F3/p/performance/tools/dotnet/x64/sdk/7.0.100-alpha.1.21518.14/
[2021/10/26 06:32:33][INFO] 
[2021/10/26 06:32:33][INFO] Host (useful for support):
[2021/10/26 06:32:33][INFO]   Version: 7.0.0-alpha.1.21480.1
[2021/10/26 06:32:33][INFO]   Commit:  9556913857
[2021/10/26 06:32:33][INFO] 
[2021/10/26 06:32:33][INFO] .NET SDKs installed:
[2021/10/26 06:32:33][INFO]   7.0.100-alpha.1.21518.14 [/home/helixbot/work/B77D09F3/p/performance/tools/dotnet/x64/sdk]
[2021/10/26 06:32:33][INFO] 
[2021/10/26 06:32:33][INFO] .NET runtimes installed:
[2021/10/26 06:32:33][INFO]   Microsoft.AspNetCore.App 7.0.0-alpha.1.21470.28 [/home/helixbot/work/B77D09F3/p/performance/tools/dotnet/x64/shared/Microsoft.AspNetCore.App]
[2021/10/26 06:32:33][INFO]   Microsoft.NETCore.App 7.0.0-alpha.1.21480.1 [/home/helixbot/work/B77D09F3/p/performance/tools/dotnet/x64/shared/Microsoft.NETCore.App]
[2021/10/26 06:32:33][INFO] 
[2021/10/26 06:32:33][INFO] To install additional .NET runtimes or SDKs:
[2021/10/26 06:32:33][INFO]   https://aka.ms/dotnet-download
[2021/10/26 06:32:33][INFO] ---------------------------------------
[2021/10/26 06:32:33][INFO] Removing packages, bin and obj folders.
[2021/10/26 06:32:33][INFO] ---------------------------------------
[2021/10/26 06:32:33][INFO] -------------------------------
[2021/10/26 06:32:33][INFO] Restoring .NET micro benchmarks
[2021/10/26 06:32:33][INFO] -------------------------------
[2021/10/26 06:32:33][INFO] $ pushd "/home/helixbot/work/B77D09F3/p/performance/src/benchmarks/micro"
[2021/10/26 06:32:33][INFO] $ dotnet restore /home/helixbot/work/B77D09F3/p/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages /home/helixbot/work/B77D09F3/p/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
[2021/10/26 06:32:33][INFO]   Determining projects to restore...
[2021/10/26 06:32:37][INFO]   Restored /home/helixbot/work/B77D09F3/p/performance/src/tools/Reporting/Reporting/Reporting.csproj (in 3.32 sec).
[2021/10/26 06:32:39][INFO]   Restored /home/helixbot/work/B77D09F3/p/performance/src/harness/BenchmarkDotNet.Extensions/BenchmarkDotNet.Extensions.csproj (in 5.14 sec).
[2021/10/26 06:32:39][INFO]   Restored /home/helixbot/work/B77D09F3/p/performance/src/benchmarks/micro/MicroBenchmarks.csproj (in 5.16 sec).
[2021/10/26 06:32:39][INFO] $ popd
[2021/10/26 06:32:39][INFO] -------------------------------------------
[2021/10/26 06:32:39][INFO] Building .NET micro benchmarks for 'net7.0'
[2021/10/26 06:32:39][INFO] -------------------------------------------
[2021/10/26 06:32:39][INFO] $ pushd "/home/helixbot/work/B77D09F3/p/performance/src/benchmarks/micro"
[2021/10/26 06:32:39][INFO] $ dotnet build /home/helixbot/work/B77D09F3/p/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net7.0 --no-restore /p:NuGetPackageRoot=/home/helixbot/work/B77D09F3/p/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1
[2021/10/26 06:32:39][INFO] Microsoft (R) Build Engine version 17.0.0-preview-21480-01+5805e3469 for .NET
[2021/10/26 06:32:39][INFO] Copyright (C) Microsoft Corporation. All rights reserved.
[2021/10/26 06:32:39][INFO] 
[2021/10/26 06:32:40][INFO]   Reporting -> /home/helixbot/work/B77D09F3/p/performance/artifacts/bin/Reporting/Release/netstandard2.0/Reporting.dll
[2021/10/26 06:32:41][INFO]   BenchmarkDotNet.Extensions -> /home/helixbot/work/B77D09F3/p/performance/artifacts/bin/BenchmarkDotNet.Extensions/Release/netstandard2.0/BenchmarkDotNet.Extensions.dll
[2021/10/26 06:32:44][INFO]   MicroBenchmarks -> /home/helixbot/work/B77D09F3/p/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/MicroBenchmarks.dll
[2021/10/26 06:32:44][INFO] 
[2021/10/26 06:32:44][INFO] Build succeeded.
[2021/10/26 06:32:44][INFO]     0 Warning(s)
[2021/10/26 06:32:44][INFO]     0 Error(s)
[2021/10/26 06:32:44][INFO] 
[2021/10/26 06:32:44][INFO] Time Elapsed 00:00:04.46
[2021/10/26 06:32:44][INFO] $ popd
[2021/10/26 06:32:44][INFO] ------------------------------------------
[2021/10/26 06:32:44][INFO] Running .NET micro benchmarks for 'net7.0'
[2021/10/26 06:32:44][INFO] ------------------------------------------
[2021/10/26 06:32:44][INFO] --------------------------------------------------
[2021/10/26 06:32:44][INFO] Dumping COMPlus environment:
[2021/10/26 06:32:44][INFO] --------------------------------------------------
[2021/10/26 06:32:44][INFO] $ pushd "/home/helixbot/work/B77D09F3/p/performance/src/benchmarks/micro"
[2021/10/26 06:32:44][INFO] $ dotnet run --project /home/helixbot/work/B77D09F3/p/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net7.0 --no-restore --no-build -- --anyCategories Libraries Runtime "" --category-exclusion-filter NoAOT --runtimes monoaotllvm --aotcompilerpath /home/helixbot/work/B77D09F3/p/monoaot/sgen/mini/mono-sgen --customruntimepack /home/helixbot/work/B77D09F3/p/monoaot/pack --aotcompilermode llvm "" --partition-count 30 --partition-index 6 --artifacts /home/helixbot/work/B77D09F3/p/artifacts/BenchmarkDotNet.Artifacts --packages /home/helixbot/work/B77D09F3/p/performance/artifacts/packages --buildTimeout 600
[2021/10/26 06:32:45][INFO] // Validating benchmarks:
[2021/10/26 06:32:46][INFO] // ***** BenchmarkRunner: Start   *****
[2021/10/26 06:32:46][INFO] // ***** Found 119 benchmark(s) in total *****
[2021/10/26 06:32:46][INFO] // ***** Building 1 exe(s) in Parallel: Start   *****
[2021/10/26 06:32:46][INFO] // start dotnet restore --packages "/home/helixbot/work/B77D09F3/p/performance/artifacts/packages" /p:DebugType=portable -bl:benchmarkdotnet.binlog /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1 /p:Deterministic=true /p:Optimize=true in /home/helixbot/work/B77D09F3/p/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/003bc754-45a0-4405-9e0b-15a3dea7a1c4
[2021/10/26 06:32:51][INFO] // command took 5.15s and exited with 1
[2021/10/26 06:32:51][INFO] // ***** Done, took 00:00:05 (5.45 sec)   *****
[2021/10/26 06:32:51][INFO] // Found 1 benchmarks:
[2021/10/26 06:32:51][INFO] //   Burgers.Burgers_3: Job-IGYCKG(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=MonoAOTLLVM, Arguments=/p:DebugType=portable,-bl:benchmarkdotnet.binlog, Toolchain=MonoAOTLLVM, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)
[2021/10/26 06:32:51][INFO] 
[2021/10/26 06:32:51][INFO] // Build Error: The project which defines benchmarks does not target 'net6.0'.
[2021/10/26 06:32:51][INFO] You need to add 'net6.0' to <TargetFrameworks> in your project file ('MicroBenchmarks').
[2021/10/26 06:32:51][INFO] Example: <TargetFrameworks>net7.0;net6.0</TargetFrameworks>
[2021/10/26 06:32:51][INFO] 
[2021/10/26 06:32:51][INFO] // BenchmarkDotNet has failed to build the auto-generated boilerplate code.
[2021/10/26 06:32:51][INFO] // It can be found in /home/helixbot/work/B77D09F3/p/performance/artifacts/bin/MicroBenchmarks/Release/net7.0/003bc754-45a0-4405-9e0b-15a3dea7a1c4
[2021/10/26 06:32:51][INFO] // Please follow the troubleshooting guide: https://benchmarkdotnet.org/articles/guides/troubleshooting.html
[2021/10/26 06:32:51][INFO] // ***** BenchmarkRunner: Finish  *****
[2021/10/26 06:32:51][INFO] 
[2021/10/26 06:32:51][INFO] // * Export *
[2021/10/26 06:32:52][INFO]   home/helixbot/work/B77D09F3/p/artifacts/BenchmarkDotNet.Artifacts/results/Burgers-report-github.md
[2021/10/26 06:32:52][INFO]   home/helixbot/work/B77D09F3/p/artifacts/BenchmarkDotNet.Artifacts/results/Burgers-report-full.json
[2021/10/26 06:32:52][INFO]   home/helixbot/work/B77D09F3/p/artifacts/BenchmarkDotNet.Artifacts/results/Burgers-perf-lab-report.json
[2021/10/26 06:32:52][INFO] 
[2021/10/26 06:32:52][INFO] // * Detailed results *
[2021/10/26 06:32:52][INFO] Burgers.Burgers_3: Job-IGYCKG(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=MonoAOTLLVM, Arguments=/p:DebugType=portable,-bl:benchmarkdotnet.binlog, Toolchain=MonoAOTLLVM, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)
[2021/10/26 06:32:52][INFO] Runtime = ; GC =
[2021/10/26 06:32:52][INFO] There are not any results runs
[2021/10/26 06:32:52][INFO] 
[2021/10/26 06:32:52][INFO] // * Summary *
[2021/10/26 06:32:52][INFO] 
[2021/10/26 06:32:52][INFO] BenchmarkDotNet=v0.13.1.1611-nightly, OS=ubuntu 18.04
[2021/10/26 06:32:52][INFO] Intel Core i7-8700 CPU 3.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
[2021/10/26 06:32:52][INFO] .NET SDK=7.0.100-alpha.1.21518.14
[2021/10/26 06:32:52][INFO]   [Host] : .NET 7.0.0 (7.0.21.48001), X64 RyuJIT
[2021/10/26 06:32:52][INFO] 
[2021/10/26 06:32:52][INFO] PowerPlanMode=00000000-0000-0000-0000-000000000000  Runtime=MonoAOTLLVM  Arguments=/p:DebugType=portable,-bl:benchmarkdotnet.binlog
[2021/10/26 06:32:52][INFO] Toolchain=MonoAOTLLVM  IterationTime=250.0000 ms  MaxIterationCount=20
[2021/10/26 06:32:52][INFO] MinIterationCount=15  WarmupCount=1
[2021/10/26 06:32:52][INFO] 
[2021/10/26 06:32:52][INFO] |    Method | Mean | Error | Median | Min | Max |
[2021/10/26 06:32:52][INFO] |---------- |-----:|------:|-------:|----:|----:|
[2021/10/26 06:32:52][INFO] | Burgers_3 |   NA |    NA |     NA |  NA |  NA |
[2021/10/26 06:32:52][INFO] 
[2021/10/26 06:32:52][INFO] Benchmarks with issues:
[2021/10/26 06:32:52][INFO]   Burgers.Burgers_3: Job-IGYCKG(PowerPlanMode=00000000-0000-0000-0000-000000000000, Runtime=MonoAOTLLVM, Arguments=/p:DebugType=portable,-bl:benchmarkdotnet.binlog, Toolchain=MonoAOTLLVM, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)
[2021/10/26 06:32:52][INFO] 
[2021/10/26 06:32:52][INFO] // * Legends *
[2021/10/26 06:32:52][INFO]   Mean   : Arithmetic mean of all measurements
[2021/10/26 06:32:52][INFO]   Error  : Half of 99.9% confidence interval
[2021/10/26 06:32:52][INFO]   Median : Value separating the higher half of all measurements (50th percentile)
[2021/10/26 06:32:52][INFO]   Min    : Minimum
[2021/10/26 06:32:52][INFO]   Max    : Maximum
[2021/10/26 06:32:52][INFO]   1 ns   : 1 Nanosecond (0.000000001 sec)
[2021/10/26 06:32:52][INFO] 
[2021/10/26 06:32:52][INFO] // * Diagnostic Output - MemoryDiagnoser *
[2021/10/26 06:32:52][INFO] 
[2021/10/26 06:32:52][INFO] 
[2021/10/26 06:32:52][INFO] // ***** BenchmarkRunner: End *****
[2021/10/26 06:32:52][INFO] // ** Remained 118 benchmark(s) to run **
[2021/10/26 06:32:52][INFO] Run time: 00:00:00 (0.27 sec), executed benchmarks: 0
[2021/10/26 06:32:52][INFO] 
[2021/10/26 06:32:52][INFO] Global total time: 00:00:05 (5.73 sec), executed benchmarks: 0
[2021/10/26 06:32:52][INFO] // * Artifacts cleanup *
[2021/10/26 06:32:52][INFO] $ popd
[2021/10/26 06:32:52][ERROR] Process exited with status 1
[2021/10/26 06:32:52][INFO] Run failure registered
Traceback (most recent call last):
  File "/home/helixbot/work/B77D09F3/p/performance/scripts/benchmarks_ci.py", line 250, in <module>
    __main(sys.argv[1:])
  File "/home/helixbot/work/B77D09F3/p/performance/scripts/benchmarks_ci.py", line 231, in __main
    args
  File "/home/helixbot/work/B77D09F3/p/performance/scripts/micro_benchmarks.py", line 314, in run
    *run_args
  File "/home/helixbot/work/B77D09F3/p/performance/scripts/dotnet.py", line 468, in run
    self.working_directory)
  File "/home/helixbot/work/B77D09F3/p/performance/scripts/performance/common.py", line 219, in run
    returncode, quoted_cmdline)
subprocess.CalledProcessError: Command '$ dotnet run --project /home/helixbot/work/B77D09F3/p/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net7.0 --no-restore --no-build -- --anyCategories Libraries Runtime "" --category-exclusion-filter NoAOT --runtimes monoaotllvm --aotcompilerpath /home/helixbot/work/B77D09F3/p/monoaot/sgen/mini/mono-sgen --customruntimepack /home/helixbot/work/B77D09F3/p/monoaot/pack --aotcompilermode llvm "" --partition-count 30 --partition-index 6 --artifacts /home/helixbot/work/B77D09F3/p/artifacts/BenchmarkDotNet.Artifacts --packages /home/helixbot/work/B77D09F3/p/performance/artifacts/packages --buildTimeout 600' returned non-zero exit status 1.
+ export _commandExitCode=1
+ export PYTHONPATH=/usr/bin/python3:/etc/helix/scripts:/etc/helix/scripts
+ /usr/bin/python3 /home/helixbot/work/B77D09F3/p/reporter/run.py https://dev.azure.com/dnceng/ internal 41565002 eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsIng1dCI6Im9PdmN6NU1fN3AtSGpJS2xGWHo5M3VfVjBabyJ9.eyJuYW1laWQiOiJkYmNiZmQzNi1jM2IyLTQ2ZDgtYTYxZS0wOWFiNzg1MmY0YmQiLCJzY3AiOiJhcHBfdG9rZW4iLCJhdWkiOiI3MWJhYWFhMy1jMGU5LTRjMTItYTgxYy00NjUxZmZkNGM4ZjAiLCJzaWQiOiJjMDNiOGZmNi1jMzVmLTQ2MDEtYTE4Ny1hOWUxMGY2ZjMzOTAiLCJCdWlsZElkIjoiN2VhOTExNmUtOWZhYy00MDNkLWIyNTgtYjMxZmNmMWJiMjkzOzE0NDA2NjMiLCJwcGlkIjoidnN0ZnM6Ly8vQnVpbGQvQnVpbGQvMTQ0MDY2MyIsIm9yY2hpZCI6ImM5MGZhMjQ3LWE0MzUtNDNiNC1iNjMxLWQ3MmY2NGZhYzZkYS5qb2I4LmpvYjEiLCJpc3MiOiJhcHAudnN0b2tlbi52aXN1YWxzdHVkaW8uY29tIiwiYXVkIjoiYXBwLnZzdG9rZW4udmlzdWFsc3R1ZGlvLmNvbXx2c286YjU1ZGU0ZWQtNGI1YS00MjE1LWE4ZTQtMGEwYTVmNzFlN2Q4IiwibmJmIjoxNjM1MjUxMTI4LCJleHAiOjE2MzUyNzE1Mjh9.VGcOaC-S5DcH7rp3K2R3Iw7DshavNL3-QlOvTo0krToboV2W57hG9xY_irMJ6676Z6QxmfVFp-lKXZBVeSdLN5nX1iXzKBub8EG0zHvOQC9W62zYD9KRPxMAqEIu5FjMs2rs7DRJdnur7EpshpAmysX4BBrhD7wKlJKjKUQ8TJJ8Tse1x3_Za74uyOaDExtIhL1U5fDvcsF0_dFqLM15iShf0vY6LAZ9sjc66MME0Ezz-gHfTuGdAAO4DX8H55jVVicEmrZtPsO2EIJwLxKA75G_ZfkcXCduz0K3zTorTVswv4i6lvDosz1DCtUO1pHlxFbDc_9kxUNFQf4ujkDLOg
2021-10-26 06:32:52,237: INFO: 139719151273792: run(48): main: Beginning reading of test results.
2021-10-26 06:32:52,237: INFO: 139719151273792: __init__(48): get_log_files: Searching '/home/helixbot/work/B77D09F3/w/ADD6094E/e/..' for log files
2021-10-26 06:32:52,237: INFO: 139719151273792: __init__(66): construct_log_list: Generated log list: console.fdc7508b.log:
  https://helixri8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-f2888cfa7c7947ceab/release.Partition6/console.fdc7508b.log?sv=2019-07-07&se=2022-01-24T12%3A41%3A38Z&sr=c&sp=rl&sig=eDgbkY3uPAnIgkAPt2cXDp0gl8u2WtGBhHL7Ku8qvFw%3D

2021-10-26 06:32:52,237: INFO: 139719151273792: __init__(90): read_results: Searching '/home/helixbot/work/B77D09F3/w/ADD6094E/e' for test results files
2021-10-26 06:32:52,238: INFO: 139719151273792: __init__(90): read_results: Searching '/home/helixbot/work/B77D09F3/w/ADD6094E/uploads' for test results files
2021-10-26 06:32:52,238: WARNING: 139719151273792: __init__(103): read_results: No results file found in any of the following formats: xunit, junit, trx
2021-10-26 06:32:52,238: INFO: 139719151273792: packing_test_reporter(30): report_results: Packing 0 test reports to '/home/helixbot/work/B77D09F3/w/ADD6094E/e/__test_report.json'
2021-10-26 06:32:52,238: INFO: 139719151273792: packing_test_reporter(33): report_results: Packed 1372 bytes
Found log '/home/helixbot/work/B77D09F3/w/ADD6094E/e/../console.fdc7508b.log'
Uri 'https://helixri8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-f2888cfa7c7947ceab/release.Partition6/console.fdc7508b.log?sv=2019-07-07&se=2022-01-24T12%3A41%3A38Z&sr=c&sp=rl&sig=eDgbkY3uPAnIgkAPt2cXDp0gl8u2WtGBhHL7Ku8qvFw%3D'
+ exit 1
SamMonoRT commented 3 years ago

@naricc @LoopedBard3 - scanning through above error - see this : You need to add 'net6.0' to in your project file ('MicroBenchmarks'). [2021/10/26 06:32:51][INFO] Example: net7.0;net6.0"

Also way above in the log it says Targetting net7.0 -- so possibly a version mismatch somewhere ?

LoopedBard3 commented 3 years ago

@SamMonoRT - I agree that there seems to be some versioning issue somewhere. I would checkout the attached issue dotnet/BenchmarkDotNet#1827 as it provides a little bit more context.

naricc commented 3 years ago

I had trouble reproducing this locally, but I think I figured this out. When we are running locally, this condition is true: https://github.com/dotnet/performance/blob/e7e94530cdce988f381ce053d28b1ee6f9d45c95/src/benchmarks/micro/MicroBenchmarks.csproj#L7, so we get all the frameworks.

In CI, however, the condition is false and TargetFrameworks is already set by the scripts. So it ends up with only .net7 in the script, which does not agree with the TargetFramework in the project generated by BDN. I will need to fix this in a way similar to what was done for wasm here: https://github.com/dotnet/BenchmarkDotNet/pull/1592

naricc commented 3 years ago

I have put up the PR that should fix this when merged. But it will require pulling a new version of Benchmark Dotnet. Who is making that change? @LoopedBard3 @DrewScoggins ?

naricc commented 3 years ago

@LoopedBard3 @DrewScoggins I don't think this is actually closed yet, right? The change on the dotnet/performance side to pull the new BDN version isn't in? But I don't appear to have the permissions to reopen the issue.

LoopedBard3 commented 3 years ago

@naricc I agree that it is not ready to be closed yet. Without changing anything in the perf repo, the MonoAOTLLVM runs are still failing in the main pipeline.

LoopedBard3 commented 3 years ago

2122 mitigated the issue by disabling the tests, #2123 is now tracking the reenabling of the tests. Closing this issue to consolidate the issues.

SamMonoRT commented 3 years ago

@LoopedBard3 - not sure how these two issues are related. What we need for this is a change very similar to this old one where we update the right BDN version with the fix : https://github.com/dotnet/performance/pull/2039/files

DrewScoggins commented 3 years ago

These two changes were not related, but since both were small changes I just stuck them in the same PR. Probably not the best way to do it.