search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.27k stars 4.73k forks source link

mono `minijit` runtime tests failing with `coreclr_initialize failed - Error: 0x80004005` #72114

Closed radical closed 2 years ago

radical commented 2 years ago

Failures 4/5-9/8 (incl. PRs and 7.0 branches):

Day Run OS
9/5 Rolling run 5191 (only in Runfo) mono Linux arm64 Release @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open
9/5 Rolling run 4698 (only in Runfo) mono Linux arm64 Release @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open
8/30 Rolling run Ubuntu.1804.Arm64.Open - Mono Interpreter
8/30 PR #74831 OSX.1200.Amd64.Open - Mono Interpreter
8/24 PR #74459 OSX 12 amd64 - Moni minijit
8/8 PR #73416 OSX 12 amd64 - Mono Interpreter
8/4 PR #73230 OSX 12 amd64 - Mono Interpreter
8/2 PR #73157 OSX 12 amd64 - Moni minijit
7/29 Rolling run Ubuntu.1804 arm64 - Mono llvm full aot
7/12-7/27 8x unique PRs OSX 12 amd64 and Ubuntu.1804 arm64
6/1-6/30 6x unique PRs OSX 12 amd64 and Ubuntu.1804 arm64
5/1-5/31 2x unique PRs OSX 12 amd64
4/1-4/30 6x unique PRs OSX 12 amd64 and Ubuntu.1804 arm64 
let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (Branch startswith 'refs/heads/release/7.0') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests('JIT/Math/Functions/Functions_ro', '', true, 'coreclr_initialize', true);

Hit on https://github.com/dotnet/runtime/pull/72094 . Build. From one of the logs: 

    JIT/Math/Functions/Functions_ro/Functions_ro.sh [FAIL]
      BEGIN: coreclr_initialize failed - Error: 0x80004005
      END: coreclr_initialize failed - Error: 0x80004005

      Return code:      1
      Raw output file:      /tmp/helix/working/B5FB0986/w/AC37094B/uploads/Reports/JIT.Math/Functions/Functions_ro/Functions_ro.output.txt
      Raw output:
      BEGIN EXECUTION
      /tmp/helix/working/B5FB0986/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Functions_ro.dll ''
      Exe path: /private/tmp/helix/working/B5FB0986/p/corerun
      Properties:
          TRUSTED_PLATFORM_ASSEMBLIES = <see console log for the full list>
          APP_PATHS = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/
          NATIVE_DLL_SEARCH_DIRECTORIES = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/:/tmp/helix/working/B5FB0986/p/:
          System.Reflection.Metadata.MetadataUpdater.IsSupported = false
      Managed assembly: /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.dll
      Arguments (0): 
      Expected: 100
      Actual: 255
      END EXECUTION - FAILED
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/tmp/helix/working/B5FB0986/p
      > /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_Math._Functions_Functions_ro_Functions_ro_._Functions_Functions_ro_Functions_ro_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        BEGIN: coreclr_initialize failed - Error: 0x80004005
        END: coreclr_initialize failed - Error: 0x80004005

        Return code:      1
        Raw output file:      /tmp/helix/working/B5FB0986/w/AC37094B/uploads/Reports/JIT.Math/Functions/Functions_ro/Functions_ro.output.txt
        Raw output:
        BEGIN EXECUTION
        /tmp/helix/working/B5FB0986/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Functions_ro.dll ''
        Exe path: /private/tmp/helix/working/B5FB0986/p/corerun
        Properties:
            TRUSTED_PLATFORM_ASSEMBLIES = <see console log for the full list>
            APP_PATHS = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/
            NATIVE_DLL_SEARCH_DIRECTORIES = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/:/tmp/helix/working/B5FB0986/p/:
            System.Reflection.Metadata.MetadataUpdater.IsSupported = false
        Managed assembly: /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.dll
        Arguments (0): 
        Expected: 100
        Actual: 255
        END EXECUTION - FAILED
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/tmp/helix/working/B5FB0986/p
        > /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.sh

cc @lewing

dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

lewing commented 2 years ago

marking future until we see it again

karelz commented 2 years ago

@lewing it happens more often - see updated top post with Kusto query. Overall about 2x per week - marking it as 'blocking-clean-ci'. Marking as untriaged again.

Also, it is causing massive failures on the runs, which makes it hard to analyze results -- see for example Rolling build from 7/29 - 1912276 has 200 failures.

fanyang-mono commented 2 years ago

The complete list of test failures with this error: https://runfo.azurewebsites.net/search/tests/?q=started%3A%7E14+definition%3Aruntime+message%3Acoreclr_initialize+jobName%3AMono

Build from 8/8 https://dev.azure.com/dnceng/public/_build/results?buildId=1929242&view=logs&j=76620f41-b685-5972-e5ab-3722b69632c4&t=e5054d15-9cd7-5687-4154-57982228d697 The failed lane was mono monointerpreter Pri0 Runtime Tests Run OSX x64 release

Build from 8/4 RP #73230 https://dev.azure.com/dnceng/public/_build/results?buildId=1922628&view=logs&j=76620f41-b685-5972-e5ab-3722b69632c4&t=e5054d15-9cd7-5687-4154-57982228d697 The failed lane was mono monointerpreter Pri0 Runtime Tests Run OSX x64 release

Build from 7/29 rolling build https://dev.azure.com/dnceng/public/_build/results?buildId=1912276&view=logs&j=d631b51d-8d68-5da3-5a13-95f6a763c7bb&t=e848a3c9-1589-513e-d5da-7aef4a24d916 The failed lane was mono llvmfullaot Pri0 Runtime Tests Run Linux arm64 release

@karelz I couldn't find the build containing this error from 8/2 PR #73157. Could you please comment it here?

fanyang-mono commented 2 years ago

I tried to run Interop/PInvoke/SetLastError/SetLastErrorTest/SetLastErrorTest.sh with interpreter locally multiple times, but wasn't able to reproduce this failure.

fanyang-mono commented 2 years ago

According to https://support.microsoft.com/en-us/topic/you-may-receive-error-code-0x80004005-or-other-error-codes-when-you-try-to-start-a-windows-xp-based-computer-a15f5b2f-642d-24ac-4912-1570a6bcedec. Error: 0x80004005 seems to be a windows related issue. However, the failures are happening on non-win platforms.

@lambdageek Do you have any thoughts on this?

lambdageek commented 2 years ago

monovm_initialize only returns 0x80004005 when parse_properties returns FALSE. Except that parse_properties always returns TRUE. This is some issue with the corerun host, not the runtime.

https://github.com/dotnet/runtime/blob/5eb776dc21d89063d66287461313c1e79073b57b/src/mono/mono/mini/monovm.c#L207-L208

lewing commented 2 years ago

cc @agocke

karelz commented 2 years ago

@fanyang-mono

I couldn't find the build containing this error from 8/2 PR #73157. Could you please comment it here?

I don't have more info than above + details from Kusto. You might be able to get to the run from JobID and other records in Kusto. Unfortunately, I don't know how to do it.

fanyang-mono commented 2 years ago

@karelz thanks for checking! We have enough information to move forward now.

ghost commented 2 years ago

Tagging subscribers to this area: @directhex See info in area-owners.md if you want to be subscribed.

Issue Details
Failures 4/5-8/15 (incl. PRs): Day | Run | OS -- | -- | -- 8/8 | PR #73416 | OSX 12 amd64 8/4 | PR #73230 | OSX 12 amd64 8/2 | PR #73157 | OSX 12 amd64 7/29 | **Rolling run** | Ubuntu.1804 arm64 7/12-7/27 | 8x unique PRs | OSX 12 amd64 and Ubuntu.1804 arm64 6/1-6/30 | 6x unique PRs | OSX 12 amd64 and Ubuntu.1804 arm64 5/1-5/31 | 2x unique PRs | OSX 12 amd64 4/1-4/30 | 6x unique PRs | OSX 12 amd64 and Ubuntu.1804 arm64 ```kusto let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) { cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests | where TestName contains testNameSubstring | where includePassedOnRerun or (Outcome == 'Failed') | extend startOfTestName = indexof_regex(TestName, @"[^.]+$") | extend Method = substring(TestName, startOfTestName) | extend Type = substring(TestName, 0, startOfTestName - 1) | project-away startOfTestName | where (methodName == '') or (Method == methodName) | where Message contains messageSubstr | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/"))) | where Type startswith "test/functional/cli/" and not(Properties contains "runtime-staging") | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId | project-rename JobType = Type) on JobId | extend PropertiesJson = parse_json(Properties) | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2') | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", "")) | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties) | extend Architecture = PropertiesJson.architecture | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario) //| extend DefinitionName = PropertiesJson.DefinitionName | project-away PropertiesJson }; failedTests('JIT/Math/Functions/Functions_ro', '', true, 'coreclr_initialize', true); ``` Hit on https://github.com/dotnet/runtime/pull/72094 . [Build](https://dev.azure.com/dnceng/public/_build/results?buildId=1877802&view=logs&j=a3584aa7-f6a0-5e90-05ca-4edce1db7c76&t=40ee3ac5-9c48-5be3-ec9b-ef68d55d9bed&s=96ac2280-8cb4-5df5-99de-dd2da759617d). From [one](https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-72094-merge-95ddd11cb5f746fe92/JIT.Math/1/console.4811214d.log?helixlogtype=result) of the logs: ``` JIT/Math/Functions/Functions_ro/Functions_ro.sh [FAIL] BEGIN: coreclr_initialize failed - Error: 0x80004005 END: coreclr_initialize failed - Error: 0x80004005 Return code: 1 Raw output file: /tmp/helix/working/B5FB0986/w/AC37094B/uploads/Reports/JIT.Math/Functions/Functions_ro/Functions_ro.output.txt Raw output: BEGIN EXECUTION /tmp/helix/working/B5FB0986/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Functions_ro.dll '' Exe path: /private/tmp/helix/working/B5FB0986/p/corerun Properties: TRUSTED_PLATFORM_ASSEMBLIES = APP_PATHS = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/ NATIVE_DLL_SEARCH_DIRECTORIES = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/:/tmp/helix/working/B5FB0986/p/: System.Reflection.Metadata.MetadataUpdater.IsSupported = false Managed assembly: /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.dll Arguments (0): Expected: 100 Actual: 255 END EXECUTION - FAILED Test Harness Exitcode is : 1 To run the test: > set CORE_ROOT=/tmp/helix/working/B5FB0986/p > /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.sh Expected: True Actual: False Stack Trace: at JIT_Math._Functions_Functions_ro_Functions_ro_._Functions_Functions_ro_Functions_ro_sh() at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor) at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) Output: BEGIN: coreclr_initialize failed - Error: 0x80004005 END: coreclr_initialize failed - Error: 0x80004005 Return code: 1 Raw output file: /tmp/helix/working/B5FB0986/w/AC37094B/uploads/Reports/JIT.Math/Functions/Functions_ro/Functions_ro.output.txt Raw output: BEGIN EXECUTION /tmp/helix/working/B5FB0986/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Functions_ro.dll '' Exe path: /private/tmp/helix/working/B5FB0986/p/corerun Properties: TRUSTED_PLATFORM_ASSEMBLIES = APP_PATHS = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/ NATIVE_DLL_SEARCH_DIRECTORIES = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/:/tmp/helix/working/B5FB0986/p/: System.Reflection.Metadata.MetadataUpdater.IsSupported = false Managed assembly: /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.dll Arguments (0): Expected: 100 Actual: 255 END EXECUTION - FAILED Test Harness Exitcode is : 1 To run the test: > set CORE_ROOT=/tmp/helix/working/B5FB0986/p > /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.sh ``` cc @lewing
Author: radical
Assignees: -
Labels: `blocking-clean-ci`, `untriaged`, `area-Infrastructure-mono`, `test-failure`
Milestone: -
fanyang-mono commented 2 years ago

According to @lambdageek

the following things happen at execution time (simplifying a bit):

  1. the CI script runs corerun
  2. corerun dynamically loads libcoreclr.so (really mono) from the Core_Root directory
  3. corerun finds the coreclr_initialize symbol and calls it
  4. the coreclr_initialize function should return S_OK or an error code.

what i (@lambdageek ) was saying in the issue is that (4) is definitely not happening - because mono cannot possibly return 0x80004005 so (1) is happening; (4) is NOT happening with mono; so the problem is in either step (2) (can't find the shared library) > or in step (3) (found the shared library, but it has no symbol) or step (4) (corerun calls coreclr_initialize but it's the wrong > runtime and it doesn't like Core_Root or some properties or something)

@LakshanF Are you the best person to take a look at this issue?

lambdageek commented 2 years ago
  1. corerun dynamically loads libcoreclr.so (really mono) from the Core_Root directory
  2. corerun finds the coreclr_initialize symbol and calls it
  3. the coreclr_initialize function should return S_OK or an error code.

So... looking at corerun.cpp, the only place where it prints out "BEGIN: coreclr_initialize failed - Error: 0x%08x\n" is when it successfully loads a libcoreclr.so and successfully binds the coreclr_initialize symbol, and manages to call it. Other cases (like not being able to load the library, or not being able to find the symbol) just return -1 from the run method which just exits MAIN.

https://github.com/dotnet/runtime/blob/a389c12782719a2d9deb076fded289f363ccb105/src/coreclr/hosts/corerun/corerun.cpp#L358

So... it loaded something. and that something had coreclr_initialize and that coreclr_initialize returned 0x80004005

Maybe this load is hitting a problem:

https://github.com/dotnet/runtime/blob/a389c12782719a2d9deb076fded289f363ccb105/src/coreclr/hosts/corerun/corerun.cpp#L277

I wonder if it's possible that the mono libcoreclr.so is not in the expected place and the dynamic linker is loading some fallback from some system folder?

SamMonoRT commented 2 years ago

Seems outside Mono scope at this time. Not certain what the best area label is though. Also moving to 8.0.0 given the failure is not deterministic and seems like a test/infrastructure issue, not a real customer issue.

cc @LakshanF - is this something you can help investigate? Please update area label appropriately.

karelz commented 2 years ago

Another massive hit on:

@SamMonoRT who is taking action on this? Can you please find someone on your team or another team? This is blocking CI ...

elinor-fung commented 2 years ago

Hit in https://github.com/dotnet/runtime/pull/75858 Example of one of the failures:

  Mono AOT options: full,nimt-trampolines=2000,ntrampolines=10000,nrgctx-fetch-trampolines=256,ngsharedvt-trampolines=4400,nftnptr-arg-trampolines=4000,nrgctx-trampolines=21000,llvm,llvm-path=/__w/1/s/artifacts/bin/mono/Linux.x64.Release,mattr=sse4.2,mattr=popcnt,mattr=lzcnt,mattr=bmi,mattr=bmi2,mattr=pclmul,mattr=aes
  Mono AOT MONO_PATH: /__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root
  aot-compile: compiling /__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/baseservices/compilerservices/RuntimeFeature/DefaultImplementationsOfInterfaces/DefaultImplementationsOfInterfaces.dll; MONO_PATH: /__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/baseservices/compilerservices/RuntimeFeature/DefaultImplementationsOfInterfaces:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root
BEGIN : coreclr_initialize failed - error : 0x80004005 [/__w/1/s/src/mono/msbuild/aot-compile.proj]
END : coreclr_initialize failed - error : 0x80004005 [/__w/1/s/src/mono/msbuild/aot-compile.proj]
  Exe path: /__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root/corerun
  Properties:
      TRUSTED_PLATFORM_ASSEMBLIES = /__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Reflection.Extensions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.Serialization.Schema.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.CSharp.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Collections.Concurrent.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Reflection.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.FileProviders.Physical.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Memory.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.Serialization.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Memory.Data.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.DirectoryServices.Protocols.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Private.Uri.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Options.ConfigurationExtensions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//WindowsBase.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Xml.XPath.XDocument.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.ServiceProcess.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Newtonsoft.Json.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Configuration.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.Quic.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Cryptography.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Hosting.WindowsServices.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Configuration.Xml.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Cryptography.Cng.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.DirectoryServices.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.NetworkInformation.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Cryptography.OpenSsl.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.DirectoryServices.AccountManagement.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Logging.Console.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Linq.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.CompilerServices.VisualC.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Text.RegularExpressions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.Packaging.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Data.OleDb.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.Compression.ZipFile.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Logging.Debug.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Cryptography.Xml.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.ComponentModel.EventBasedAsync.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Dia2Lib.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Diagnostics.TraceSource.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Numerics.Vectors.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Private.DataContractSerialization.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.CodeAnalysis.VisualBasic.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Threading.Tasks.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.Pipes.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Diagnostics.Tracing.TraceEvent.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Caching.Memory.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Drawing.Primitives.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.Http.Json.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.InteropServices.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.Http.WinHttpHandler.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Formats.Tar.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.ServiceProcess.ServiceController.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Hosting.Abstractions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Windows.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.MemoryMappedFiles.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.Serialization.Formatters.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Cryptography.Csp.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.IsolatedStorage.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Threading.Thread.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Data.DataSetExtensions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Xml.ReaderWriter.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Threading.ThreadPool.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.Http.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Composition.Runtime.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.Caching.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.Hashing.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Logging.Configuration.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Primitives.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.AppContext.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Permissions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Reflection.TypeExtensions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Xml.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.DotNet.XUnitExtensions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Reflection.Emit.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Cryptography.Pkcs.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Reflection.Metadata.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Dynamic.Runtime.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Resources.Reader.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Cryptography.Encoding.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Logging.EventLog.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.ComponentModel.DataAnnotations.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Formats.Asn1.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//xunit.assert.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//mscorlib.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Threading.Tasks.Dataflow.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Web.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Composition.TypedParts.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.InteropServices.JavaScript.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Bcl.AsyncInterfaces.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.Primitives.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Transactions.Local.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Resources.ResourceManager.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.Pipes.AccessControl.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//FSharp.Core.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Diagnostics.TextWriterTraceListener.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Http.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//tieringtest.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Linq.Expressions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Logging.EventSource.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.Ports.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Resources.Extensions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.Serialization.Json.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.FileSystem.AccessControl.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.DependencyModel.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Win32.SystemEvents.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.SecureString.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Xml.XPath.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Private.CoreLib.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Xml.XDocument.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Xml.Linq.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Private.Xml.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.Sockets.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Configuration.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Options.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Data.Common.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Reflection.DispatchProxy.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Private.Xml.Linq.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Drawing.Common.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Collections.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.Handles.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.FileProviders.Composite.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//OSExtensions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.WebHeaderCollection.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//CommandLine.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Text.Encodings.Web.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.VisualBasic.Core.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Logging.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Configuration.UserSecrets.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.WebProxy.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Reflection.MetadataLoadContext.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Hosting.Systemd.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//TraceReloggerLib.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Collections.NonGeneric.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.ComponentModel.Composition.Registration.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.VisualBasic.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Core.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Win32.Registry.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Diagnostics.Process.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Composition.Convention.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Threading.Tasks.Parallel.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.ComponentModel.Primitives.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//ILCompiler.Reflection.ReadyToRun.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Globalization.Calendars.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Reflection.Emit.ILGeneration.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Collections.Specialized.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Claims.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Text.Encoding.Extensions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.ObjectModel.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Reflection.Primitives.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Newtonsoft.Json.Bson.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Globalization.Extensions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//xunit.execution.dotnet.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Threading.Timer.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Numerics.Tensors.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Configuration.Binder.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Configuration.Json.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Options.DataAnnotations.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Diagnostics.Contracts.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Reflection.Emit.Lightweight.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.InteropServices.RuntimeInformation.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Cryptography.X509Certificates.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Text.Encoding.CodePages.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.FileSystemGlobbing.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.Mail.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Diagnostics.FileVersionInfo.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Threading.Channels.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.ServiceModel.Web.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Linq.Queryable.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.ValueTuple.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.ComponentModel.Annotations.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//xunit.runner.utility.netcoreapp10.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//netstandard.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Configuration.ConfigurationManager.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.ComponentModel.Composition.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.HttpListener.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.CodeDom.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.FileSystem.DriveInfo.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Cryptography.Algorithms.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Diagnostics.EventLog.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Buffers.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.Serialization.Xml.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Transactions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Composition.Hosting.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Hosting.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.FileSystem.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.Intrinsics.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.DependencyInjection.Abstractions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.UnmanagedMemoryStream.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Threading.RateLimiting.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Windows.Extensions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Configuration.FileExtensions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Text.Encoding.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Principal.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Cryptography.Cose.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//xunit.abstractions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.CodeAnalysis.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Speech.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.WebSockets.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Formats.Cbor.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.WebSockets.Client.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.CompilerServices.Unsafe.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Cryptography.ProtectedData.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.Ping.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Diagnostics.Tools.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Diagnostics.Tracing.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Linq.Parallel.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Diagnostics.Debug.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.ComponentModel.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Diagnostics.FastSerialization.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//xunit.runner.reporters.netcoreapp10.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Xml.XmlDocument.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.Pipelines.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Principal.Windows.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Web.HttpUtility.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.FileProviders.Abstractions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Configuration.Ini.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.FileSystem.Primitives.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.Loader.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Data.Odbc.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.FileSystem.Watcher.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Logging.TraceSource.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Diagnostics.Tools.RuntimeClient.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//xunit.console.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Threading.AccessControl.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Configuration.Abstractions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.Numerics.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Diagnostics.StackTrace.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Data.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Text.Json.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Xml.XmlSerializer.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Drawing.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Xml.Serialization.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Logging.Abstractions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Collections.Immutable.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Configuration.CommandLine.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Diagnostics.DiagnosticSource.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.Compression.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Threading.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.Security.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//xunit.core.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.ServicePoint.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.Compression.Brotli.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Management.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Win32.Registry.AccessControl.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Configuration.EnvironmentVariables.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Composition.AttributedModel.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.WebClient.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.Caching.Abstractions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Reflection.Context.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Numerics.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Resources.Writer.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.ServiceModel.Syndication.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Win32.Primitives.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.Extensions.DependencyInjection.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Console.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//runincontext.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Threading.Overlapped.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.Serialization.Primitives.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.Requests.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.ComponentModel.TypeConverter.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Net.NameResolution.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Globalization.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Threading.Tasks.Extensions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Diagnostics.PerformanceCounter.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.AccessControl.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.IO.Compression.FileSystem.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//Microsoft.CodeAnalysis.CSharp.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Runtime.Extensions.dll:/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root//System.Security.Cryptography.Primitives.dll:
      APP_PATHS = /__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/baseservices/compilerservices/RuntimeFeature/DefaultImplementationsOfInterfaces/
      NATIVE_DLL_SEARCH_DIRECTORIES = /__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/baseservices/compilerservices/RuntimeFeature/DefaultImplementationsOfInterfaces/:
  Managed assembly: /__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/baseservices/compilerservices/RuntimeFeature/DefaultImplementationsOfInterfaces/DefaultImplementationsOfInterfaces.dll
  Arguments (0): 
/__w/1/s/src/mono/msbuild/aot-compile.proj(19,9): error MSB3073: The command "/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root/corerun /__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/baseservices/compilerservices/RuntimeFeature/DefaultImplementationsOfInterfaces/DefaultImplementationsOfInterfaces.dll" exited with code 255.

Test run logs

elinor-fung commented 2 years ago

I believe it is as @lambdageek said:

I wonder if it's possible that the mono libcoreclr.so is not in the expected place

When generating CORE_ROOT, both the coreclr and mono versions of libcoreclr.so are being copied over, so it is a race of which is last. In the case of the runtime pipeline build from my previous comment, the coreclr version is last.

test_dependencies.fsproj net7.0  → CopyDependencyToCoreRoot
    Target CopyDependencyToCoreRoot
        Task Copy
            Message  Copying file from "/__w/1/s/artifacts/bin/coreclr/Linux.x64.Release/libcoreclr.so" to "/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root/libcoreclr.so".
            Message  Copying file from "/__w/1/s/artifacts/bin/mono/Linux.x64.Release/libcoreclr.so" to "/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root/libcoreclr.so".
test_dependencies.csproj net7.0  → CopyDependencyToCoreRoot
    Target CopyDependencyToCoreRoot
        Task Copy
            Message  Did not copy from file "/__w/1/s/artifacts/bin/mono/Linux.x64.Release//libcoreclr.so" to file "/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root/libcoreclr.so" because the "SkipUnchangedFiles" parameter was set to "true" in the project and the...
            Message  Copying file from "/__w/1/s/artifacts/bin/coreclr/Linux.x64.Release/libcoreclr.so" to "/__w/1/s/artifacts/tests/coreclr/Linux.x64.Release/Tests/Core_Root/libcoreclr.so".

Relevant logs from the runtime pipeline build: Layout.Linux.x64.Release.binlog.txt (actually a binlog, just renamed) Layout.Linux.x64.Release.log

elinor-fung commented 2 years ago

The coreclr version is being included RunTimeDependencyCopyLocal from: https://github.com/dotnet/runtime/blob/3435b39194137676d9b2da12c8ebd21d69cd06b9/src/tests/Common/Directory.Build.targets#L101-L111

Perhaps we just need to condition those under '$(RuntimeFlavor)' == 'coreclr'.

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/runtime-infrastructure See info in area-owners.md if you want to be subscribed.

Issue Details
Failures 4/5-9/8 (incl. PRs and 7.0 branches): Day | Run | OS -- | -- | -- 9/5 | **Rolling run** [5191](https://runfo.azurewebsites.net/view/build/?number=5191) (only in Runfo) | mono Linux arm64 Release @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open 9/5 | **Rolling run** [4698](https://runfo.azurewebsites.net/view/build/?number=4698) (only in Runfo) | mono Linux arm64 Release @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open 8/30 | **Rolling run** | Ubuntu.1804.Arm64.Open - Mono Interpreter 8/30 | PR #74831 | OSX.1200.Amd64.Open - Mono Interpreter 8/24 | PR #74459 | OSX 12 amd64 - Moni minijit 8/8 | PR #73416 | OSX 12 amd64 - Mono Interpreter 8/4 | PR #73230 | OSX 12 amd64 - Mono Interpreter 8/2 | PR #73157 | OSX 12 amd64 - Moni minijit 7/29 | **Rolling run** | Ubuntu.1804 arm64 - Mono llvm full aot 7/12-7/27 | 8x unique PRs | OSX 12 amd64 and Ubuntu.1804 arm64 6/1-6/30 | 6x unique PRs | OSX 12 amd64 and Ubuntu.1804 arm64 5/1-5/31 | 2x unique PRs | OSX 12 amd64 4/1-4/30 | 6x unique PRs | OSX 12 amd64 and Ubuntu.1804 arm64 ```kusto let failedTests = (testNameSubstring : string, methodName : string, includePR : bool, messageSubstr: string, includePassedOnRerun : bool) { cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests | where TestName contains testNameSubstring | where includePassedOnRerun or (Outcome == 'Failed') | extend startOfTestName = indexof_regex(TestName, @"[^.]+$") | extend Method = substring(TestName, startOfTestName) | extend Type = substring(TestName, 0, startOfTestName - 1) | project-away startOfTestName | where (methodName == '') or (Method == methodName) | where Message contains messageSubstr | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (Branch startswith 'refs/heads/release/7.0') or (includePR and (Source startswith "pr/"))) | where Type startswith "test/functional/cli/" and not(Properties contains "runtime-staging") | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId | project-rename JobType = Type) on JobId | extend PropertiesJson = parse_json(Properties) | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2') | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", "")) | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties) | extend Architecture = PropertiesJson.architecture | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario) //| extend DefinitionName = PropertiesJson.DefinitionName | project-away PropertiesJson }; failedTests('JIT/Math/Functions/Functions_ro', '', true, 'coreclr_initialize', true); ``` Hit on https://github.com/dotnet/runtime/pull/72094 . [Build](https://dev.azure.com/dnceng/public/_build/results?buildId=1877802&view=logs&j=a3584aa7-f6a0-5e90-05ca-4edce1db7c76&t=40ee3ac5-9c48-5be3-ec9b-ef68d55d9bed&s=96ac2280-8cb4-5df5-99de-dd2da759617d). From [one](https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-72094-merge-95ddd11cb5f746fe92/JIT.Math/1/console.4811214d.log?helixlogtype=result) of the logs: ``` JIT/Math/Functions/Functions_ro/Functions_ro.sh [FAIL] BEGIN: coreclr_initialize failed - Error: 0x80004005 END: coreclr_initialize failed - Error: 0x80004005 Return code: 1 Raw output file: /tmp/helix/working/B5FB0986/w/AC37094B/uploads/Reports/JIT.Math/Functions/Functions_ro/Functions_ro.output.txt Raw output: BEGIN EXECUTION /tmp/helix/working/B5FB0986/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Functions_ro.dll '' Exe path: /private/tmp/helix/working/B5FB0986/p/corerun Properties: TRUSTED_PLATFORM_ASSEMBLIES = APP_PATHS = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/ NATIVE_DLL_SEARCH_DIRECTORIES = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/:/tmp/helix/working/B5FB0986/p/: System.Reflection.Metadata.MetadataUpdater.IsSupported = false Managed assembly: /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.dll Arguments (0): Expected: 100 Actual: 255 END EXECUTION - FAILED Test Harness Exitcode is : 1 To run the test: > set CORE_ROOT=/tmp/helix/working/B5FB0986/p > /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.sh Expected: True Actual: False Stack Trace: at JIT_Math._Functions_Functions_ro_Functions_ro_._Functions_Functions_ro_Functions_ro_sh() at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor) at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) Output: BEGIN: coreclr_initialize failed - Error: 0x80004005 END: coreclr_initialize failed - Error: 0x80004005 Return code: 1 Raw output file: /tmp/helix/working/B5FB0986/w/AC37094B/uploads/Reports/JIT.Math/Functions/Functions_ro/Functions_ro.output.txt Raw output: BEGIN EXECUTION /tmp/helix/working/B5FB0986/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Functions_ro.dll '' Exe path: /private/tmp/helix/working/B5FB0986/p/corerun Properties: TRUSTED_PLATFORM_ASSEMBLIES = APP_PATHS = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/ NATIVE_DLL_SEARCH_DIRECTORIES = /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/:/tmp/helix/working/B5FB0986/p/: System.Reflection.Metadata.MetadataUpdater.IsSupported = false Managed assembly: /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.dll Arguments (0): Expected: 100 Actual: 255 END EXECUTION - FAILED Test Harness Exitcode is : 1 To run the test: > set CORE_ROOT=/tmp/helix/working/B5FB0986/p > /private/tmp/helix/working/B5FB0986/w/AC37094B/e/JIT/Math/Functions/Functions_ro/Functions_ro.sh ``` cc @lewing
Author: radical
Assignees: -
Labels: `blocking-clean-ci`, `area-Infrastructure`, `test-failure`
Milestone: 8.0.0