Work item tests crash on OSX x64 Debug

eerhardt commented 1 year ago

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158 Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution Pull request: https://github.com/dotnet/runtime/pull/86974

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "END OF WORK ITEM LOG: Command exited with 127",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Build	Definition	Test	Pull Request
475355	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
474849	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
474532	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
473727	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
473597	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
473431	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
473397	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
472898	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
472852	dotnet/runtime	System.IO.Compression.Tests.WorkItemExecution
472282	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
471895	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
471740	dotnet/runtime	System.IO.Compression.Tests.WorkItemExecution
471580	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
471389	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
470637	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
470321	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
469012	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
468964	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
468841	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
468587	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
468089	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
467619	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
467146	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
466691	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
465768	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
465695	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
465642	dotnet/runtime	XUnitLogChecker.WorkItemExecution	dotnet/runtime#93906
465587	dotnet/runtime	XUnitLogChecker.WorkItemExecution	dotnet/runtime#93906
465447	dotnet/runtime	XUnitLogChecker.WorkItemExecution	dotnet/runtime#93906
465319	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
465327	dotnet/runtime	System.IO.FileSystem.Primitives.Tests.WorkItemExecution
464592	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
463142	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
462608	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
461771	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
460596	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
460238	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
459941	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
459223	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
458289	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
457386	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
456786	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
456243	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
456245	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
455009	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
454543	dotnet/runtime	System.IO.FileSystem.Manual.Tests.WorkItemExecution
454533	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
454134	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
452981	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
451747	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
450630	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
449186	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
447511	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution
446965	dotnet/runtime	System.IO.FileSystem.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	21	54

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=323604 Error message validated: Work item System.IO.FileSystem.Watcher.Tests in job Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 6/28/2023 9:26:54 PM UTC

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/area-infrastructure-libraries See info in area-owners.md if you want to be subscribed.

Issue Details

## Build Information Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158 Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution Pull request: https://github.com/dotnet/runtime/pull/86974 ## Error Message Fill the error message using [step by step known issues guidance](https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssueJsonStepByStep.md#how-to-create-a-known-issue-step-by-step). ```json { "ErrorMessage": "Work item System.IO.FileSystem.Watcher.Tests in job", "ErrorPattern": "", "BuildRetry": false, "ExcludeConsoleLog": false } ```

Author:	eerhardt
Assignees:	-
Labels:	`area-Infrastructure-libraries`, `blocking-clean-ci`, `untriaged`, `Known Build Error`
Milestone:	-

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/area-system-io See info in area-owners.md if you want to be subscribed.

Issue Details

## Build Information Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158 Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution Pull request: https://github.com/dotnet/runtime/pull/86974 ## Error Message Fill the error message using [step by step known issues guidance](https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssueJsonStepByStep.md#how-to-create-a-known-issue-step-by-step). ```json { "ErrorMessage": "Work item System.IO.FileSystem.Watcher.Tests in job", "ErrorPattern": "", "BuildRetry": false, "ExcludeConsoleLog": false } ```

Author:	eerhardt
Assignees:	-
Labels:	`area-System.IO`, `blocking-clean-ci`, `untriaged`, `Known Build Error`
Milestone:	-

jozkee commented 1 year ago

The logs not only show FileSystem.Watcher.Tests. I think this is a broader issue.

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/334444/logs/123

e.g:

2023-07-11T00:12:22.4993970Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Work item Common.Tests in job 9199670f-aa4c-4e8e-8150-b650ad0bb59f has failed. [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]
2023-07-11T00:12:22.4995970Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Failure log: https://helix.dot.net/api/2019-06-17/jobs/9199670f-aa4c-4e8e-8150-b650ad0bb59f/workitems/Common.Tests/console [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]
2023-07-11T00:12:22.5108980Z ##[error].packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item Common.Tests in job 9199670f-aa4c-4e8e-8150-b650ad0bb59f has failed.
Failure log: https://helix.dot.net/api/2019-06-17/jobs/9199670f-aa4c-4e8e-8150-b650ad0bb59f/workitems/Common.Tests/console
2023-07-11T00:12:22.5122420Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Work item IcuAppLocal.Tests in job 9199670f-aa4c-4e8e-8150-b650ad0bb59f has failed. [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]
2023-07-11T00:12:22.5126220Z /Users/runner/work/1/s/.packages/microsoft.dotnet.helix.sdk/7.0.0-beta.23313.4/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(90,5): error : Failure log: https://helix.dot.net/api/2019-06-17/jobs/9199670f-aa4c-4e8e-8150-b650ad0bb59f/workitems/IcuAppLocal.Tests/console [/Users/runner/work/1/s/src/libraries/sendtohelixhelp.proj]

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/runtime-infrastructure See info in area-owners.md if you want to be subscribed.

Issue Details

## Build Information Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=300158 Build error leg or test failing: System.IO.FileSystem.Watcher.Tests.WorkItemExecution Pull request: https://github.com/dotnet/runtime/pull/86974 ## Error Message Fill the error message using [step by step known issues guidance](https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssueJsonStepByStep.md#how-to-create-a-known-issue-step-by-step). ```json { "ErrorMessage": "Work item System.IO.FileSystem.Watcher.Tests in job", "ErrorPattern": "", "BuildRetry": false, "ExcludeConsoleLog": false } ``` ### Report |Build|Definition|Step Name|Console log|Pull Request| |---|---|---|---|---| |[347502](https://dev.azure.com/dnceng-public/public/_build/results?buildId=347502)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/347502/logs/319)|| |[347514](https://dev.azure.com/dnceng-public/public/_build/results?buildId=347514)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/347514/logs/67)|| |[347516](https://dev.azure.com/dnceng-public/public/_build/results?buildId=347516)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/347516/logs/77)|| |[347506](https://dev.azure.com/dnceng-public/public/_build/results?buildId=347506)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/347506/logs/164)|| |[347504](https://dev.azure.com/dnceng-public/public/_build/results?buildId=347504)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/347504/logs/156)|| |[347527](https://dev.azure.com/dnceng-public/public/_build/results?buildId=347527)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/347527/logs/301)|| |[346179](https://dev.azure.com/dnceng-public/public/_build/results?buildId=346179)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/346179/logs/67)|| |[346184](https://dev.azure.com/dnceng-public/public/_build/results?buildId=346184)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/346184/logs/297)|| |[346183](https://dev.azure.com/dnceng-public/public/_build/results?buildId=346183)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/346183/logs/77)|| |[346189](https://dev.azure.com/dnceng-public/public/_build/results?buildId=346189)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/346189/logs/170)|| |[346180](https://dev.azure.com/dnceng-public/public/_build/results?buildId=346180)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/346180/logs/162)|| |[346188](https://dev.azure.com/dnceng-public/public/_build/results?buildId=346188)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/346188/logs/303)|| |[345385](https://dev.azure.com/dnceng-public/public/_build/results?buildId=345385)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/345385/logs/2037)|dotnet/runtime#89145| |[344799](https://dev.azure.com/dnceng-public/public/_build/results?buildId=344799)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/344799/logs/67)|| |[344802](https://dev.azure.com/dnceng-public/public/_build/results?buildId=344802)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/344802/logs/297)|| |[344773](https://dev.azure.com/dnceng-public/public/_build/results?buildId=344773)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/344773/logs/77)|| |[344775](https://dev.azure.com/dnceng-public/public/_build/results?buildId=344775)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/344775/logs/164)|| |[344801](https://dev.azure.com/dnceng-public/public/_build/results?buildId=344801)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/344801/logs/303)|| |[344791](https://dev.azure.com/dnceng-public/public/_build/results?buildId=344791)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/344791/logs/156)|| |[344313](https://dev.azure.com/dnceng-public/public/_build/results?buildId=344313)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/344313/logs/1871)|dotnet/runtime#89145| |[344014](https://dev.azure.com/dnceng-public/public/_build/results?buildId=344014)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/344014/logs/1017)|dotnet/runtime#89061| |[343534](https://dev.azure.com/dnceng-public/public/_build/results?buildId=343534)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/343534/logs/2419)|| |[343456](https://dev.azure.com/dnceng-public/public/_build/results?buildId=343456)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/343456/logs/1223)|dotnet/runtime#89074| |[343307](https://dev.azure.com/dnceng-public/public/_build/results?buildId=343307)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/343307/logs/67)|| |[343300](https://dev.azure.com/dnceng-public/public/_build/results?buildId=343300)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/343300/logs/279)|| |[343299](https://dev.azure.com/dnceng-public/public/_build/results?buildId=343299)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/343299/logs/77)|| |[343301](https://dev.azure.com/dnceng-public/public/_build/results?buildId=343301)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/343301/logs/164)|| |[343298](https://dev.azure.com/dnceng-public/public/_build/results?buildId=343298)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/343298/logs/303)|| |[343296](https://dev.azure.com/dnceng-public/public/_build/results?buildId=343296)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/343296/logs/156)|| |[343319](https://dev.azure.com/dnceng-public/public/_build/results?buildId=343319)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/343319/logs/1306)|dotnet/runtime#89074| |[342971](https://dev.azure.com/dnceng-public/public/_build/results?buildId=342971)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/342971/logs/108)|| |[342902](https://dev.azure.com/dnceng-public/public/_build/results?buildId=342902)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/342902/logs/1504)|dotnet/runtime#87847| |[342704](https://dev.azure.com/dnceng-public/public/_build/results?buildId=342704)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/342704/logs/136)|dotnet/runtime#88974| |[342426](https://dev.azure.com/dnceng-public/public/_build/results?buildId=342426)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/342426/logs/2422)|| |[342101](https://dev.azure.com/dnceng-public/public/_build/results?buildId=342101)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/342101/logs/137)|| |[341764](https://dev.azure.com/dnceng-public/public/_build/results?buildId=341764)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/341764/logs/67)|| |[341747](https://dev.azure.com/dnceng-public/public/_build/results?buildId=341747)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/341747/logs/302)|| |[341763](https://dev.azure.com/dnceng-public/public/_build/results?buildId=341763)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/341763/logs/77)|| |[341766](https://dev.azure.com/dnceng-public/public/_build/results?buildId=341766)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/341766/logs/164)|| |[341752](https://dev.azure.com/dnceng-public/public/_build/results?buildId=341752)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/341752/logs/303)|| |[341745](https://dev.azure.com/dnceng-public/public/_build/results?buildId=341745)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/341745/logs/156)|| |[341422](https://dev.azure.com/dnceng-public/public/_build/results?buildId=341422)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/341422/logs/66)|dotnet/runtime#88502| |[341117](https://dev.azure.com/dnceng-public/public/_build/results?buildId=341117)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/341117/logs/286)|| |[341130](https://dev.azure.com/dnceng-public/public/_build/results?buildId=341130)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/341130/logs/62)|| |[341115](https://dev.azure.com/dnceng-public/public/_build/results?buildId=341115)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/341115/logs/77)|| |[341116](https://dev.azure.com/dnceng-public/public/_build/results?buildId=341116)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/341116/logs/164)|| |[341137](https://dev.azure.com/dnceng-public/public/_build/results?buildId=341137)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/341137/logs/309)|| |[341114](https://dev.azure.com/dnceng-public/public/_build/results?buildId=341114)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/341114/logs/156)|| |[340730](https://dev.azure.com/dnceng-public/public/_build/results?buildId=340730)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/340730/logs/62)|| |[340706](https://dev.azure.com/dnceng-public/public/_build/results?buildId=340706)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/340706/logs/297)|| |[340711](https://dev.azure.com/dnceng-public/public/_build/results?buildId=340711)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/340711/logs/77)|| |[340713](https://dev.azure.com/dnceng-public/public/_build/results?buildId=340713)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/340713/logs/164)|| |[340722](https://dev.azure.com/dnceng-public/public/_build/results?buildId=340722)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/340722/logs/156)|| |[340731](https://dev.azure.com/dnceng-public/public/_build/results?buildId=340731)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/340731/logs/303)|| |[340606](https://dev.azure.com/dnceng-public/public/_build/results?buildId=340606)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/340606/logs/69)|dotnet/runtime#74623| |[339887](https://dev.azure.com/dnceng-public/public/_build/results?buildId=339887)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/339887/logs/123)|| |[339546](https://dev.azure.com/dnceng-public/public/_build/results?buildId=339546)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/339546/logs/67)|| |[339567](https://dev.azure.com/dnceng-public/public/_build/results?buildId=339567)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/339567/logs/292)|| |[339550](https://dev.azure.com/dnceng-public/public/_build/results?buildId=339550)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/339550/logs/77)|| |[339557](https://dev.azure.com/dnceng-public/public/_build/results?buildId=339557)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/339557/logs/170)|| |[339565](https://dev.azure.com/dnceng-public/public/_build/results?buildId=339565)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/339565/logs/162)|| |[339554](https://dev.azure.com/dnceng-public/public/_build/results?buildId=339554)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/339554/logs/303)|| |[339453](https://dev.azure.com/dnceng-public/public/_build/results?buildId=339453)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/339453/logs/1197)|dotnet/runtime#88892| |[339194](https://dev.azure.com/dnceng-public/public/_build/results?buildId=339194)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/339194/logs/63)|dotnet/runtime#74623| |[339190](https://dev.azure.com/dnceng-public/public/_build/results?buildId=339190)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/339190/logs/2404)|| |[338975](https://dev.azure.com/dnceng-public/public/_build/results?buildId=338975)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/338975/logs/111)|| |[338958](https://dev.azure.com/dnceng-public/public/_build/results?buildId=338958)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/338958/logs/116)|| |[338949](https://dev.azure.com/dnceng-public/public/_build/results?buildId=338949)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/338949/logs/2400)|| |[338278](https://dev.azure.com/dnceng-public/public/_build/results?buildId=338278)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/338278/logs/297)|| |[338279](https://dev.azure.com/dnceng-public/public/_build/results?buildId=338279)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/338279/logs/67)|| |[338284](https://dev.azure.com/dnceng-public/public/_build/results?buildId=338284)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/338284/logs/77)|| |[338258](https://dev.azure.com/dnceng-public/public/_build/results?buildId=338258)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/338258/logs/15)|| |[338277](https://dev.azure.com/dnceng-public/public/_build/results?buildId=338277)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/338277/logs/164)|| |[338291](https://dev.azure.com/dnceng-public/public/_build/results?buildId=338291)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/338291/logs/303)|| |[338281](https://dev.azure.com/dnceng-public/public/_build/results?buildId=338281)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/338281/logs/156)|| |[337928](https://dev.azure.com/dnceng-public/public/_build/results?buildId=337928)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/337928/logs/139)|| |[337971](https://dev.azure.com/dnceng-public/public/_build/results?buildId=337971)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/337971/logs/2096)|| |[336905](https://dev.azure.com/dnceng-public/public/_build/results?buildId=336905)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/336905/logs/125)|| |[336900](https://dev.azure.com/dnceng-public/public/_build/results?buildId=336900)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/336900/logs/2393)|| |[336626](https://dev.azure.com/dnceng-public/public/_build/results?buildId=336626)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/336626/logs/297)|| |[336635](https://dev.azure.com/dnceng-public/public/_build/results?buildId=336635)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/336635/logs/67)|| |[336627](https://dev.azure.com/dnceng-public/public/_build/results?buildId=336627)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/336627/logs/77)|| |[336647](https://dev.azure.com/dnceng-public/public/_build/results?buildId=336647)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/336647/logs/164)|| |[336638](https://dev.azure.com/dnceng-public/public/_build/results?buildId=336638)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/336638/logs/303)|| |[336646](https://dev.azure.com/dnceng-public/public/_build/results?buildId=336646)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/336646/logs/156)|| |[336353](https://dev.azure.com/dnceng-public/public/_build/results?buildId=336353)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/336353/logs/117)|| |[336347](https://dev.azure.com/dnceng-public/public/_build/results?buildId=336347)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/336347/logs/2406)|| |[335895](https://dev.azure.com/dnceng-public/public/_build/results?buildId=335895)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/335895/logs/1945)|dotnet/runtime#88626| |[335021](https://dev.azure.com/dnceng-public/public/_build/results?buildId=335021)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/335021/logs/67)|| |[335024](https://dev.azure.com/dnceng-public/public/_build/results?buildId=335024)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/335024/logs/285)|| |[335003](https://dev.azure.com/dnceng-public/public/_build/results?buildId=335003)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/335003/logs/72)|| |[335004](https://dev.azure.com/dnceng-public/public/_build/results?buildId=335004)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/335004/logs/164)|| |[334953](https://dev.azure.com/dnceng-public/public/_build/results?buildId=334953)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/334953/logs/116)|| |[335012](https://dev.azure.com/dnceng-public/public/_build/results?buildId=335012)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/335012/logs/162)|| |[335002](https://dev.azure.com/dnceng-public/public/_build/results?buildId=335002)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/335002/logs/303)|| |[334959](https://dev.azure.com/dnceng-public/public/_build/results?buildId=334959)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/334959/logs/2352)|| |[334772](https://dev.azure.com/dnceng-public/public/_build/results?buildId=334772)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/334772/logs/2425)|| |[334717](https://dev.azure.com/dnceng-public/public/_build/results?buildId=334717)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/334717/logs/123)|| |[334699](https://dev.azure.com/dnceng-public/public/_build/results?buildId=334699)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/334699/logs/2418)|| |[334444](https://dev.azure.com/dnceng-public/public/_build/results?buildId=334444)|dotnet/runtime|Send to Helix|[Log](https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/334444/logs/123)|| Displaying 100 of 262 results #### Summary |24-Hour Hit Count|7-Day Hit Count|1-Month Count| |---|---|---| |6|55|262| ### Known issue validation **Build: :mag_right:** https://dev.azure.com/dnceng-public/public/_build/results?buildId=323604 **Error message validated:** `Work item System.IO.FileSystem.Watcher.Tests in job` **Result validation: :white_check_mark:** Known issue matched with the provided build. **Validation performed at:** 6/28/2023 9:26:54 PM UTC

Author:	eerhardt
Assignees:	-
Labels:	`blocking-clean-ci`, `area-Infrastructure`, `untriaged`, `Known Build Error`
Milestone:	-

carlossanlop commented 1 year ago

Yea, doesn't seem to be a FileSystemWatcher-only issue. The jobs are getting unexpectedly killed, according to the console log outputs. Example:

Log: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-release-70-staging-9199670faa4c4e8e81/Microsoft.Bcl.AsyncInterfaces.Tests/1/console.7114ae01.log?helixlogtype=result Output:


2023-07-11T00:05:26.562Z    INFO    run.py  run(48) main    Beginning reading of test results.
2023-07-11T00:05:26.563Z    INFO    run.py  __init__(42)    read_results    Searching '/private/tmp/helix/working/A8D4097B/w/9393087D/e' for test results files
2023-07-11T00:05:26.563Z    INFO    run.py  __init__(42)    read_results    Searching '/tmp/helix/working/A8D4097B/w/9393087D/uploads' for test results files
2023-07-11T00:05:26.563Z    WARNING run.py  __init__(55)    read_results    No results file found in any of the following formats: xunit, junit, trx
2023-07-11T00:05:26.563Z    INFO    run.py  packing_test_reporter(30)   report_results  Packing 0 test reports to '/tmp/helix/working/A8D4097B/w/9393087D/e/__test_report.json'
2023-07-11T00:05:26.564Z    INFO    run.py  packing_test_reporter(33)   report_results  Packed 1523 bytes

exit 127 ['Microsoft.Bcl.AsyncInterfaces.Tests' END OF WORK ITEM LOG: Command exited with 127]

akoeplinger commented 11 months ago

A couple observations from looking through a couple results:

Only happens on the 7.0 branches
Only happens on runtime-libraries-coreclr outerloop pipelines, across all OSes
The test doesn't get killed, instead it fails because dotnet couldn't be found:

./RunTests.sh: line 168: /tmp/helix/working/B2D409A4/p/dotnet: No such file or directory

I downloaded the test-runtime-net7.0-OSX-Release-arm64.zip correlation payload and it is indeed missing the dotnet binary and host, it only contains the shared framework.

akoeplinger commented 11 months ago

The 7.0 branch is missing this fix: https://github.com/dotnet/runtime/pull/79652

akoeplinger commented 11 months ago

Fixed by https://github.com/dotnet/runtime/pull/95011

dotnet / runtime