search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.42k stars 4.76k forks source link

[6.0] GC test failures: Handle collector count overflows or underflows #81383

Open buyaa-n opened 1 year ago

buyaa-n commented 1 year ago

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=603897 Build error leg or test failing: GC/API/GCHandleCollector/Usage/Usage.sh Pull request: https://github.com/dotnet/runtime/pull/81291

Error Message

Fill the error message using known issues guidance.

{
  "ErrorMessage": "System.InvalidOperationException: Handle collector count overflows or underflows.",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Failed tests:

  1. GC_API._GCHandleCollector_UsageUsage._GCHandleCollector_Usage_Usage_sh()
  2. GC_Scenarios._DoublinkList_doublinkstaydoublinkstay._DoublinkList_doublinkstay_doublinkstay_sh()
  3. GC_Scenarios._DoublinkList_dlstackdlstack._DoublinkList_dlstack_dlstack_sh()
GC/API/GCHandleCollector/Usage/Usage.sh [FAIL]

      Unhandled Exception:
      System.InvalidOperationException: Handle collector count overflows or underflows.
         at System.Runtime.InteropServices.HandleCollector.Remove()
         at HandleCollectorTest.Finalize()
      [ERROR] FATAL UNHANDLED EXCEPTION: System.InvalidOperationException: Handle collector count overflows or underflows.
         at System.Runtime.InteropServices.HandleCollector.Remove()
         at HandleCollectorTest.Finalize()

      Return code:      1
      Raw output file:      /root/helix/work/workitem/uploads/Reports/GC.API/GCHandleCollector/Usage/Usage.output.txt
      Raw output:
      BEGIN EXECUTION
      /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Usage.dll ''
      Case 1 Passed!
      100, 100
      Case 2 Passed!
      Expected: 100
      Actual: 255
      END EXECUTION - FAILED
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/root/helix/work/correlation
      > /root/helix/work/workitem/e/GC/API/GCHandleCollector/Usage/Usage.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at GC_API._GCHandleCollector_Usage_Usage_._GCHandleCollector_Usage_Usage_sh()
      Output:

        Unhandled Exception:
        System.InvalidOperationException: Handle collector count overflows or underflows.
           at System.Runtime.InteropServices.HandleCollector.Remove()
           at HandleCollectorTest.Finalize()
        [ERROR] FATAL UNHANDLED EXCEPTION: System.InvalidOperationException: Handle collector count overflows or underflows.
           at System.Runtime.InteropServices.HandleCollector.Remove()
           at HandleCollectorTest.Finalize()

        Return code:      1
        Raw output file:      /root/helix/work/workitem/uploads/Reports/GC.API/GCHandleCollector/Usage/Usage.output.txt
        Raw output:
        BEGIN EXECUTION
        /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Usage.dll ''
        Case 1 Passed!
        100, 100
        Case 2 Passed!
        Expected: 100
        Actual: 255
        END EXECUTION - FAILED
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/root/helix/work/correlation
        > /root/helix/work/workitem/e/GC/API/GCHandleCollector/Usage/Usage.sh
  Finished:    GC.API.XUnitWrapper
  Discovering: GC.LargeMemory.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  GC.LargeMemory.XUnitWrapper (found 3 test cases)
  Starting:    GC.LargeMemory.XUnitWrapper (parallel test collections = on, max threads = 2)
  Finished:    GC.LargeMemory.XUnitWrapper
  Discovering: GC.Scenarios.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  GC.Scenarios.XUnitWrapper (found 31 test cases)
  Starting:    GC.Scenarios.XUnitWrapper (parallel test collections = on, max threads = 2)
    GC/Scenarios/DoublinkList/doublinkstay/doublinkstay.sh [FAIL]

      Return code:      1
      Raw output file:      /root/helix/work/workitem/uploads/Reports/GC.Scenarios/DoublinkList/doublinkstay/doublinkstay.output.txt
      Raw output:
      BEGIN EXECUTION
      /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false doublinkstay.dll ''
      Test should return with ExitCode 100 ...
      19996 DLinkNodes finalized
      Test Failed
      Expected: 100
      Actual: 1
      END EXECUTION - FAILED
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/root/helix/work/correlation
      > /root/helix/work/workitem/e/GC/Scenarios/DoublinkList/doublinkstay/doublinkstay.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at GC_Scenarios._DoublinkList_doublinkstay_doublinkstay_._DoublinkList_doublinkstay_doublinkstay_sh()
      Output:

        Return code:      1
        Raw output file:      /root/helix/work/workitem/uploads/Reports/GC.Scenarios/DoublinkList/doublinkstay/doublinkstay.output.txt
        Raw output:
        BEGIN EXECUTION
        /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false doublinkstay.dll ''
        Test should return with ExitCode 100 ...
        19996 DLinkNodes finalized
        Test Failed
        Expected: 100
        Actual: 1
        END EXECUTION - FAILED
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/root/helix/work/correlation
        > /root/helix/work/workitem/e/GC/Scenarios/DoublinkList/doublinkstay/doublinkstay.sh
    GC/Scenarios/DoublinkList/dlstack/dlstack.sh [FAIL]

      Return code:      1
      Raw output file:      /root/helix/work/workitem/uploads/Reports/GC.Scenarios/DoublinkList/dlstack/dlstack.output.txt
      Raw output:
      BEGIN EXECUTION
      /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false dlstack.dll ''
      Test should return with ExitCode 100 ...
       Performing Collect/Wait/Collect cycle again
       Finalized number stable at 9995
      9995 DLinkNodes finalized
      Test Failed
      Expected: 100
      Actual: 1
      END EXECUTION - FAILED
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/root/helix/work/correlation
      > /root/helix/work/workitem/e/GC/Scenarios/DoublinkList/dlstack/dlstack.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at GC_Scenarios._DoublinkList_dlstack_dlstack_._DoublinkList_dlstack_dlstack_sh()
      Output:

        Return code:      1
        Raw output file:      /root/helix/work/workitem/uploads/Reports/GC.Scenarios/DoublinkList/dlstack/dlstack.output.txt
        Raw output:
        BEGIN EXECUTION
        /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false dlstack.dll ''
        Test should return with ExitCode 100 ...
         Performing Collect/Wait/Collect cycle again
         Finalized number stable at 9995
        9995 DLinkNodes finalized
        Test Failed
        Expected: 100
        Actual: 1
        END EXECUTION - FAILED
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/root/helix/work/correlation
        > /root/helix/work/workitem/e/GC/Scenarios/DoublinkList/dlstack/dlstack.sh

Report

Build Definition Test Pull Request
2575696 dotnet-runtime GC/API/GCHandleCollector/Usage/Usage.sh #44696

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 1

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=603897 Error message validated: [System.InvalidOperationException: Handle collector count overflows or underflows.] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 3/14/2024 10:51:23 PM UTC

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/gc See info in area-owners.md if you want to be subscribed.

Issue Details
## Build Information Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=151010 Build error leg or test failing: GC/API/GCHandleCollector/Usage/Usage.sh Pull request: https://github.com/dotnet/runtime/pull/81291 ## Error Message Fill the error message using [known issues guidance](https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssues.md#how-to-fill-out-a-known-issue-error-message-section). Failed tests: 1. GC_API._GCHandleCollector_Usage_Usage_._GCHandleCollector_Usage_Usage_sh() 2. GC_Scenarios._DoublinkList_doublinkstay_doublinkstay_._DoublinkList_doublinkstay_doublinkstay_sh() 3. GC_Scenarios._DoublinkList_dlstack_dlstack_._DoublinkList_dlstack_dlstack_sh() ```log GC/API/GCHandleCollector/Usage/Usage.sh [FAIL] Unhandled Exception: System.InvalidOperationException: Handle collector count overflows or underflows. at System.Runtime.InteropServices.HandleCollector.Remove() at HandleCollectorTest.Finalize() [ERROR] FATAL UNHANDLED EXCEPTION: System.InvalidOperationException: Handle collector count overflows or underflows. at System.Runtime.InteropServices.HandleCollector.Remove() at HandleCollectorTest.Finalize() Return code: 1 Raw output file: /root/helix/work/workitem/uploads/Reports/GC.API/GCHandleCollector/Usage/Usage.output.txt Raw output: BEGIN EXECUTION /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Usage.dll '' Case 1 Passed! 100, 100 Case 2 Passed! Expected: 100 Actual: 255 END EXECUTION - FAILED Test Harness Exitcode is : 1 To run the test: > set CORE_ROOT=/root/helix/work/correlation > /root/helix/work/workitem/e/GC/API/GCHandleCollector/Usage/Usage.sh Expected: True Actual: False Stack Trace: at GC_API._GCHandleCollector_Usage_Usage_._GCHandleCollector_Usage_Usage_sh() Output: Unhandled Exception: System.InvalidOperationException: Handle collector count overflows or underflows. at System.Runtime.InteropServices.HandleCollector.Remove() at HandleCollectorTest.Finalize() [ERROR] FATAL UNHANDLED EXCEPTION: System.InvalidOperationException: Handle collector count overflows or underflows. at System.Runtime.InteropServices.HandleCollector.Remove() at HandleCollectorTest.Finalize() Return code: 1 Raw output file: /root/helix/work/workitem/uploads/Reports/GC.API/GCHandleCollector/Usage/Usage.output.txt Raw output: BEGIN EXECUTION /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Usage.dll '' Case 1 Passed! 100, 100 Case 2 Passed! Expected: 100 Actual: 255 END EXECUTION - FAILED Test Harness Exitcode is : 1 To run the test: > set CORE_ROOT=/root/helix/work/correlation > /root/helix/work/workitem/e/GC/API/GCHandleCollector/Usage/Usage.sh Finished: GC.API.XUnitWrapper Discovering: GC.LargeMemory.XUnitWrapper (method display = ClassAndMethod, method display options = None) Discovered: GC.LargeMemory.XUnitWrapper (found 3 test cases) Starting: GC.LargeMemory.XUnitWrapper (parallel test collections = on, max threads = 2) Finished: GC.LargeMemory.XUnitWrapper Discovering: GC.Scenarios.XUnitWrapper (method display = ClassAndMethod, method display options = None) Discovered: GC.Scenarios.XUnitWrapper (found 31 test cases) Starting: GC.Scenarios.XUnitWrapper (parallel test collections = on, max threads = 2) GC/Scenarios/DoublinkList/doublinkstay/doublinkstay.sh [FAIL] Return code: 1 Raw output file: /root/helix/work/workitem/uploads/Reports/GC.Scenarios/DoublinkList/doublinkstay/doublinkstay.output.txt Raw output: BEGIN EXECUTION /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false doublinkstay.dll '' Test should return with ExitCode 100 ... 19996 DLinkNodes finalized Test Failed Expected: 100 Actual: 1 END EXECUTION - FAILED Test Harness Exitcode is : 1 To run the test: > set CORE_ROOT=/root/helix/work/correlation > /root/helix/work/workitem/e/GC/Scenarios/DoublinkList/doublinkstay/doublinkstay.sh Expected: True Actual: False Stack Trace: at GC_Scenarios._DoublinkList_doublinkstay_doublinkstay_._DoublinkList_doublinkstay_doublinkstay_sh() Output: Return code: 1 Raw output file: /root/helix/work/workitem/uploads/Reports/GC.Scenarios/DoublinkList/doublinkstay/doublinkstay.output.txt Raw output: BEGIN EXECUTION /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false doublinkstay.dll '' Test should return with ExitCode 100 ... 19996 DLinkNodes finalized Test Failed Expected: 100 Actual: 1 END EXECUTION - FAILED Test Harness Exitcode is : 1 To run the test: > set CORE_ROOT=/root/helix/work/correlation > /root/helix/work/workitem/e/GC/Scenarios/DoublinkList/doublinkstay/doublinkstay.sh GC/Scenarios/DoublinkList/dlstack/dlstack.sh [FAIL] Return code: 1 Raw output file: /root/helix/work/workitem/uploads/Reports/GC.Scenarios/DoublinkList/dlstack/dlstack.output.txt Raw output: BEGIN EXECUTION /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false dlstack.dll '' Test should return with ExitCode 100 ... Performing Collect/Wait/Collect cycle again Finalized number stable at 9995 9995 DLinkNodes finalized Test Failed Expected: 100 Actual: 1 END EXECUTION - FAILED Test Harness Exitcode is : 1 To run the test: > set CORE_ROOT=/root/helix/work/correlation > /root/helix/work/workitem/e/GC/Scenarios/DoublinkList/dlstack/dlstack.sh Expected: True Actual: False Stack Trace: at GC_Scenarios._DoublinkList_dlstack_dlstack_._DoublinkList_dlstack_dlstack_sh() Output: Return code: 1 Raw output file: /root/helix/work/workitem/uploads/Reports/GC.Scenarios/DoublinkList/dlstack/dlstack.output.txt Raw output: BEGIN EXECUTION /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false dlstack.dll '' Test should return with ExitCode 100 ... Performing Collect/Wait/Collect cycle again Finalized number stable at 9995 9995 DLinkNodes finalized Test Failed Expected: 100 Actual: 1 END EXECUTION - FAILED Test Harness Exitcode is : 1 To run the test: > set CORE_ROOT=/root/helix/work/correlation > /root/helix/work/workitem/e/GC/Scenarios/DoublinkList/dlstack/dlstack.sh ```
Author: buyaa-n
Assignees: -
Labels: `area-GC-coreclr`, `blocking-clean-ci`, `untriaged`, `Known Build Error`
Milestone: -
Maoni0 commented 1 year ago

these all have to do with running finalizers and we haven't made any changes there in a long time. these tests are flaky and of low value. I'd be fine if we just disabled them.

I'm not aware anyone is using this HandleCollector thing.

carlossanlop commented 1 year ago
carlossanlop commented 1 year ago

Still being hit. Seen in this 6.0 PR today: https://github.com/dotnet/runtime/pull/86295

mangod9 commented 1 year ago

should we disable these tests, @Maoni0 ?

Maoni0 commented 1 year ago

I'm fine with disabling.

carlossanlop commented 1 year ago
carlossanlop commented 1 year ago

Still happening in 6.0 @dotnet/gc. This is the most recent hit:

  Starting:    GC.API.XUnitWrapper (parallel test collections = on, max threads = 2)
    GC/API/GCHandleCollector/Usage/Usage.sh [FAIL]

      Unhandled Exception:
      System.InvalidOperationException: Handle collector count overflows or underflows.
         at System.Runtime.InteropServices.HandleCollector.Remove()
         at HandleCollectorTest.Finalize()
      [ERROR] FATAL UNHANDLED EXCEPTION: System.InvalidOperationException: Handle collector count overflows or underflows.
         at System.Runtime.InteropServices.HandleCollector.Remove()
         at HandleCollectorTest.Finalize()

      Return code:      1
      Raw output file:      /root/helix/work/workitem/uploads/Reports/GC.API/GCHandleCollector/Usage/Usage.output.txt
      Raw output:
      BEGIN EXECUTION
      /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Usage.dll ''
      Case 1 Passed!
      100, 100
      Case 2 Passed!
      Expected: 100
      Actual: 255
      END EXECUTION - FAILED
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/root/helix/work/correlation
      > /root/helix/work/workitem/e/GC/API/GCHandleCollector/Usage/Usage.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at GC_API._GCHandleCollector_Usage_Usage_._GCHandleCollector_Usage_Usage_sh()
      Output:

        Unhandled Exception:
        System.InvalidOperationException: Handle collector count overflows or underflows.
           at System.Runtime.InteropServices.HandleCollector.Remove()
           at HandleCollectorTest.Finalize()
        [ERROR] FATAL UNHANDLED EXCEPTION: System.InvalidOperationException: Handle collector count overflows or underflows.
           at System.Runtime.InteropServices.HandleCollector.Remove()
           at HandleCollectorTest.Finalize()

        Return code:      1
        Raw output file:      /root/helix/work/workitem/uploads/Reports/GC.API/GCHandleCollector/Usage/Usage.output.txt
        Raw output:
        BEGIN EXECUTION
        /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Usage.dll ''
        Case 1 Passed!
        100, 100
        Case 2 Passed!
        Expected: 100
        Actual: 255
        END EXECUTION - FAILED
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/root/helix/work/correlation
        > /root/helix/work/workitem/e/GC/API/GCHandleCollector/Usage/Usage.sh
  Finished:    GC.API.XUnitWrapper

carlossanlop commented 9 months ago

I saw a new hit today. Did we get a chance to disable the test or did we miss it?:

carlossanlop commented 8 months ago

Still affecting 6.0. Example: https://github.com/dotnet/runtime/pull/99787