search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.17k stars 4.72k forks source link

System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod failing on arm64 #1871

Open jaredpar opened 4 years ago

jaredpar commented 4 years ago

System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod

Console Log Summary

* ExceptionTests - reported call stack:
   at System.Tests.ExceptionTests.ThrowException() in /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs:line 143
   at System.Tests.ExceptionTests.ThrowAndRethrowOtherMethod(ValueTuple`3& rethrownExceptionStackFrame) in /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs:line 130
   at System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod() in /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs:line 118
    System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod [FAIL]
      Assert.Equal() Failure
      Expected: 131
      Actual:   130
      Stack Trace:
        /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs(154,0): at System.Tests.ExceptionTests.VerifyCallStack(ValueTuple`3 expectedStackFrame, String reportedCallStack, Int32 skipFrames)
        /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs(122,0): at System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod()
  Finished:    System.Runtime.Tests

Builds

Build Test Failure Count
#506244 2
#506364 2

Configurations

janvorli commented 4 years ago

@jaredpar it actually shows a message indicating the problem: _ZNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEC1Ev: symbol not found See #1723 for more details.

jaredpar commented 4 years ago

@janvorli thanks! Missed that.

Does look like that is the same issue as #1723. Will keep this one open to track getting the CI runs back to green and have #1723 track fixing the bug + enabling the run again.

safern commented 4 years ago

I just merged: https://github.com/dotnet/runtime/pull/2290 which fixes this, and now exposes a real test failure: 

* ExceptionTests - reported call stack:
   at System.Tests.ExceptionTests.ThrowException() in /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs:line 143
   at System.Tests.ExceptionTests.ThrowAndRethrowOtherMethod(ValueTuple`3& rethrownExceptionStackFrame) in /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs:line 130
   at System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod() in /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs:line 118
    System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod [FAIL]
      Assert.Equal() Failure
      Expected: 131
      Actual:   130
      Stack Trace:
        /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs(154,0): at System.Tests.ExceptionTests.VerifyCallStack(ValueTuple`3 expectedStackFrame, String reportedCallStack, Int32 skipFrames)
        /_/src/libraries/System.Runtime/tests/System/ExceptionTests.cs(122,0): at System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod()
  Finished:    System.Runtime.Tests
=== TEST EXECUTION SUMMARY ===

@jaredpar do we want to open a follow up issue for that one, or should we just keep this one open and track this failure getting fixed as part of this issue.

safern commented 4 years ago

I updated the title and description for this issue. @jkotas @stephentoub do you have any idea what might be happening and what would the difference be on alpine arm64 vs amd64? Could it be a docker image setup? @janvorli @wfurt?

stephentoub commented 4 years ago

cc: @mikem8361

safern commented 4 years ago

Actually it seems like it is not only alpine, also Ubuntu. https://helix.dot.net/api/2019-06-17/jobs/9983a98e-06ab-40bf-94f9-1a4216df4da0/workitems/System.Runtime.Tests/console

Which also runs in a docker container: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-helix-arm64v8-bfcd90a-20200127194925

Both are arm64. Updating the title.

mikem8361 commented 4 years ago

My recent PR #https://github.com/dotnet/runtime/pull/2269 to fix the stack trace line numbers may have broken this test. Investigating.

mikem8361 commented 4 years ago

Looks like PR https://github.com/dotnet/runtime/pull/31696 didn't fix this problem. I'm going to have to figure out how to debug it on one of our local arm64 pi's.

It is weird that it only fails on linux arm64 not windows arm64 though I couldn't find a windows arm64 library test leg.

jaredpar commented 4 years ago

@mikem8361

Don't believe we run the libraries tests on windows arm64 at the moment. Here is the relevant entry from runtime.yml

- template: /eng/pipelines/common/platform-matrix.yml
  parameters:
    jobTemplate: /eng/pipelines/libraries/run-test-job.yml
    buildConfig: ${{ variables.debugOnPrReleaseOnRolling }}
    platforms:
    - Windows_NT_x64
    - OSX_x64
    - Linux_x64
    - Linux_musl_x64
    - ${{ if eq(variables['isFullMatrix'], true) }}:
      - Linux_arm
      - Linux_arm64
      - Linux_musl_arm64
    - ${{ if eq(variables['isFullMatrix'], false) }}:
      - Windows_NT_x86

@safern can confirm my reading here.

safern commented 4 years ago

Yes that is correct, it fails on Linux arm64 and Linux_musl_arm64. We're waiting for necessary resources in order to be able to run tests on windows arm64, @trylek knows more about windows arm/arm64.

trylek commented 4 years ago

In mid-December the DDFUN team reinstalled the queue Windows.10.Arm64.Open we had been using previously for Windows arm32 testing to a new chipset that doesn't support the arm32 subsystem. That nuked Windows arm32 testing that doesn't work to this day. Based on my discussions with @MattGal and other dnceng and DDFUN people I received repeated confirmation that the queue Windows.10.Arm64.Open should be sufficient for Windows arm64 testing of the runtime repo. Last week I managed to push the first successful run through the queue and I sent out a PR for adding the Windows arm64 job to the PR runs:

https://github.com/dotnet/runtime/pull/2095

Once this gets approved and merged in, we'll start receiving Windows arm64 results in PR runs. Initially I don't want to add the arm64 job to outerloop runs as they are noisy enough already - let's monitor stability and pass rate of the arm64 runs say for a week and based on the results we can decide what other pipelines we want to add arm64 jobs to.

safern commented 4 years ago

I put up a PR to disable this tests for arm64 in the effort of getting the rolling builds green: https://github.com/dotnet/runtime/pull/31735

trylek commented 4 years ago

Thank you, just approved that.

tommcdon commented 3 years ago

This is an off-by-one-line number issue. We suggest updating the tests to be resilient against off-by one line issues between architectures. This is a common issue on both native and managed code.

safern commented 3 years ago

@tommcdon the test is disabled, should we keep the issue opened until we add that resiliency and we re-enable the test?

janvorli commented 3 years ago

This is just a test issue, moving to future.

jeffschwMSFT commented 1 year ago

Linking the following customer reported issue to this issue: https://developercommunity.visualstudio.com/t/NET-SDK-on-Ubuntu-18046-libnethosta-/10269507

jkotas commented 1 year ago

@jeffschwMSFT This does not look like the right issue.

jeffschwMSFT commented 1 year ago

I will create a new one ;)