search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.47k stars 4.76k forks source link

Test failure WindowsAlternateDataStreamOverwrite #83659

Open v-wenyuxu opened 1 year ago

v-wenyuxu commented 1 year ago

Run: runtime-coreclr libraries-jitstress 20230319.1

Failed test:

net8.0-windows-Release-arm64-CoreCLR_checked-jitminopts-Windows.11.Arm64.Open

- System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar:$DATA\")
- System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar\")
- System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"\", alternateStream: \":bar:$DATA\")
- System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"\", alternateStream: \":bar\")
- System.IO.Tests.FileInfo_CopyTo_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar:$DATA\")
... ...

Error message:

System.IO.IOException : The parameter is incorrect. : 'C:\h\w\A6E40946\t#File_Copy_str_str_b_bq13ehkl.mbj\WindowsAlternateDataStreamOverwrite_330_kaeldovu\WindowsAlternateDataStreamOverwrite_331_pdfk8999:bar:$DATA'

Stack trace
   at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Windows.cs:line 84
   at System.IO.Tests.File_Copy_str_str_b.Copy(String source, String dest, Boolean overwrite) in /_/src/libraries/System.IO.FileSystem/tests/File/Copy.cs:line 267
   at System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(String defaultStream, String alternateStream) in /_/src/libraries/System.IO.FileSystem/tests/File/Copy.cs:line 346
   at InvokeStub_File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(Object, Object, IntPtr*)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=675511 Error message validated: [System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 5/16/2024 10:38:57 PM UTC

Report

Build Definition Test Pull Request
882912 dotnet/runtime System.IO.Tests.FileInfo_CopyTo_str_b.WindowsAlternateDataStreamOverwrite dotnet/runtime#109087
882909 dotnet/runtime System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite dotnet/runtime#109087
881273 dotnet/runtime System.IO.Tests.FileInfo_CopyTo_str_b.WindowsAlternateDataStreamOverwrite dotnet/runtime#110165
866690 dotnet/runtime System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite dotnet/runtime#109694
861633 dotnet/runtime System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite dotnet/runtime#109523
859520 dotnet/runtime System.IO.Tests.FileInfo_CopyTo_str_b.WindowsAlternateDataStreamOverwrite dotnet/runtime#109488

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 3 6

Known Issue Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}
ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/area-system-io See info in area-owners.md if you want to be subscribed.

Issue Details
Run: [runtime-coreclr libraries-jitstress 20230319.1](https://dev.azure.com/dnceng-public/public/_build/results?buildId=209919&view=ms.vss-test-web.build-test-results-tab&runId=3879622&resultId=162665&paneView=debug) Failed test: ``` net8.0-windows-Release-arm64-CoreCLR_checked-jitminopts-Windows.11.Arm64.Open - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar:$DATA\") - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar\") - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"\", alternateStream: \":bar:$DATA\") - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"\", alternateStream: \":bar\") - System.IO.Tests.FileInfo_CopyTo_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar:$DATA\") ... ... ``` **Error message:** ``` System.IO.IOException : The parameter is incorrect. : 'C:\h\w\A6E40946\t#File_Copy_str_str_b_bq13ehkl.mbj\WindowsAlternateDataStreamOverwrite_330_kaeldovu\WindowsAlternateDataStreamOverwrite_331_pdfk8999:bar:$DATA' Stack trace at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Windows.cs:line 84 at System.IO.Tests.File_Copy_str_str_b.Copy(String source, String dest, Boolean overwrite) in /_/src/libraries/System.IO.FileSystem/tests/File/Copy.cs:line 267 at System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(String defaultStream, String alternateStream) in /_/src/libraries/System.IO.FileSystem/tests/File/Copy.cs:line 346 at InvokeStub_File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(Object, Object, IntPtr*) at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) ```
Author: v-wenyuxu
Assignees: -
Labels: `arch-arm64`, `area-System.IO`, `os-windows`, `JitStress`
Milestone: -
danmoseley commented 1 year ago

Hi @v-wenyuxu thanks, please put test name not url in the title.

jozkee commented 1 year ago

FWIW: There are other tests using alternate data stream paths and they didn't fail. Only CopyFileEx with overwrite: true fails. I don't know what JitStress may be doing that affects this specific scenario.

jakobbotsch commented 1 year ago

Failed in https://dev.azure.com/dnceng-public/public/_build/results?buildId=250159&view=results again (in the zapdisable scenario, so there is no special JIT modes enabled there).

v-wenyuxu commented 1 year ago

Failed again in: runtime-coreclr libraries-jitstress 20230519.1

Failed test:

net8.0-windows-Release-arm64-CoreCLR_checked-zapdisable-Windows.11.Arm64.Open

- System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar:$DATA\")
- System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar\")
- System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"\", alternateStream: \":bar:$DATA\")
- System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"\", alternateStream: \":bar\")

Error message:

System.IO.IOException : The parameter is incorrect. : 'C:\h\w\BDDE0A0A\t#File_Copy_str_str_b_snfjrv1o.lza\WindowsAlternateDataStreamOverwrite_330_xpdes8ds\WindowsAlternateDataStreamOverwrite_331_2j4b3v36:bar:$DATA'

Stack trace
   at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Windows.cs:line 84
   at System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(String defaultStream, String alternateStream) in /_/src/libraries/System.IO.FileSystem/tests/File/Copy.cs:line 346
   at InvokeStub_File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(Object, Object, IntPtr*)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
adamsitnik commented 1 year ago

The error The parameter is incorrect usually means ERROR_INVALID_PARAMETER which means that and invalid value was passed to sys-call.

The logic around the sys-call seems very simple:

https://github.com/dotnet/runtime/blob/e91db04be24aac28fd041425fac014ef04d940b1/src/libraries/Common/src/Interop/Windows/Kernel32/Interop.CopyFile.cs#L11-L15

If we had a bug in System.IO, we would get it almost always. And it the provided logs I can see that it's throw on the second call to this sys-call (the first worked fine):

https://github.com/dotnet/runtime/blob/e91db04be24aac28fd041425fac014ef04d940b1/src/libraries/System.IO.FileSystem/tests/File/Copy.cs#L336-L346

Since it's specific to Windows arm64, I suspect that it's either a marshalling or runtime bug. @jkotas what do you think?

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/interop-contrib See info in area-owners.md if you want to be subscribed.

Issue Details
Run: [runtime-coreclr libraries-jitstress 20230319.1](https://dev.azure.com/dnceng-public/public/_build/results?buildId=209919&view=ms.vss-test-web.build-test-results-tab&runId=3879622&resultId=162665&paneView=debug) Failed test: ``` net8.0-windows-Release-arm64-CoreCLR_checked-jitminopts-Windows.11.Arm64.Open - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar:$DATA\") - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar\") - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"\", alternateStream: \":bar:$DATA\") - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"\", alternateStream: \":bar\") - System.IO.Tests.FileInfo_CopyTo_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar:$DATA\") ... ... ``` **Error message:** ``` System.IO.IOException : The parameter is incorrect. : 'C:\h\w\A6E40946\t#File_Copy_str_str_b_bq13ehkl.mbj\WindowsAlternateDataStreamOverwrite_330_kaeldovu\WindowsAlternateDataStreamOverwrite_331_pdfk8999:bar:$DATA' Stack trace at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Windows.cs:line 84 at System.IO.Tests.File_Copy_str_str_b.Copy(String source, String dest, Boolean overwrite) in /_/src/libraries/System.IO.FileSystem/tests/File/Copy.cs:line 267 at System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(String defaultStream, String alternateStream) in /_/src/libraries/System.IO.FileSystem/tests/File/Copy.cs:line 346 at InvokeStub_File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(Object, Object, IntPtr*) at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) ```
Author: v-wenyuxu
Assignees: -
Labels: `arch-arm64`, `area-System.Runtime.InteropServices`, `os-windows`, `JitStress`, `blocking-clean-ci-optional`
Milestone: Future
AaronRobinsonMSFT commented 1 year ago

And it the provided logs I can see that it's throw on the second call to this sys-call (the first worked fine): Since it's specific to Windows arm64, I suspect that it's either a marshalling or runtime bug.

@adamsitnik This is unlikely to an interop issue. I disagree with your analysis based on the above (2) statements.

Since all interop stub logic is idempotent, the first (1) and all subsequent operations run the same code. It is unlikely therefore that if the first succeeds and the second fails it has anything to do with the interop stub itself.

The second doesn't indicate anything to do with marshalling since interop generates IL and is platform agnostic - other than pointer size, in all cases I can think of. Also, since this is passing on the first run the IL and all interop code is identical across runs.

It is entirely possible there is a codegen issue, which we should consider. However, this is File I/O and based on the path in the failure it is entirely possible the Win32 File API is doing something the test isn't handling.

'C:\h\w\BDDE0A0A\t#File_Copy_str_str_b_snfjrv1o.lza\WindowsAlternateDataStreamOverwrite_330_xpdes8ds\WindowsAlternateDataStreamOverwrite_331_2j4b3v36:bar:$DATA'

In summary I don't think there is enough data to warrant sending this over to the interop team or any other runtime team at present. The owner of the File IO APIs should be able to run this on a Windows ARM64 machine and see if they can reproduce it and perhaps collect a bit more information.

AaronRobinsonMSFT commented 1 year ago

Ugh. This is a JIT stress issue. Nevermind, the codegen team needs to take a look.

ghost commented 1 year ago

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch See info in area-owners.md if you want to be subscribed.

Issue Details
Run: [runtime-coreclr libraries-jitstress 20230319.1](https://dev.azure.com/dnceng-public/public/_build/results?buildId=209919&view=ms.vss-test-web.build-test-results-tab&runId=3879622&resultId=162665&paneView=debug) Failed test: ``` net8.0-windows-Release-arm64-CoreCLR_checked-jitminopts-Windows.11.Arm64.Open - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar:$DATA\") - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar\") - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"\", alternateStream: \":bar:$DATA\") - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"\", alternateStream: \":bar\") - System.IO.Tests.FileInfo_CopyTo_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar:$DATA\") ... ... ``` **Error message:** ``` System.IO.IOException : The parameter is incorrect. : 'C:\h\w\A6E40946\t#File_Copy_str_str_b_bq13ehkl.mbj\WindowsAlternateDataStreamOverwrite_330_kaeldovu\WindowsAlternateDataStreamOverwrite_331_pdfk8999:bar:$DATA' Stack trace at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Windows.cs:line 84 at System.IO.Tests.File_Copy_str_str_b.Copy(String source, String dest, Boolean overwrite) in /_/src/libraries/System.IO.FileSystem/tests/File/Copy.cs:line 267 at System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(String defaultStream, String alternateStream) in /_/src/libraries/System.IO.FileSystem/tests/File/Copy.cs:line 346 at InvokeStub_File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(Object, Object, IntPtr*) at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) ```
Author: v-wenyuxu
Assignees: -
Labels: `arch-arm64`, `os-windows`, `JitStress`, `area-CodeGen-coreclr`, `blocking-clean-ci-optional`
Milestone: Future
AaronRobinsonMSFT commented 1 year ago

@dotnet/jit-contrib Is there anything the owner of this test could collect that would help with the investigation?

jakobbotsch commented 1 year ago

As I pointed out above the failure was in the jitminopts scenario and the recent one is in zapdisable. There's no abnormal jit stress enabled in these scenarios; the first one is like running debug codegen, the second one is like running without R2R. Is there any reason to believe this is a codegen issue? Have we tried reproducing the issue with the paths involved?

AaronRobinsonMSFT commented 1 year ago

As I pointed out above the failure was in the zapdisable stress mode and the recent one is in jitminopts.

Ah. I missed that this was reproducing on a different leg as well. Apologies.

Is there any reason to believe this is a codegen issue?

Not from my end. I naively set it to that since it was marked "JitStress".

Have we tried reproducing the issue with the paths involved?

+1

/cc @adamsitnik

jakobbotsch commented 1 year ago

Another data point: it frequently fails in runtime as part of NAOT runs too: https://runfo.azurewebsites.net/search/tests/?q=started%3A%7E7+definition%3Aruntime+name%3ASystem.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite

jkotas commented 1 year ago

@jkotas what do you think?

Based on the data presented so far, it is most likely Windows Arm64 specific OS issues. area-System.IO owners should investigate it and file a bug against the OS team as appropriate.

Also, I would check for potential machine setup issue. It is possible that the file system is misconfigured on some of the Windows Helix machines that makes the test fail.

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/area-system-io See info in area-owners.md if you want to be subscribed.

Issue Details
Run: [runtime-coreclr libraries-jitstress 20230319.1](https://dev.azure.com/dnceng-public/public/_build/results?buildId=209919&view=ms.vss-test-web.build-test-results-tab&runId=3879622&resultId=162665&paneView=debug) Failed test: ``` net8.0-windows-Release-arm64-CoreCLR_checked-jitminopts-Windows.11.Arm64.Open - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar:$DATA\") - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar\") - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"\", alternateStream: \":bar:$DATA\") - System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"\", alternateStream: \":bar\") - System.IO.Tests.FileInfo_CopyTo_str_b.WindowsAlternateDataStreamOverwrite(defaultStream: \"::$DATA\", alternateStream: \":bar:$DATA\") ... ... ``` **Error message:** ``` System.IO.IOException : The parameter is incorrect. : 'C:\h\w\A6E40946\t#File_Copy_str_str_b_bq13ehkl.mbj\WindowsAlternateDataStreamOverwrite_330_kaeldovu\WindowsAlternateDataStreamOverwrite_331_pdfk8999:bar:$DATA' Stack trace at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite) in /_/src/libraries/System.Private.CoreLib/src/System/IO/FileSystem.Windows.cs:line 84 at System.IO.Tests.File_Copy_str_str_b.Copy(String source, String dest, Boolean overwrite) in /_/src/libraries/System.IO.FileSystem/tests/File/Copy.cs:line 267 at System.IO.Tests.File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(String defaultStream, String alternateStream) in /_/src/libraries/System.IO.FileSystem/tests/File/Copy.cs:line 346 at InvokeStub_File_Copy_str_str_b.WindowsAlternateDataStreamOverwrite(Object, Object, IntPtr*) at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) ```
Author: v-wenyuxu
Assignees: -
Labels: `arch-arm64`, `area-System.IO`, `os-windows`, `area-CodeGen-coreclr`, `blocking-clean-ci-optional`
Milestone: Future
adamsitnik commented 1 year ago

Thank you all for your feedback, I am going to try to repro it on my Windows x64 and arm64 machines.

BruceForstall commented 1 year ago

@Jozkee @adamsitnik This issue is marked as both "blocking-clean-ci-optional" and "Milestone: Future". That doesn't make sense. If it's actually blocking, it should be in the current release milestone. If it's not blocking, remove the blocking label.

agocke commented 1 year ago

This is failing in Native AOT now as well. Since jitstress saw this first it would be good if someone from JIT could take a look.

jakobbotsch commented 1 year ago

@agocke Do you have any new evidence to suggest this is a JIT issue? I do not believe this is a JIT issue -- please see the comments above for context.

jeffhandley commented 1 year ago

Added this to the 9.0.0 milestone because of the frequency

AndyAyersMS commented 12 months ago

Another set of failures here: https://dev.azure.com/dnceng-public/public/_build/results?buildId=487061&view=ms.vss-test-web.build-test-results-tab

carlossanlop commented 9 months ago

I adjusted the KnownBuildError pattern. It was not catching many hits.

I found this in a 7.0 PR: https://github.com/dotnet/runtime/pull/98223

jkotas commented 8 months ago

Hit in https://github.com/dotnet/runtime/pull/100233

carlossanlop commented 6 months ago

@dotnet/area-system-io An unrealted deps flow PR hit this failure in 8.0: