search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.27k stars 4.73k forks source link

TarWriter_WriteEntry_LongFile_Tests require 104GB free #77012

Closed danmoseley closed 1 year ago

danmoseley commented 2 years ago 
C:\git\runtime\artifacts\bin\System.Formats.Tar.Tests\Debug\net7.0-windows\testResults.xml:
 6061:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(entryFormat: Gnu, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests" method="WriteEntry_LongFileSize" time="9.3895568" result="Fail">
 6084:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(entryFormat: Pax, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests" method="WriteEntry_LongFileSize" time="0.0203511" result="Fail">
 6107:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(entryFormat: Pax, size: 8589934592, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests" method="WriteEntry_LongFileSize" time="0.0119002" result="Fail">
 6130:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: V7, size: 8589934591, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0089555" result="Fail">
 6152:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Ustar, size: 8589934591, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0068875" result="Fail">
 6174:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Gnu, size: 8589934591, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0039391" result="Fail">
 6189:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Pax, size: 8589934591, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0017366" result="Fail">
 6204:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Pax, size: 8589934592, unseekableStream: False)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.001866" result="Fail">
 6219:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: V7, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0020215" result="Fail">
 6234:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Ustar, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0022796" result="Fail">
 6249:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Gnu, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0016468" result="Fail">
 6264:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Pax, size: 8589934591, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0016205" result="Fail">
 6279:       <test name="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync(entryFormat: Pax, size: 8589934592, unseekableStream: True)" type="System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests" method="WriteEntry_LongFileSizeAsync" time="0.0019141" result="Fail">

example stack

        <failure exception-type="System.IO.IOException">
          <message><![CDATA[System.IO.IOException : There is not enough space on the disk. : 'C:\\Users\\test\\AppData\\Local\\Temp\\#TarWriter_WriteEntry_LongFile_Tests_ufupzyvy.aau\\WriteEntry_LongFileSize_33_hr1z2gp6']]></message>
          <stack-trace><![CDATA[   at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\RandomAccess.Windows.cs:line 171
   at System.IO.Strategies.OSFileStreamStrategy.Write(ReadOnlySpan`1 buffer) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\OSFileStreamStrategy.cs:line 246
   at System.IO.Strategies.OSFileStreamStrategy.Write(Byte[] buffer, Int32 offset, Int32 count) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\OSFileStreamStrategy.cs:line 236
   at System.IO.Strategies.BufferedFileStreamStrategy.WriteSpan(ReadOnlySpan`1 source, ArraySegment`1 arraySegment) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\BufferedFileStreamStrategy.cs:line 486
   at System.IO.Strategies.BufferedFileStreamStrategy.Write(Byte[] buffer, Int32 offset, Int32 count) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Strategies\BufferedFileStreamStrategy.cs:line 474
   at System.Formats.Tar.WrappedStream.Write(Byte[] buffer, Int32 offset, Int32 count) in C:\git\runtime\src\libraries\System.Formats.Tar\tests\WrappedStream.cs:line 65
   at System.IO.Stream.CopyTo(Stream destination, Int32 bufferSize) in C:\git\runtime\src\libraries\System.Private.CoreLib\src\System\IO\Stream.cs:line 79
   at System.Formats.Tar.TarHeader.WriteData(Stream archiveStream, Stream dataStream, Int64 actualLength) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarHeader.Write.cs:line 611
   at System.Formats.Tar.TarHeader.WriteAsGnuInternal(Stream archiveStream, Span`1 buffer) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarHeader.Write.cs:line 249
   at System.Formats.Tar.TarHeader.WriteAsGnu(Stream archiveStream, Span`1 buffer) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarHeader.Write.cs:line 193
   at System.Formats.Tar.TarWriter.WriteEntryInternal(TarEntry entry) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarWriter.cs:line 304
   at System.Formats.Tar.TarWriter.WriteEntry(TarEntry entry) in C:\git\runtime\src\libraries\System.Formats.Tar\src\System\Formats\Tar\TarWriter.cs:line 225
   at System.Formats.Tar.Tests.TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(TarEntryFormat entryFormat, Int64 size, Boolean unseekableStream) in C:\git\runtime\src\libraries\System.Formats.Tar\tests\TarWriter\TarWriter.WriteEntry.LongFile.Tests.cs:line 40
   at InvokeStub_TarWriter_WriteEntry_LongFile_Tests.WriteEntry_LongFileSize(Object, Object, IntPtr*)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)]]></stack-trace>

I have 68GB free on this disk (which includes %temp%). The failure is because these are all async tests, so they are running concurrently, requiring 13 * 8 GB = 104 GB which I don't have.

edit: actually there are 10 async tests, so 80GB, but the non async tests run concurrently, so at least 88GB plus some margin is required.

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/area-system-io See info in area-owners.md if you want to be subscribed.

Issue Details
``` C:\git\runtime\artifacts\bin\System.Formats.Tar.Tests\Debug\net7.0-windows\testResults.xml: 6061: 6084: 6107: 6130: 6152: 6174: 6189: 6204: 6219: 6234: 6249: 6264: 6279: ``` example stack ``` ``` I have 68GB free on this disk (which includes %temp%). The failure is because these are all async tests, so they are running concurrently, requiring 13 * 8 GB = 104 GB which I don't have.
Author: danmoseley
Assignees: -
Labels: `area-System.IO`
Milestone: -
jozkee commented 2 years ago

I don't understand, why does [Collection(nameof(DisableParallelization))] doesn't have an effect on async Task tests?

https://github.com/dotnet/runtime/blob/a41ac2c9457a91b1a2c3d5f06aad7fe1e249b658/src/libraries/System.Formats.Tar/tests/TarWriter/TarWriter.WriteEntryAsync.LongFile.Tests.cs#L11-L21

jozkee commented 2 years ago

cc @carlossanlop

danmoseley commented 2 years ago

I don't understand, why does [Collection(nameof(DisableParallelization))] doesn't have an effect on async Task tests?

Seems a lot of people are surprised by this -- https://github.com/xunit/xunit/issues/2003 You could change them to non-async tests, and make the tests wait on the async API. Or maybe create some kind of mutex they share.

BTW I was curious how paralleism in xunit works generally. Each class is an implicit collection of its own, and within a collection everything is serial (except async case above) but collections can run in parallel with each other. So [Collection(nameof(DisableParallelization))] does not mean "don't run anything else in parallel" it means "for the purposes of parallelism, treat all classes with this marker as a single class" or put another way "don't run any tests in this custom collection concurrently with each other" (except async case!) https://xunit.net/docs/running-tests-in-parallel#parallelism-in-test-frameworks

danmoseley commented 2 years ago

this should probably backport to 7 (when that opens again - after GA)

wfurt commented 1 year ago

I'm not sure if the link is relevant @danmoseley. They use MaxParallelThreads as method to disable parallelism and that is different from what we do IMHO. People seems to comment that disabling parallelization works ... with large performance hit. We use it for networking and it seems to work as expected. A single test takes more than hour to execute on my MacBook. (#79773)

jozkee commented 1 year ago

I ran the test locally with a Console.WriteLine at the time the file is created and other when the FileStream is disposed. It seems that for me locally the tests do run serial. @danmoseley could it be possible that the file deletion is no occurring at the moment that the FS is disposed but rather it is being scheduled and that's why you get "System.IO.IOException : There is not enough space on the disk"?

We could just catch this one exception, ignore it and continue execution. 

C:\git\runtime\src\libraries\System.Formats.Tar\tests [main]> dotnet build /t:Test /p:xunitmethodname=*WriteEntry_LongFileSizeAsync /p:outerloop=true
MSBuild version 17.5.0-preview-22580-01+ee6f71d15 for .NET
  Determining projects to restore...
  All projects are up-to-date for restore.
  Microsoft.Interop.SourceGeneration -> C:\git\runtime\artifacts\bin\Microsoft.Interop.SourceGeneration\Debug\netstandard2.0\Microsoft.Interop.SourceGeneration.dll
  LibraryImportGenerator -> C:\git\runtime\artifacts\bin\LibraryImportGenerator\Debug\netstandard2.0\Microsoft.Interop.LibraryImportGenerator.dll
  TestUtilities -> C:\git\runtime\artifacts\bin\TestUtilities\Debug\net6.0\TestUtilities.dll
  StreamConformanceTests -> C:\git\runtime\artifacts\bin\StreamConformanceTests\Debug\net8.0\StreamConformanceTests.dll
  System.Formats.Tar.Tests -> C:\git\runtime\artifacts\bin\System.Formats.Tar.Tests\Debug\net8.0-windows\System.Formats.Tar.Tests.dll
  ----- start Tue 12/20/2022 15:00:53.44 ===============  To repro directly: =====================================================
  pushd C:\git\runtime\artifacts\bin\System.Formats.Tar.Tests\Debug\net8.0-windows\
  "C:\git\runtime\artifacts\bin\testhost\net8.0-windows-Debug-x64\dotnet.exe" exec --runtimeconfig System.Formats.Tar.Tests.runtimeconfig.json --depsfile System.Formats.Tar.Tests.deps.js
  on xunit.console.dll System.Formats.Tar.Tests.dll -xml testResults.xml -nologo -method *WriteEntry_LongFileSizeAsync -notrait category=failing
  popd
  ===========================================================================================================
    Discovering: System.Formats.Tar.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Formats.Tar.Tests (found 1 of 625 test case)
    Starting:    System.Formats.Tar.Tests (parallel test collections = on, max threads = 12)
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_qq3iub23.epw\WriteEntry_LongFileSizeAsync_23_l4oaiclx
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_qq3iub23.epw\WriteEntry_LongFileSizeAsync_23_l4oaiclx
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_ud3zfpxs.eik\WriteEntry_LongFileSizeAsync_23_lhfctyer
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_ud3zfpxs.eik\WriteEntry_LongFileSizeAsync_23_lhfctyer
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_xt2yltgk.m24\WriteEntry_LongFileSizeAsync_23_xitgx3mm
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_xt2yltgk.m24\WriteEntry_LongFileSizeAsync_23_xitgx3mm
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_2ptcj32w.h0e\WriteEntry_LongFileSizeAsync_23_sclzpnul
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_2ptcj32w.h0e\WriteEntry_LongFileSizeAsync_23_sclzpnul
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_pirj3yeg.hba\WriteEntry_LongFileSizeAsync_23_i0r8hy3k
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_pirj3yeg.hba\WriteEntry_LongFileSizeAsync_23_i0r8hy3k
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_g4si45zo.xqo\WriteEntry_LongFileSizeAsync_23_b25vzlgh
     System.Formats.Tar.Tests: [Long Running Test] 'System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync', Elapsed: 00:02:00
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_g4si45zo.xqo\WriteEntry_LongFileSizeAsync_23_b25vzlgh
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_x4x0dcco.uzc\WriteEntry_LongFileSizeAsync_23_rt042dl6
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_x4x0dcco.uzc\WriteEntry_LongFileSizeAsync_23_rt042dl6
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_wkga4ufa.hci\WriteEntry_LongFileSizeAsync_23_6aokaa32
     System.Formats.Tar.Tests: [Long Running Test] 'System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync', Elapsed: 00:04:00
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_wkga4ufa.hci\WriteEntry_LongFileSizeAsync_23_6aokaa32
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_sg5xkwgi.cvt\WriteEntry_LongFileSizeAsync_23_ih5btuol
     System.Formats.Tar.Tests: [Long Running Test] 'System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync', Elapsed: 00:06:00
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_sg5xkwgi.cvt\WriteEntry_LongFileSizeAsync_23_ih5btuol
  Start - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_ti2a5q2b.04n\WriteEntry_LongFileSizeAsync_23_3q4obmgc
     System.Formats.Tar.Tests: [Long Running Test] 'System.Formats.Tar.Tests.TarWriter_WriteEntryAsync_LongFile_Tests.WriteEntry_LongFileSizeAsync', Elapsed: 00:08:00
  Finish - C:\Users\dacantu\AppData\Local\Temp\#TarWriter_WriteEntryAsync_LongFile_Tests_ti2a5q2b.04n\WriteEntry_LongFileSizeAsync_23_3q4obmgc
    Finished:    System.Formats.Tar.Tests
  === TEST EXECUTION SUMMARY ===
     System.Formats.Tar.Tests  Total: 10, Errors: 0, Failed: 0, Skipped: 0, Time: 496.031s
  ----- end Tue 12/20/2022 15:09:11.46 ----- exit code 0 ----------------------------------------------------------

Build succeeded.
    0 Warning(s)
    0 Error(s)
adamsitnik commented 1 year ago

could it be possible that the file deletion is no occurring at the moment that the FS is disposed but rather it is being scheduled

It should get deleted as soon as it gets disposed (https://stackoverflow.com/a/1895066/5852046). The question is when does it get disposed?

FileStream gets created with FileOptions.DeleteOnClose without being wrapped in an using block, then it's wrapped by WrappedStream with also no using block.

https://github.com/dotnet/runtime/blob/a41ac2c9457a91b1a2c3d5f06aad7fe1e249b658/src/libraries/System.Formats.Tar/tests/TarWriter/TarWriter.WriteEntryAsync.LongFile.Tests.cs#L23-L24

Later the WrappedStream is passed to TarReader which should get disposed at the end of the test:

https://github.com/dotnet/runtime/blob/a41ac2c9457a91b1a2c3d5f06aad7fe1e249b658/src/libraries/System.Formats.Tar/tests/TarWriter/TarWriter.WriteEntryAsync.LongFile.Tests.cs#L36

So the quesiton is whether the disposal of TarReader closes FileStream?

jozkee commented 1 year ago

So the quesiton is whether the disposal of TarReader closes FileStream?

It is not disposing it, but it should. So leaveOpen currently has no effect on the stream argument of TarReader. Fixing that may fix this issue.

Alternatively, we can manually dispose the FileStream after we are done with TarReader if we don't want to block on fixing the TarReader.Dispose bug.

Great question, Adam.

adamsitnik commented 1 year ago

Even with the correct disposal it takes around 10 minutes to execute single test case (https://github.com/dotnet/runtime/pull/79920#issuecomment-1363953136). We either need to optimize these tests, reduce the number of test cases or just turn them into manual tests that are not included in Outerloop

akoeplinger commented 1 year ago

The disk space issue was fixed, we already have one tracking the timeout issue here so closing this: https://github.com/dotnet/runtime/issues/79773