ZipFile.ExtractToDirectory failing with "The filename, directory name, or volume label syntax is incorrect." (works in SharpZipLib)

[smoothdeveloper]

Description

I have some code using System.IO.Compression:

ZipFile.ExtractToDirectory(archive, Path.Combine(directory.FullName, "groove-v1.0.0-midionly"))

Reproduction Steps

putting this in a .fsx file and running it with dotnet fsi:

#r "nuget: System.IO.Compression"
open System
open System.IO
open System.IO.Compression
open System.Net.Http
let directory = DirectoryInfo __SOURCE_DIRECTORY__
let archive = Path.Combine(__SOURCE_DIRECTORY__, "groove-v1.0.0-midionly.zip")
do
    let uri = "https://storage.googleapis.com/magentadata/datasets/groove/groove-v1.0.0-midionly.zip"
    use client = new HttpClient(Timeout = TimeSpan.FromMinutes 30)
    use resp = client.Send(new HttpRequestMessage(RequestUri=Uri uri))
    use stream = resp.Content.ReadAsStream()
    use file = File.OpenWrite(archive)
    stream.CopyTo(file)
//let fz = ICSharpCode.SharpZipLib.Zip.FastZip()
//fz.ExtractZip(archive, Path.Combine(directory.FullName, filesetName), "")
// dotnet one fails one the groove dataset.
ZipFile.ExtractToDirectory(archive, Path.Combine(directory.FullName, "groove-v1.0.0-midionly"))

I tried on macos, and it doesn't seem to fail, so maybe it is platform dependent.

Expected behavior

I wish it would work, extracting without an exception.

Actual behavior

System.IO.IOException: The filename, directory name, or volume label syntax is incorrect. : 'C:\dev\src\github.com\smoothdeveloper\zcore-midi-fs\demo\groove-v1.0.0-midionly\groove\drummer8\session2\Icon ' at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options) at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize) at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize) at System.IO.Compression.ZipFileExtensions.ExtractToFile(ZipArchiveEntry source, String destinationFileName, Boolean overwrite) at System.IO.Compression.ZipFileExtensions.ExtractToDirectory(ZipArchive source, String destinationDirectoryName, Boolean overwriteFiles) at System.IO.Compression.ZipFile.ExtractToDirectory(String sourceArchiveFileName, String destinationDirectoryName, Encoding entryNameEncoding, Boolean overwriteFiles) at <StartupCode$FSI_0003>.$FSI_0003.main@() in

Regression?

No response

Known Workarounds

I am using FastZip class from ICSharpCode.SharpZipLib.Zip to work around the issue.

Configuration

Welcome to F# Interactive for .NET Core in Visual Studio. To execute code, either
  1. Use 'Send to Interactive' (Alt-Enter or right-click) from an F# script. The F# Interactive process will
     use any global.json settings associated with that script.
  2. Press 'Enter' to start. The F# Interactive process will use default settings.
> 

Microsoft (R) F# Interactive version 12.0.2.0 for F# 6.0
Copyright (c) Microsoft Corporation. All Rights Reserved.

• Windows 11 x64
• VS2022 preview
• dotnet 7 preview 2

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/area-system-io-compression See info in area-owners.md if you want to be subscribed.

Issue Details

---

### Description I have some code using System.IO.Compression: ```fsharp ZipFile.ExtractToDirectory(archive, Path.Combine(directory.FullName, "groove-v1.0.0-midionly")) ``` ### Reproduction Steps putting this in a .fsx file and running it with `dotnet fsi`: ```fsharp #r "nuget: System.IO.Compression" open System open System.IO open System.IO.Compression open System.Net.Http let directory = DirectoryInfo __SOURCE_DIRECTORY__ let archive = Path.Combine(__SOURCE_DIRECTORY__, "groove-v1.0.0-midionly.zip") do let uri = "https://storage.googleapis.com/magentadata/datasets/groove/groove-v1.0.0-midionly.zip" use client = new HttpClient(Timeout = TimeSpan.FromMinutes 30) use resp = client.Send(new HttpRequestMessage(RequestUri=Uri uri)) use stream = resp.Content.ReadAsStream() use file = File.OpenWrite(archive) stream.CopyTo(file) //let fz = ICSharpCode.SharpZipLib.Zip.FastZip() //fz.ExtractZip(archive, Path.Combine(directory.FullName, filesetName), "") // dotnet one fails one the groove dataset. ZipFile.ExtractToDirectory(archive, Path.Combine(directory.FullName, "groove-v1.0.0-midionly")) ``` I tried on macos, and it doesn't seem to fail, so maybe it is platform dependent. ### Expected behavior I wish it would work, extracting without an exception. ### Actual behavior > System.IO.IOException: The filename, directory name, or volume label syntax is incorrect. : 'C:\dev\src\github.com\smoothdeveloper\zcore-midi-fs\demo\groove-v1.0.0-midionly\groove\drummer8\session2\Icon ' at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options) at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize) at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize) at System.IO.Compression.ZipFileExtensions.ExtractToFile(ZipArchiveEntry source, String destinationFileName, Boolean overwrite) at System.IO.Compression.ZipFileExtensions.ExtractToDirectory(ZipArchive source, String destinationDirectoryName, Boolean overwriteFiles) at System.IO.Compression.ZipFile.ExtractToDirectory(String sourceArchiveFileName, String destinationDirectoryName, Encoding entryNameEncoding, Boolean overwriteFiles) at .$FSI_0003.main@() in ### Regression? _No response_ ### Known Workarounds I am using FastZip class from ICSharpCode.SharpZipLib.Zip to work around the issue. ### Configuration ``` Welcome to F# Interactive for .NET Core in Visual Studio. To execute code, either 1. Use 'Send to Interactive' (Alt-Enter or right-click) from an F# script. The F# Interactive process will use any global.json settings associated with that script. 2. Press 'Enter' to start. The F# Interactive process will use default settings. > Microsoft (R) F# Interactive version 12.0.2.0 for F# 6.0 Copyright (c) Microsoft Corporation. All Rights Reserved. ``` * Windows 11 x64 * VS2022 preview * dotnet 7 preview 2 ### Other information _No response_

Author:	smoothdeveloper
Assignees:	-
Labels:	`area-System.IO.Compression`
Milestone:	-

[danmoseley]

C:\dev\src\github.com\smoothdeveloper\zcore-midi-fs\demo\groove-v1.0.0-midionly\groove\drummer8\session2\Icon
'

Looks like there's a newline at the end of the path. That would cause this exception, too.

[danmoseley]

Newlines in paths are valid in POSIX (although wierd). Looking at SharpZipLib, it does some cleaning of paths before writing them in Windows

https://github.com/icsharpcode/SharpZipLib/blob/cc8dd78ed989888f6685da4cc009c529158738b4/src/ICSharpCode.SharpZipLib/Zip/WindowsNameTransform.cs#L29-L37 https://github.com/icsharpcode/SharpZipLib/blob/cc8dd78ed989888f6685da4cc009c529158738b4/src/ICSharpCode.SharpZipLib/Zip/WindowsNameTransform.cs#L210

In our code see essentially no sanitization. I notice that this sanitization idea was discussed here https://github.com/dotnet/runtime/issues/15938#issuecomment-169192513 and rejected, but nobody felt strongly. The idea was you could instead enumerate the zip entries manually and write them out with your own names. I think it is more useful for ExtractToDirectory to sanitize with underscore replacements, as 7zip and SharpZipLib behavior suggests. If someone didn't like the sanitization scheme, they could do it manually rather than using ExtractToDirectory.

The work required is

1. add a test zip file into runtime-assets that contains files with these various characters that should be sanitized. You might need to use Linux, such as WSL, to be able to create this zip - not sure. (As a test, try extracting it with sharpziplib or possibly 7zip and verify that it extracts correctly, using underscores in place of each special character)
2. after that PR is merged, wait a few hours for a PR to appear like this one that updates this repo to consume it.
3. add tests similar to this one, but that succeed rather than throw. verify they fail.
4. make tests pass by sanitizing probably here https://github.com/dotnet/runtime/blob/09a38d4807b0aa5e3ad3e69a4bf5bb50c710cd76/src/libraries/System.IO.Compression/src/System/IO/Compression/ZipArchiveEntry.cs#L1090 using the same rules as SharpZipLib. you might need to fix some pre existing tests that were previously expecting this to fail.
5. put up the changes as a PR into this repo.

[danmoseley]

@smoothdeveloper any interest in offering a PR for this?

[Danyy427]

@danmoseley I would like to help, but this is my first time. Could you point me in the right direction to start?

[danmoseley]

@Danyy427 for sure, I've broken out the steps above. First thing is to make a test zip and put it in the runtime-assets repo.

Before you get started, let's just double check with the owners of this area: @dotnet/area-system-io-compression that they support this change.

[danmoseley]

Meantime, you can follow the 'getting started' steps in this repo (linked from the README) to clone, build, and get tests passing for this repo without any changes.

[Danyy427]

@danmoseley the original question is in F# but I may write code in C#, right? Also if I understood correctly I am going to make some tests where the characters that need sanitization are being used, and upload them to runtime-assets?

Sure will read the "getting started"

[danmoseley]

Yes we exclusively use C# in this repo (except when testing VB or F# scenarios.)

Yes, you will want to create a little zip file that has all the wierd characters in its file names listed here. You will need to figure out how to do that, and it may involve (if you're on Windows) using Linux (via WSL) and the Unix 'zip' utility, for example. https://github.com/icsharpcode/SharpZipLib/blob/cc8dd78ed989888f6685da4cc009c529158738b4/src/ICSharpCode.SharpZipLib/Zip/WindowsNameTransform.cs#L29

[smoothdeveloper]

@danmoseley, thanks for the feedback on confirming the issue and the reason of it surfacing, sharing great insights.

I can't help with a fix for it but thanks for the offer; I am also happy to see @Danny427 is going to give it a try based on your guidance, and confirm it doesn't matter I was using it from F# code for purpose of the fix.

Thanks all!

[Danyy427]

@danmoseley I made that zip file with some test files/folders inside it. However, it is not possible to put '\0' in a filename even in Linux/UNIX. Should I attempt to write directly to my hard drive to put a null character or is it overkill/not necessary?

Update: I have just tested extracting the said zip file with 7zip and indeed all of the characters are replaced with underscores.

[danmoseley]

I wouldn't bother with \0 if it isn't obvious how someone would make such things.

The concept here is that a zip that extracts successfully on Unix should extract successfully on Windows. It's not a goal to extract random synthetic zips.

[Danyy427]

@danmoseley Ok, then how would I go about uploading the zip? Do I make a clone of runtime-assets, add the zip file and put up a pull request?

[danmoseley]

@Danyy427 exactly, once you have tested it by extracting it yourself with sharpziplib and verified that it produces the same content you put in, with the expected chars replaced in the file names.

[Danyy427]

@danmoseley will do, thank you!

[Danyy427]

@danmoseley I have verified with SharpLibZip FastZip that the zip file results in all characters replaced with underscores. Creating a pull request now.

[danmoseley]

I wouldn't bother with \0 if it isn't obvious how someone would make such things.

Also of course that would be invalid even on Unix, so it's unlikely any mainstream tool would allow you to name streams that way, as they could never be written to a file under that name.

[Danyy427]

@danmoseley How should I proceed now? Your original post suggests that I should wait for a PR and add tests. How would I go about adding tests and running them?

(I won't be able to attend to this for a few hours since I will probably have to sleep (it's late here))

[danmoseley]

How would I go about adding tests and running them?

First you need to get this repo cloned, built, and existing tests passing. Start at https://github.com/dotnet/runtime/blob/main/docs/workflow/README.md -- follow instructions for libraries.

[smoothdeveloper]

@Danny427 at first measure you may:

• pull the same .zip file, to iron your testing a bit (which will be a non reg one, once adjustment is done)
• try to write a small archive with "non cross platform file names" in it with the underlying Zip infrastructure in the BCL (not sure it is doable as I never tried)

[danmoseley]

@Danyy427 once he has tests passing locally (with no changes) should follow these instructions https://github.com/dotnet/runtime-assets/blob/main/README.md#optional-step-local-testing in order to be able to add a new test that consumes his new zip file.

[Danyy427]

@danmoseley I have been attempting to build the repo with the instructions but the build fails with a couple of errors such as this one:

error SYSLIB0003: 'PermissionSet' is obsolete: 'Code Access Security is not supported or honored by the runtime.' [D:\Runtime\runtime\src\tasks\MonoTargetsTasks\ILStrip\AssemblyStripper\AssemblyStripper.csproj]

I run the build command like this:

./build.cmd clr+libs -rc Release

and I have all of the required components that are listed. I came across this page but I don't know how to fix it in the build per se.

[danmoseley]

Wierd. @steveisok any idea?

[danmoseley]

I'll try building myself.

[Danyy427]

@danmoseley Just for clarification, I am talking about the base repo, I haven't changed anything, simply cloned using github desktop and ran the build command

[akoeplinger]

Hmm this is very weird, we explicitly disable SYSLIB0003 in the .csproj: https://github.com/dotnet/runtime/blob/bdbdaa8e2b4159c6839ca1c65b280fc25b0bbb82/src/tasks/MonoTargetsTasks/ILStrip/AssemblyStripper/AssemblyStripper.csproj#L7-L11

[Danyy427]

@akoeplinger @danmoseley I have checked my AssemblyStripper.csproj. It also contains the NoWarn

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>$(TargetFrameworkForNETCoreTasks);$(TargetFrameworkForNETFrameworkTasks)</TargetFrameworks>
    <EnableDefaultCompileItems>false</EnableDefaultCompileItems>
    <!--  CilStrip.Sources doesn't play nicely with nullability -->
    <Nullable>disable</Nullable>
    <!-- SYSLIB0003: Mono.Cecil uses Code Access Security types
         CS0618: Mono.Cecil uses a number of other obsolete types
         CS0649: Mono.Cecil.Binary.ExportTable has some unused fields
    -->
    <NoWarn>$(NoWarn),CS0618,CS0649,SYSLIB0003</NoWarn>
    <!-- the analyzers don't like Mono.Cecil either -->
    <RunAnalyzers>false</RunAnalyzers>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.DotNet.CilStrip.Sources" Version="$(MicrosoftDotNetCilStripSourcesVersion)" />
  </ItemGroup>

  <ItemGroup>
    <Compile Include="AssemblyStripper.cs" />
  </ItemGroup>
</Project>

Here is the entirety of the errors for completeness:

:\Users\Asus\.nuget\packages\microsoft.dotnet.cilstrip.sources\7.0.0-beta.22171.1\contentFiles\cs\netstandard2.0\Mono.Cecil\SecurityDeclaration.cs(54,10): error SYSLIB0003: 'PermissionSet' is obsolete: 'Code Access Security is not supported or honored by the runtime.' [D:\Runtime\runtime\src\tasks\MonoTargetsTasks\ILStrip\AssemblyStripper\AssemblyStripper.csproj]
C:\Users\Asus\.nuget\packages\microsoft.dotnet.cilstrip.sources\7.0.0-beta.22171.1\contentFiles\cs\netstandard2.0\Mono.Cecil\SecurityDeclaration.cs(42,3): error SYSLIB0003: 'PermissionSet' is obsolete: 'Code Access Security is not supported or honored by the runtime.' [D:\Runtime\runtime\src\tasks\MonoTargetsTasks\ILStrip\AssemblyStripper\AssemblyStripper.csproj]
C:\Users\Asus\.nuget\packages\microsoft.dotnet.cilstrip.sources\7.0.0-beta.22171.1\contentFiles\cs\netstandard2.0\Mono.Cecil\SecurityDeclarationReader.cs(125,11): error SYSLIB0003: 'SecurityAttribute' is obsolete: 'Code Access Security is not supported or honored by the runtime.' [D:\Runtime\runtime\src\tasks\MonoTargetsTasks\ILStrip\AssemblyStripper\AssemblyStripper.csproj]
C:\Users\Asus\.nuget\packages\microsoft.dotnet.cilstrip.sources\7.0.0-beta.22171.1\contentFiles\cs\netstandard2.0\Mono.Cecil\SecurityDeclarationReader.cs(75,28): error SYSLIB0003: 'PermissionSet' is obsolete: 'Code Access Security is not supported or honored by the runtime.' [D:\Runtime\runtime\src\tasks\MonoTargetsTasks\ILStrip\AssemblyStripper\AssemblyStripper.csproj]
C:\Users\Asus\.nuget\packages\microsoft.dotnet.cilstrip.sources\7.0.0-beta.22171.1\contentFiles\cs\netstandard2.0\Mono.Cecil\SecurityDeclarationReader.cs(75,43): error SYSLIB0003: 'PermissionState' is obsolete: 'Code Access Security is not supported or honored by the runtime.' [D:\Runtime\runtime\src\tasks\MonoTargetsTasks\ILStrip\AssemblyStripper\AssemblyStripper.csproj]
C:\Users\Asus\.nuget\packages\microsoft.dotnet.cilstrip.sources\7.0.0-beta.22171.1\contentFiles\cs\netstandard2.0\Mono.Cecil\SecurityDeclarationReader.cs(91,6): error SYSLIB0003: 'SecurityAttribute' is obsolete: 'Code Access Security is not supported or honored by the runtime.' [D:\Runtime\runtime\src\tasks\MonoTargetsTasks\ILStrip\AssemblyStripper\AssemblyStripper.csproj]
C:\Users\Asus\.nuget\packages\microsoft.dotnet.cilstrip.sources\7.0.0-beta.22171.1\contentFiles\cs\netstandard2.0\Mono.Cecil\SecurityDeclarationReader.cs(99,7): error SYSLIB0003: 'IPermission' is obsolete: 'Code Access Security is not supported or honored by the runtime.' [D:\Runtime\runtime\src\tasks\MonoTargetsTasks\ILStrip\AssemblyStripper\AssemblyStripper.csproj]
C:\Users\Asus\.nuget\packages\microsoft.dotnet.cilstrip.sources\7.0.0-beta.22171.1\contentFiles\cs\netstandard2.0\Mono.Cecil\SecurityDeclarationReader.cs(131,4): error SYSLIB0003: 'SecurityAttribute' is obsolete: 'Code Access Security is not supported or honored by the runtime.' [D:\Runtime\runtime\src\tasks\MonoTargetsTasks\ILStrip\AssemblyStripper\AssemblyStripper.csproj]
C:\Users\Asus\.nuget\packages\microsoft.dotnet.cilstrip.sources\7.0.0-beta.22171.1\contentFiles\cs\netstandard2.0\Mono.Cecil\SecurityDeclarationReader.cs(137,61): error SYSLIB0003: 'SecurityAction' is obsolete: 'Code Access Security is not supported or honored by the runtime.' [D:\Runtime\runtime\src\tasks\MonoTargetsTasks\ILStrip\AssemblyStripper\AssemblyStripper.csproj]
C:\Users\Asus\.nuget\packages\microsoft.dotnet.cilstrip.sources\7.0.0-beta.22171.1\contentFiles\cs\netstandard2.0\Mono.Cecil\SecurityDeclarationReader.cs(137,93): error SYSLIB0003: 'SecurityAttribute' is obsolete: 'Code Access Security is not supported or honored by the runtime.' [D:\Runtime\runtime\src\tasks\MonoTargetsTasks\ILStrip\AssemblyStripper\AssemblyStripper.csproj]

[danmoseley]

BTW, the ingestion of runtime-assets is going on here https://github.com/dotnet/runtime/pull/67112

[danmoseley]

Hmm, succeeds for me also. How curious.

build.cmd clr+libs+libs.tests -rc Release
...
Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:15:06.92

[danmoseley]

@Danyy427 please try building just that folder with dotnet build src\tasks\MonoTargetsTasks\ILStrip\AssemblyStripper /bl

this will create an msbuild.binlog. can you share that? Note, it will contain your environment variables so you may choose to not share it publicly. In which case you can send it to the email address in my profile.

[Danyy427]

@danmoseley where is the log file located? Oops found it log.zip

[Danyy427]

@danmoseley were you able to download the file?

[danmoseley]

Yes. Here it is. Could it be the quoting?

DisabledWarnings = ;1701;1702;1705;1591,CS8969,CS0618,CS0649,SYSLIB0003;NU5105;1701;1702
..
CommandLineArguments = D:\Runtime\runtime\.dotnet\dotnet.exe exec "C:\Users\Asus\.nuget\packages\microsoft.net.compilers.toolset\4.2.0-2.22128.1\tasks\net6.0\bincore\csc.dll" /noconfig /unsafe- /checked- /nowarn:1701,1702,1705,1591,CS8969,CS0618,CS0649,"SYSLIB0003",NU5105,1701,1702 /fullpaths /nostdlib+ /errorreport:prompt /warn:9999 /define:TRACE;DEBUG;NET;NET6_0;NETCOREAPP;NET5_0_OR_GREATER;NET6_0_OR_GREATER;NETCOREAPP1_0_OR_GREATER;NETCOREAPP1_1_OR_GREATER;NETCOREAPP2_0_OR_GREATER;NETCOREAPP2_1_OR_GREATER;NETCOREAPP2_2_OR_GREATER;NETCOREAPP3_0_OR_GREATER;NETCOREAPP3_1_OR_GREATER \es\7.0.0-beta.22171.1\contentFiles\cs\netstandard2.0\Mono.Cecil.Binary\DOSHeader.cs C:,,,,...

C:\Users\Asus\.nuget\packages\microsoft.dotnet.cilstrip.sources\7.0.0-beta.22171.1\contentFiles\cs\netstandard2.0\Mono.Cecil\SecurityDeclaration.cs(54,10): error SYSLIB0003: 'PermissionSet' is obsolete: 'Code Access Security is not supported or honored by the runtime.' [D:\Runtime\runtime\src\tasks\MonoTargetsTasks\ILStrip\AssemblyStripper\AssemblyStripper.csproj]

[danmoseley]

Seems likely. My machine isn't quoting it:

DisabledWarnings = ;1701;1702;1705;1591,CS8969,CS0618,CS0649,SYSLIB0003;NU5105;1701;1702

Assembly = C:\Users\danmose\.nuget\packages\microsoft.net.compilers.toolset\4.2.0-2.22128.1\build\..\tasks\net6.0\Microsoft.Build.Tasks.CodeAnalysis.dll

CommandLineArguments = C:\git\runtime\.dotnet\dotnet.exe exec "C:\Users\danmose\.nuget\packages\microsoft.net.compilers.toolset\4.2.0-2.22128.1\tasks\net6.0\bincore\csc.dll" /noconfig /unsafe- /checked- /nowarn:1701,1702,1705,1591,CS8969,CS0618,CS0649,SYSLIB0003,NU5105,1701,1702 /fullpaths /nostdlib+ /errorreport:prompt /warn:9999 /define:TRACE;DEBUG;NETFRAMEWORK;NET472;NET20_OR_GREATER;NET30_OR_GREATER;NET35_OR_GREATER;NET40_OR_GREATER;NET45_OR_GREATER;NET451_OR_GREATER;NET452_OR_GREATER;NET46_OR_GREATER;NET461_OR_GREATER;NET462_OR_GREATER;NET47_OR_GREATER;NET471_OR_GREATER;NET472_OR_GREATER /highentropyva+ /nullable:disable /referen....

[Danyy427]

@danmoseley wow that's weird, how do I fix that exactly?

[danmoseley]

@Danyy427 I am psychically going to guess that your OS is set to the Turkish locale?

Therefore https://github.com/dotnet/msbuild/issues/7154 ... fix is to add "RegexOptions.CultureInvariant" here https://github.com/dotnet/msbuild/blob/6a79376cc50e5f0d829475adc418e7078776ccb7/src/Utilities/CommandLineBuilder.cs#L165

@Danyy427 try changing your OS to another locale (eg US English) - -does it work? @Forgind is that still scheduled to fix? It's going to become more impactful as we have more warnings named like SYSLIBnnnn (ie with an "i" in)

[danmoseley]

another workaround is to temporariy add #pragma warning disable SYSLIB0003 at the top of each of the files containing thes warnings.

[Danyy427]

@danmoseley Yes I am in Turkey, and yes it start the build without errors once I changed my region settings to the En (US), apparently display language was not enough. Thank you for that!

[danmoseley]

Great. I'm guessing you don't want to leave your OS set that way so you might want to do the other workaround I suggested. You can commit it locally to make your life easier -- of course when you have a PR, you would not include this change.

[Danyy427]

@danmoseley I don't think I will change it, at least for the duration of this PR, I don't think I must have my locale settings in Turkey for any particular reason. Thank you for your concerns though. I will mess around with the repo for now and as soon as the PR is merged will try to make a PR for the tests/fix itself.

[Danyy427]

@danmoseley Also, the ingestion thread seems to be closed, are the assets merged or are they going to be merged with the Roslyn stuff?

[danmoseley]

I put up a PR for the build issue https://github.com/dotnet/msbuild/pull/7498

yes, you're waiting on https://github.com/dotnet/runtime/pull/67189 now.

[Danyy427]

@danmoseley I have run all tests, to see if they pass but several don't pass and this of course tied to the locale business.

D:\Runtime\runtime\eng\testing\tests.targets(142,5): error : One or more tests failed while running tests from 'System.Runtime.Tests'. Please check D:\Runtime\runtime\artifacts\bin\System.Runtime.Tests\Debug\net7.0-windows\testResults.xml for details! [D:\Runtime\runtime\src\libraries\System.Runtime\tests\System.Runtime.Tests.csproj]
D:\Runtime\runtime\eng\testing\tests.targets(142,5): error : One or more tests failed while running tests from 'XsltCompiler.Tests'. Please check D:\Runtime\runtime\artifacts\bin\XsltCompiler.Tests\Debug\net7.0\testResults.xml for details! [D:\Runtime\runtime\src\libraries\System.Private.Xml\tests\Xslt\XsltCompiler\XsltCompiler.Tests.csproj]
D:\Runtime\runtime\eng\testing\tests.targets(142,5): error : One or more tests failed while running tests from 'System.Speech.Tests'. Please check D:\Runtime\runtime\artifacts\bin\System.Speech.Tests\Debug\net7.0-windows\testResults.xml for details! [D:\Runtime\runtime\src\libraries\System.Speech\tests\System.Speech.Tests.csproj]

The relevant lines in those xml files would be

<test name="System.Tests.TimeZoneInfoTests.TestNameWithInvariantCulture" type="System.Tests.TimeZoneInfoTests" method="TestNameWithInvariantCulture" time="4.2256622" result="Fail">

and also this one

<test name="System.Xml.Tests.XsltcTestSettings.Var1(param0: \&quot;/SETTİNGS:SCRİPT+ sft14.xsl\&quot;, param1: \&quot;sft14.dll\&quot;, param2: \&quot;no\&quot;, param3: \&quot;sft14\&quot;, param4: \&quot;sft14.pdb\&quot;, param5: \&quot;no\&quot;, param6: \&quot;sft14.txt\&quot;, englishOnly: True)" type="System.Xml.Tests.XsltcTestSettings" method="Var1" time="0.2884227" result="Fail">
        <traits>
          <trait name="category" value="XsltcExeRequired" />
        </traits>
        <failure exception-type="OLEDB.Test.ModuleCore.CTestFailedException">
          <message><![CDATA[OLEDB.Test.ModuleCore.CTestFailedException : Output was not as expected.\r\nExpected: `fatal error : Unrecognized option: '/SETTINGS:SCRIPT+'.\r\n`\r\nActual: `fatal error : Unrecognized option: '/SETTİNGS:SCRİPT+'.\r\n`\r\nResult: 0\r\n]]></message>
          <stack-trace><![CDATA[   at System.Xml.Tests.XsltcTestCaseBase.CompareOutput(Stream expectedStream, Stream actualStream, Int32 count) in D:\Runtime\runtime\src\libraries\System.Private.Xml\tests\Xslt\XsltCompiler\CommonScenarios\XsltcTestCaseBase.cs:line 96
   at System.Xml.Tests.XsltcTestCaseBase.VerifyTest(String cmdLine, String asmName, Boolean asmCreated, String typeName, String pdbName, Boolean pdbCreated, String baselineFile, Boolean runAssemblyVerification, Boolean loadFromFile) in D:\Runtime\runtime\src\libraries\System.Private.Xml\tests\Xslt\XsltCompiler\CommonScenarios\XsltcTestCaseBase.cs:line 173
   at System.Xml.Tests.XsltcTestCaseBase.VerifyTest(String cmdLine, String asmName, Boolean asmCreated, String typeName, String pdbName, Boolean pdbCreated, String baselineFile, Boolean loadFromFile) in D:\Runtime\runtime\src\libraries\System.Private.Xml\tests\Xslt\XsltCompiler\CommonScenarios\XsltcTestCaseBase.cs:line 132
   at System.Xml.Tests.XsltcTestSettings.Var1(Object param0, Object param1, Object param2, Object param3, Object param4, Object param5, Object param6, Boolean englishOnly) in D:\Runtime\runtime\src\libraries\System.Private.Xml\tests\Xslt\XsltCompiler\CommonScenarios\XsltcTestSettings.cs:line 67]]></stack-trace>
        </failure>
      </test>

I do not understand why these tests don't pass, should I also make my Region set to US? I will try to set my region as well and write the results here.

The third test didn't pass due to a firewall exception or so I believe.

[danmoseley]

@Danyy427 likely they are all related to Turkish culture. I am not sure what setting would work around that, but ultimately we need to fix them. Not here though -- for fixing this bug, simply ignore it. (Perhaps open a separate issue with the failures you hit)

Now that you have done this, you can run specifically only the tests in the libraries you care about. I think that is System.IO.Compression and/or System.IO.Compression.ZipFile. This doc shows how to do that.

Your next step is to make a test that consumes it and fails. You can't do this until #67189 goes in unless you do https://github.com/dotnet/runtime-assets/blob/main/README.md#optional-step-local-testing

[Danyy427]

@danmoseley Yup, I get off topic easily sorry about that, once done with this I will perhaps open an issue on those.

I will try the local testing and see if I can get it to fail, will put a PR once the tests are complete, If I cannot get local testing working I'll just wait for #67189 and clone the git repo again.

[danmoseley]

@Danyy427 the runtime-assets was merged so you should be good to go.

[Danyy427]

@danmoseley Thank you, will get to it.

[Danyy427]

@danmoseley I did write a test after cloning the repo and the file that I uploaded to runtime-assets isn't there. I also checked my local .nuget directory for the file and I found all the other tests cases but not the one that I require. Are we certain that the zip has been merged?

<test name="System.IO.Compression.Tests.ZipFile_Extract.Windows_ZipWithInvalidFileNames(zipFile: \&quot;InvalidWindowsFileNameChars.zip\&quot;, path: \&quot;Test______________________________________.txt\&quot;)" type="System.IO.Compression.Tests.ZipFile_Extract" method="Windows_ZipWithInvalidFileNames" time="0.0060686" result="Fail">
        <failure exception-type="System.IO.FileNotFoundException">
          <message><![CDATA[System.IO.FileNotFoundException : Could not find file 'D:\\Runtime\\runtime\\artifacts\\bin\\System.IO.Compression.ZipFile.Tests\\Debug\\net7.0-windows\\ZipTestData\\compat\\InvalidWindowsFileNameChars.zip'.]]></message>
          <stack-trace><![CDATA[   at Microsoft.Win32.SafeHandles.SafeFileHandle.CreateFile(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String fullPath, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.OSFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.SyncWindowsFileStreamStrategy..ctor(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategyCore(String path, FileMode mode, FileAccess access, FileShare share, FileOptions options, Int64 preallocationSize)
   at System.IO.Strategies.FileStreamHelpers.ChooseStrategy(FileStream fileStream, String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, Int64 preallocationSize)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean useAsync)
   at System.IO.Compression.ZipFile.Open(String archiveFileName, ZipArchiveMode mode, Encoding entryNameEncoding) in D:\Runtime\runtime\src\libraries\System.IO.Compression.ZipFile\src\System\IO\Compression\ZipFile.Create.cs:line 183
   at System.IO.Compression.ZipFile.ExtractToDirectory(String sourceArchiveFileName, String destinationDirectoryName, Encoding entryNameEncoding, Boolean overwriteFiles) in D:\Runtime\runtime\src\libraries\System.IO.Compression.ZipFile\src\System\IO\Compression\ZipFile.Extract.cs:line 186
   at System.IO.Compression.ZipFile.ExtractToDirectory(String sourceArchiveFileName, String destinationDirectoryName) in D:\Runtime\runtime\src\libraries\System.IO.Compression.ZipFile\src\System\IO\Compression\ZipFile.Extract.cs:line 43
   at System.IO.Compression.Tests.ZipFile_Extract.Windows_ZipWithInvalidFileNames(String zipFile, String path) in D:\Runtime\runtime\src\libraries\System.IO.Compression.ZipFile\tests\ZipFile.Extract.cs:line 119]]></stack-trace>
        </failure>
      </test>

[Danyy427]

@danmoseley I tried also with local testing but couldn't get it to work, do we have any idea why the repo doesn't contain the zip?

[danmoseley]

@Danyy427 try cleaning (git clean -fdx) the repo, git pull, and building from the root again?