Reading and writing in sync mode problems after update to 1.4

hhyyrylainen commented 1 year ago

Describe the bug

I'm seeing issues in both reading data and writing data after updating this library from 1.3.3 to 1.4.2 (I think also an earlier 1.4.x version had similar issues).

For reading I randomly see the following issue (I use sync reading):

System.ArgumentException: BUG: buffer must have length BlockSize
  at ICSharpCode.SharpZipLib.Tar.TarBuffer.ReadBlockIntAsync (System.Byte[] buffer, System.Threading.CancellationToken ct, System.Boolean isAsync) [0x00023] in <82d3b1a41c8a4daeb7a8d496582b1183>:0 
  at System.Threading.Tasks.ValueTask.ThrowIfCompletedUnsuccessfully () [0x0001b] in <c50882760828481c927e00b7db67d1a5>:0 
  at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable+ConfiguredValueTaskAwaiter.GetResult () [0x00000] in <c50882760828481c927e00b7db67d1a5>:0 
  at ICSharpCode.SharpZipLib.Tar.TarInputStream.GetNextEntryAsync (System.Threading.CancellationToken ct, System.Boolean isAsync) [0x0016a] in <82d3b1a41c8a4daeb7a8d496582b1183>:0 
  at System.Threading.Tasks.ValueTask`1[TResult].get_Result () [0x00022] in <c50882760828481c927e00b7db67d1a5>:0 
  at System.Runtime.CompilerServices.ValueTaskAwaiter`1[TResult].GetResult () [0x00000] in <c50882760828481c927e00b7db67d1a5>:0 
  at ICSharpCode.SharpZipLib.Tar.TarInputStream.GetNextEntry () [0x00015] in <82d3b1a41c8a4daeb7a8d496582b1183>:0 
  at (wrapper remoting-invoke-with-check) ICSharpCode.SharpZipLib.Tar.TarInputStream.GetNextEntry()
  at Save.LoadDataFromFile (System.String file, System.Boolean info, System.Boolean save, System.Boolean screenshot) [0x0018e] in /home/hhyyrylainen/Projects/Thrive/src/saving/Save.cs:363 
  at Save.LoadFromFile (System.String file, System.Boolean info, System.Boolean save, System.Boolean screenshot, System.Action readFinished) [0x00035] in /home/hhyyrylainen/Projects/Thrive/src/saving/Save.cs:281 
  at Save.LoadFromFile (System.String saveName, System.Action readFinished) [0x00008] in /home/hhyyrylainen/Projects/Thrive/src/saving/Save.cs:89 
  at InProgressLoad.Step () [0x00097] in /home/hhyyrylainen/Projects/Thrive/src/saving/InProgressLoad.cs:101

I assume the exception BUG: buffer must have length BlockSize means that there is likely a bug in this library?

The other issue is in writing data, I use this in a game using Godot engine which means that the mono runtime is used, which might be the cause of this exception:

System.TypeInitializationException: The type initializer for 'ICSharpCode.SharpZipLib.Zip.ZipStrings' threw an exception. ---> System.EntryPointNotFoundException: GetCPInfoExW assembly:<unknown assembly> type:<unknown type> member:(null)
  at (wrapper managed-to-native) Interop+Kernel32.GetCPInfoExW(uint,uint,Interop/Kernel32/CPINFOEXW*)
  at Interop+Kernel32.TryGetACPCodePage (System.Int32& codePage) [0x00000] in <8020ff214f044836812da4de6be2aa56>:0 
  at System.Text.CodePagesEncodingProvider.get_SystemDefaultCodePage () [0x00000] in <8020ff214f044836812da4de6be2aa56>:0 
  at System.Text.CodePagesEncodingProvider.GetEncoding (System.Int32 codepage) [0x00011] in <8020ff214f044836812da4de6be2aa56>:0 
  at System.Text.EncodingProvider.GetEncodingFromProvider (System.Int32 codepage) [0x00017] in <5cf9631a76c548989e3186162d571fee>:0 
  at System.Text.Encoding.GetEncoding (System.Int32 codepage) [0x00000] in <5cf9631a76c548989e3186162d571fee>:0 
  at ICSharpCode.SharpZipLib.Zip.StringCodec.get_SystemDefaultEncoding () [0x00000] in <82d3b1a41c8a4daeb7a8d496582b1183>:0 
  at ICSharpCode.SharpZipLib.Zip.StringCodec.get_Default () [0x00000] in <82d3b1a41c8a4daeb7a8d496582b1183>:0 
  at ICSharpCode.SharpZipLib.Zip.ZipStrings..cctor () [0x00000] in <82d3b1a41c8a4daeb7a8d496582b1183>:0 
   --- End of inner exception stack trace ---
  at ICSharpCode.SharpZipLib.Zip.Compression.Streams.DeflaterOutputStream..ctor (System.IO.Stream baseOutputStream, ICSharpCode.SharpZipLib.Zip.Compression.Deflater deflater, System.Int32 bufferSize) [0x00007] in <82d3b1a41c8a4daeb7a8d496582b1183>:0 
  at ICSharpCode.SharpZipLib.GZip.GZipOutputStream..ctor (System.IO.Stream baseOutputStream, System.Int32 size) [0x00014] in <82d3b1a41c8a4daeb7a8d496582b1183>:0 
  at ICSharpCode.SharpZipLib.GZip.GZipOutputStream..ctor (System.IO.Stream baseOutputStream) [0x00000] in <82d3b1a41c8a4daeb7a8d496582b1183>:0 
  at (wrapper remoting-invoke-with-check) ICSharpCode.SharpZipLib.GZip.GZipOutputStream..ctor(System.IO.Stream)
  at Save.WriteDataToSaveFile (System.String target, System.String justInfo, System.String serialized, Godot.Image screenshot) [0x00049] in /home/hhyyrylainen/Projects/Thrive/src/saving/Save.cs:252 
  at Save.WriteRawSaveDataToFile (SaveInformation saveInfo, System.String saveContent, Godot.Image screenshot, System.String saveName) [0x00020] in /home/hhyyrylainen/Projects/Thrive/src/saving/Save.cs:239 
  at Save.SaveToFile () [0x00001] in /home/hhyyrylainen/Projects/Thrive/src/saving/Save.cs:214 
  at SaveHelper.PerformSave (InProgressSave inProgress, Save save) [0x0004e] in /home/hhyyrylainen/Projects/Thrive/src/saving/SaveHelper.cs:540 
  at SaveHelper+<>c__DisplayClass28_0.<InternalSaveHelper>b__1 (InProgressSave inProgress, Save save) [0x00028] in /home/hhyyrylainen/Projects/Thrive/src/saving/SaveHelper.cs:467 
  at InProgressSave.Step () [0x00155] in /home/hhyyrylainen/Projects/Thrive/src/saving/InProgressSave.cs:226 
  at Invoke._Process (System.Single delta) [0x0003c] in /home/hhyyrylainen/Projects/Thrive/src/engine/Invoke.cs:45

I tried looking but there doesn't seem to be a GZipOutputStream constructor that takes in an encoding object.

Reproduction Code

No response

Steps to reproduce

Try to write a tar.gz file in the following way:

https://github.com/Revolutionary-Games/Thrive/blob/a8ad16dfd02cda02839158149546b0362644f6c3/src/saving/Save.cs#L242

Or try to read such a created file like this (normal tar on my system can extract the file perfectly, which I think indicates that the 1.3.3 created file is fully correct): https://github.com/Revolutionary-Games/Thrive/blob/a8ad16dfd02cda02839158149546b0362644f6c3/src/saving/Save.cs#L328

Expected behavior

I expect that the code, which I linked above, would work in version 1.4.2. I hope I didn't use the library incorrectly, which is possible. I tried searching but I didn't find any porting guide for what changes I would need to make to keep my code working with 1.4.2 version.

Operating System

Linux

Framework Version

Other

Additional context

Update: I tried switching out the gzip layer from this library for the .NET system one and with that everything seems to work correctly. Or at least the error is so rare that I didn't manage to see it from a couple of testing runs. So maybe the issue is reading a combined tar and gzip (from this library) stream in a sync manner?

piksel commented 1 year ago

The gzip issue is related to this problem with mono, not sure how we should work around that... Perhaps some special runtime checks need to be performed to avoid it... https://github.com/mono/mono/issues/12603

The tar issue is probably related to some other bugs in tar that have been really hard to reproduce, bit this gives me another lead, thanks!

hhyyrylainen commented 1 year ago

I see, mono is being problematic again... Luckily we might at some point actually be able to update to Godot Engine 4.0 and get on the .NET 7 runtime, which will solve also other really persistent bugs with mono we get somewhat regular reports about.

The tar issue is probably related to some other bugs in tar that have been really hard to reproduce, bit this gives me another lead, thanks!

I had testing from other team members and it seems that the tar reader and writer work correctly when used in combination with the C# standard gzip stream type. Or again, the problem is just so rare that we won't see it happening before this change is included in a public release of ours.

icsharpcode / SharpZipLib