search

nesrak1 / UABEA

c# uabe for newer versions of unity
MIT License
1.14k stars 145 forks source link

Crash when compress LZ4 #144

Closed Hiroshi-Tsukamoto closed 2 years ago

Hiroshi-Tsukamoto commented 2 years ago

One game has 3GB of data.unity3d. I decompress it with UABEA and it produces a 5GB file (it doesn't crash). I compress the file with LZ4. After waiting for a while, UABEA crashes.

Crash report:

Application: UABEAvalonia.exe
CoreCLR Version: 5.0.1321.56516
.NET Version: 5.0.13
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ArgumentException: Provided data seems to be corrupted.
   at LZ4ps.LZ4Codec.Encode32HC(Byte[] input, Int32 inputOffset, Int32 inputLength) in C:\Users\nesquack\Documents\GitReposLocal\AssetsTools.NET\AssetTools.NET\Extra\Decompressors\LZ4\Safe\LZ4Codec.Safe.cs:line 674
   at AssetsTools.NET.AssetBundleFile.Pack(AssetsFileReader reader, AssetsFileWriter writer, AssetBundleCompressionType compType) in C:\Users\nesquack\Documents\GitReposLocal\AssetsTools.NET\AssetTools.NET\Standard\AssetsBundleFileFormat\AssetsBundleFile.cs:line 445
   at UABEAvalonia.MainWindow.CompressBundle(BundleFileInstance bundleInst, String path, AssetBundleCompressionType compType) in C:\Users\nesquack\Documents\GitRepos\uabequickfixes\UABEA\UABEAvalonia\MainWindow.axaml.cs:line 663
   at UABEAvalonia.MainWindow.AskForLocationAndCompress() in C:\Users\nesquack\Documents\GitRepos\uabequickfixes\UABEA\UABEAvalonia\MainWindow.axaml.cs:line 490
   at UABEAvalonia.MainWindow.MenuCompress_Click(Object sender, RoutedEventArgs e) in C:\Users\nesquack\Documents\GitRepos\uabequickfixes\UABEA\UABEAvalonia\MainWindow.axaml.cs:line 202
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__140_0(Object state)
   at Avalonia.Threading.AvaloniaSynchronizationContext.<>c__DisplayClass5_0.<Post>b__0() in /_/src/Avalonia.Base/Threading/AvaloniaSynchronizationContext.cs:line 33
   at Avalonia.Threading.JobRunner.RunJobs(Nullable`1 priority) in /_/src/Avalonia.Base/Threading/JobRunner.cs:line 37
   at Avalonia.Win32.Win32Platform.WndProc(IntPtr hWnd, UInt32 msg, IntPtr wParam, IntPtr lParam) in /_/src/Windows/Avalonia.Win32/Win32Platform.cs:line 210
   at Avalonia.Win32.Interop.UnmanagedMethods.DispatchMessage(MSG& lpmsg)
   at Avalonia.Win32.Win32Platform.RunLoop(CancellationToken cancellationToken) in /_/src/Windows/Avalonia.Win32/Win32Platform.cs:line 157
   at Avalonia.Threading.Dispatcher.MainLoop(CancellationToken cancellationToken) in /_/src/Avalonia.Base/Threading/Dispatcher.cs:line 61
   at Avalonia.Controls.ApplicationLifetimes.ClassicDesktopStyleApplicationLifetime.Start(String[] args) in /_/src/Avalonia.Controls/ApplicationLifetimes/ClassicDesktopStyleApplicationLifetime.cs:line 107
   at Avalonia.ClassicDesktopStyleApplicationLifetimeExtensions.StartWithClassicDesktopLifetime[T](T builder, String[] args, ShutdownMode shutdownMode) in /_/src/Avalonia.Controls/ApplicationLifetimes/ClassicDesktopStyleApplicationLifetime.cs:line 134
   at UABEAvalonia.Program.Main(String[] args) in C:\Users\nesquack\Documents\GitRepos\uabequickfixes\UABEA\UABEAvalonia\Program.cs:line 47
nesrak1 commented 2 years ago

This will be fixed in a later version, but bundles larger than 2GB will have issues right now.

nesrak1 commented 2 years ago

This should be fixed in the latest nightly build.

Hiroshi-Tsukamoto commented 2 years ago

I've tried it from the build here now, but it still crashes. https://nightly.link/nesrak1/UABEA/workflows/dotnet-desktop/master/uabea-windows.zip

Crash report:

Application: UABEAvalonia.exe
CoreCLR Version: 5.0.1321.56516
.NET Version: 5.0.13
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ArgumentException: Provided data seems to be corrupted.
   at LZ4ps.LZ4Codec.Encode32HC(Byte[] input, Int32 inputOffset, Int32 inputLength)
   at AssetsTools.NET.AssetBundleFile.Pack(AssetsFileReader reader, AssetsFileWriter writer, AssetBundleCompressionType compType, Boolean blockDirAtEnd)
   at UABEAvalonia.MainWindow.CompressBundle(BundleFileInstance bundleInst, String path, AssetBundleCompressionType compType) in D:\a\UABEA\UABEA\UABEAvalonia\MainWindow.axaml.cs:line 762
   at UABEAvalonia.MainWindow.AskForLocationAndCompress() in D:\a\UABEA\UABEA\UABEAvalonia\MainWindow.axaml.cs:line 584
   at UABEAvalonia.MainWindow.MenuCompress_Click(Object sender, RoutedEventArgs e) in D:\a\UABEA\UABEA\UABEAvalonia\MainWindow.axaml.cs:line 212
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__140_0(Object state)
   at Avalonia.Threading.AvaloniaSynchronizationContext.<>c__DisplayClass5_0.<Post>b__0() in /_/src/Avalonia.Base/Threading/AvaloniaSynchronizationContext.cs:line 33
   at Avalonia.Threading.JobRunner.RunJobs(Nullable`1 priority) in /_/src/Avalonia.Base/Threading/JobRunner.cs:line 37
   at Avalonia.Win32.Win32Platform.WndProc(IntPtr hWnd, UInt32 msg, IntPtr wParam, IntPtr lParam) in /_/src/Windows/Avalonia.Win32/Win32Platform.cs:line 271
   at Avalonia.Win32.Interop.UnmanagedMethods.DispatchMessage(MSG& lpmsg)
   at Avalonia.Win32.Win32Platform.RunLoop(CancellationToken cancellationToken) in /_/src/Windows/Avalonia.Win32/Win32Platform.cs:line 198
   at Avalonia.Threading.Dispatcher.MainLoop(CancellationToken cancellationToken) in /_/src/Avalonia.Base/Threading/Dispatcher.cs:line 61
   at Avalonia.Controls.ApplicationLifetimes.ClassicDesktopStyleApplicationLifetime.Start(String[] args) in /_/src/Avalonia.Controls/ApplicationLifetimes/ClassicDesktopStyleApplicationLifetime.cs:line 126
   at Avalonia.ClassicDesktopStyleApplicationLifetimeExtensions.StartWithClassicDesktopLifetime[T](T builder, String[] args, ShutdownMode shutdownMode) in /_/src/Avalonia.Controls/ApplicationLifetimes/ClassicDesktopStyleApplicationLifetime.cs:line 175
   at UABEAvalonia.Program.Main(String[] args) in D:\a\UABEA\UABEA\UABEAvalonia\Program.cs:line 47

Do you need any other information?

nesrak1 commented 2 years ago

I actually have no idea what could cause the lz4 compressor to fail since it's just compressing data, there's no way that should become corrupted. On all of the games I have with large bundles, none of them have thrown this exception (including some 3 and 4gb ones). Is the game free? I might be able to download it and debug.

Hiroshi-Tsukamoto commented 2 years ago

This is it. But it's not free, and be careful NSFW. https://www.dlsite.com/maniax/work/=/product_id/RJ297937.html

dillytin commented 2 years ago

I think I'm having the same problem. I'm dealing with the 'core' file from West of Loathing game. It becomes more than 2GB when decompressed.

CoreCLR Version: 5.0.1321.56516
.NET Version: 5.0.13
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ArgumentOutOfRangeException: Non-negative number required. (Parameter 'count')
   at System.IO.BinaryReader.ReadBytes(Int32 count)
   at AssetsTools.NET.AssetBundleFile.Pack(AssetsFileReader reader, AssetsFileWriter writer, AssetBundleCompressionType compType) in C:\Users\nesquack\Documents\GitReposLocal\AssetsTools.NET\AssetTools.NET\Standard\AssetsBundleFileFormat\AssetsBundleFile.cs:line 414
   at UABEAvalonia.MainWindow.CompressBundle(BundleFileInstance bundleInst, String path, AssetBundleCompressionType compType) in C:\Users\nesquack\Documents\GitRepos\uabequickfixes\UABEA\UABEAvalonia\MainWindow.axaml.cs:line 663
   at UABEAvalonia.MainWindow.AskForLocationAndCompress() in C:\Users\nesquack\Documents\GitRepos\uabequickfixes\UABEA\UABEAvalonia\MainWindow.axaml.cs:line 490
   at UABEAvalonia.MainWindow.MenuCompress_Click(Object sender, RoutedEventArgs e) in C:\Users\nesquack\Documents\GitRepos\uabequickfixes\UABEA\UABEAvalonia\MainWindow.axaml.cs:line 202
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__140_0(Object state)
   at Avalonia.Threading.AvaloniaSynchronizationContext.<>c__DisplayClass5_0.<Post>b__0() in /_/src/Avalonia.Base/Threading/AvaloniaSynchronizationContext.cs:line 33
   at Avalonia.Threading.JobRunner.RunJobs(Nullable`1 priority) in /_/src/Avalonia.Base/Threading/JobRunner.cs:line 37
   at Avalonia.Win32.Win32Platform.WndProc(IntPtr hWnd, UInt32 msg, IntPtr wParam, IntPtr lParam) in /_/src/Windows/Avalonia.Win32/Win32Platform.cs:line 210
   at Avalonia.Win32.Interop.UnmanagedMethods.DispatchMessage(MSG& lpmsg)
   at Avalonia.Win32.Win32Platform.RunLoop(CancellationToken cancellationToken) in /_/src/Windows/Avalonia.Win32/Win32Platform.cs:line 157
   at Avalonia.Threading.Dispatcher.MainLoop(CancellationToken cancellationToken) in /_/src/Avalonia.Base/Threading/Dispatcher.cs:line 61
   at Avalonia.Controls.ApplicationLifetimes.ClassicDesktopStyleApplicationLifetime.Start(String[] args) in /_/src/Avalonia.Controls/ApplicationLifetimes/ClassicDesktopStyleApplicationLifetime.cs:line 107
   at Avalonia.ClassicDesktopStyleApplicationLifetimeExtensions.StartWithClassicDesktopLifetime[T](T builder, String[] args, ShutdownMode shutdownMode) in /_/src/Avalonia.Controls/ApplicationLifetimes/ClassicDesktopStyleApplicationLifetime.cs:line 134
   at UABEAvalonia.Program.Main(String[] args) in C:\Users\nesquack\Documents\GitRepos\uabequickfixes\UABEA\UABEAvalonia\Program.cs:line 47
nesrak1 commented 2 years ago

@dillytin This isn't the same issue. Non-negative number required was fixed in a newer version of AssetsTools.NET/UABEA. The issue is about Provided data seems to be corrupted which I still haven't figured out yet. Please download the latest nightly build and try again.

nesrak1 commented 2 years ago

I've finally found a file to reproduce on (a free game) and I've been able to track the issue down to a bug in lz4net that was fixed in K4os.Compression.LZ4. K4os isn't supported on .net 3.5 which is why I'm using the older lz4net. The commit I just made (in AssetsTools.NET) should fix this.

Hiroshi-Tsukamoto commented 2 years ago

I used lz4 compress by fourth release, and checked it not crash! Thank you!