Error decompressing on IL2CPP (Unity)

[in0finite]

ZstdSharp fails to decompress when running in IL2CPP build. When using .NET runtime (in Unity Editor), it works, but when building the application with IL2CPP backend, it fails to decompress data, and sometimes causes the application process to crash.

Steps to reproduce:

• Attach this script to any GameObject in scene:

using System;
using System.IO;
using UnityEngine;

namespace ZstdSharp
{
    public class ZstdTest : MonoBehaviour
    {
        public string[] sampleFileNames = new string[]
        {
            "Zstd_data_1.txt",
            "Zstd_data_2.txt",
        };

        void Start()
        {
            foreach (string sampleFile in this.sampleFileNames)
            {
                Debug.Log($"Testing sample: {sampleFile}");
                var compressed = File.ReadAllBytes(sampleFile).AsSpan();
                using var zstd = new ZstdSharp.Decompressor();
                var decompressed = zstd.Unwrap(compressed);
                Debug.Log($"Decompressed {compressed.Length} -> {decompressed.Length}");
            }
        }
    }
}

• Add these 2 files to project folder (next to Assets folder) :

  • Zstd_data_1.txt
  • Zstd_data_2.txt

• Enter play mode, and files will be decompressed normally (as visible in Console)

• Build and run the project with IL2CPP -> decompression fails -> application will log exceptions to C:\Users\{User}\AppData\LocalLow\DefaultCompany\{Project name}\Player.log :

  ZstdException: Data corruption detected
  at ZstdSharp.ThrowHelper.ThrowException (System.UIntPtr returnValue, System.String message) [0x00000] in <00000000000000000000000000000000>:0 
  at ZstdSharp.ThrowHelper.EnsureZstdSuccess (System.UIntPtr returnValue) [0x00000] in <00000000000000000000000000000000>:0 
  at ZstdSharp.Decompressor.Unwrap (System.ReadOnlySpan`1[T] src, System.Span`1[T] dest) [0x00000] in <00000000000000000000000000000000>:0 
  at ZstdSharp.Decompressor.Unwrap (System.ReadOnlySpan`1[T] src, System.Int32 maxDecompressedSize) [0x00000] in <00000000000000000000000000000000>:0 
  at ZstdSharp.ZstdTest.Decompress (System.Span`1[T] input) [0x00000] in <00000000000000000000000000000000>:0 
  at ZstdSharp.ZstdTest.Start () [0x00000] in <00000000000000000000000000000000>:0 

If you need any help to reproduce, let me know.

The version of Unity that I used is 2022.3.2.

[harayuu9]

I'm using it on 2022.1.24f1, but the compression seems strange. It was not restored to the correct binary when decompressing with zst compression → .net with IL2CPP.

[in0finite]

For me, decompression doesn't work at all, because right after the exception is logged, the crash happens - meaning process crashes instantly, probably because Decompressor writes outside of memory bounds.

@harayuu9 can you try to decompress those 2 files that I attached, in IL2CPP build ?

[oleg-st]

@in0finite

IL2CPP has some problems with nuint arithmetic

Code to reproduce:

private unsafe static nuint BIT_lookBitsFast(nuint bitContainer, uint bitsConsumed, uint nbBits)
{
    uint regMask = (uint)(sizeof(nuint) * 8 - 1);
    return bitContainer << (int)(bitsConsumed & regMask) >> (int)(regMask + 1 - nbBits & regMask);
}

The result of BIT_lookBitsFast((nuint)0xbbf569f6904bc319, 7, 7) should be 0x7d. Under IL2CPP I got 0xffffffffffffffd.

[in0finite]

Thank you for your reply.

Yeah, unfortunately IL2CPP is broken. Is there any trick we can make it work, eg. using #if ENABLE_IL2CPP and doing some specific stuff that would work on IL2CPP ? Because if not, then this library is basically unusable in Unity builds, so we would have to search for other solutions like native (c++) libraries.

[oleg-st]

I think the best solution is to report the error to IL2CPP.

Replacing nuint/nint with 64-bit ulong/long makes the decompression work. so in IL2CPP they just need to fix the nuint / nint type arithmetic. I can't replace the native integer types with 64-bit types because it would degrade performance on 32-bit platforms.

[oleg-st]

@in0finite could you report the issue to the Unity/IL2CPP?

[in0finite]

I submitted a bug report to Unity.

They gave me this link for it's status : https://unity3d.atlassian.net/servicedesk/customer/portal/2/IN-45480

I also commented on this post to notify Unity developer directly : https://discussions.unity.com/t/webgl-build-il2cpp-error-with-zstdsharp-dll/257862

[in0finite]

Update : they sent me email saying "Fixed and currently awaits internal confirmation", but no any additional info (eg. which version of Unity).

So I tried your suggestion to replace nuint/nint with integers, and indeed it worked :).

[oleg-st]

@in0finite good news :) I also submited a bug report yesterday.

[oleg-st]

The issue has been confirmed by Unity https://issuetracker.unity3d.com/issues/arithmetic-operations-are-incorrect-when-using-uint-in-il2cpp-player

[in0finite]

I just got an email saying that fix was verified for Unity 2023.1.11f1

[oleg-st]

Decompression seems to work in Unity 2022.3.9f1 (LTS), other versions with the fix are listed here: https://issuetracker.unity3d.com/issues/il2cpp-build-returns-incorrect-results-when-using-pointer-arithmetic

@in0finite Could you check it?

[in0finite]

Don't have the time right now to upgrade Unity version. Feel free to close the issue if you want. I solved the problem with #25.

[oleg-st]

So the problem is solved

@in0finite Thank you