Example needed: TrainFromBufferFastCover

minimalisticMe commented 11 months ago

Hi,

after the recent update there is a new function ZstdSharp.DictBuilder.TrainFromBufferFastCover, which I think enriches this library massively and makes zstd perfect for .NET usage. I want to try and train a custom compression dictionary but have trouble getting results.

Here is my code roughly minimized. First I collect a list of byte[] by reading all files (json files) and adding them to a list

private static List<byte[]> TestData()
{
    var testData = TestDataList(); // list of paths to json-files

    var testDataList = new List<byte[]>();
    foreach (var item in testData)
    {
        var content = File.ReadAllBytes(item);
        testDataList.Add(content);
    }

    return testDataList;
}

After that I want to train them by calling

var testData = TestData();
var bytes = ZstdSharp.DictBuilder.TrainFromBufferFastCover(testData, 5);

and write the result to a file for future use.

However calling ZstdSharp.DictBuilder.TrainFromBufferFastCover results in the following error and I don't know how to handle this correctly:

ZstdSharp.ZstdException : Src size is incorrect

    ThrowHelper.ThrowException(UIntPtr returnValue, String message)
    ThrowHelper.EnsureZdictSuccess(UIntPtr returnValue)
    DictBuilder.TrainFromBufferFastCover(IEnumerable`1 samples, ZDICT_fastCover_params_t params, Int32 dictCapacity)
    DictBuilder.TrainFromBufferFastCover(IEnumerable`1 samples, Int32 level, Int32 dictCapacity)
    TrainZstdDictionary.TrainDictionary() line 20
    RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
    MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

How do I train a custom dictionary on my own files correctly?