Closed Meivyn closed 7 months ago
I would only comment on the code you posted,
var array = CustomSongsDataList.ToArray();
Make sure this is an array of ExtraSongSerializedData.
If possible, could you provide a minimal sample that works? (The above code is not executable, and the type of CustomSongsDataList is unknown. That way we may be able to investigate a bit further.
What I currently use is this, which also produces the same error
using System.Collections.Concurrent;
using System.ComponentModel;
using System.Diagnostics;
using MemoryPack;
using MemoryPack.Streaming;
using Newtonsoft.Json;
namespace ConsoleApp
{
internal class Program
{
private static ConcurrentDictionary<string, ExtraSongData> CustomSongsData = new();
private static string DataPath = "SongCoreExtraData.json";
private static async Task Main(string[] args)
{
var json = await File.ReadAllTextAsync(DataPath);
CustomSongsData = JsonConvert.DeserializeObject<ConcurrentDictionary<string, ExtraSongData>>(json)!;
await MemoryPack();
}
private static async Task MemoryPack()
{
var startTime = Stopwatch.GetTimestamp();
await using var fileStream = File.Open("data.json", FileMode.Create);
await MemoryPackStreamingSerializer.SerializeAsync(fileStream, CustomSongsData.Count, CustomSongsData);
Console.WriteLine($"Serialization: {CustomSongsData.Count} entries in {Stopwatch.GetElapsedTime(startTime)}");
fileStream.Position = 0;
startTime = Stopwatch.GetTimestamp();
CustomSongsData.Clear();
await foreach (var pair in MemoryPackStreamingSerializer.DeserializeAsync<KeyValuePair<string, ExtraSongData>>(fileStream))
{
CustomSongsData.TryAdd(pair.Key, pair.Value);
}
Console.WriteLine($"Deserialization: {CustomSongsData.Count} entries in {Stopwatch.GetElapsedTime(startTime)}");
}
}
[MemoryPackable]
public partial class ExtraSongData
{
public string[] _genreTags;
public Contributor[] contributors;
public string _customEnvironmentName;
public string _customEnvironmentHash;
public DifficultyData[] _difficulties;
public string _defaultCharacteristic = null;
public ColorScheme[] _colorSchemes;
public string[] _environmentNames;
public CharacteristicDetails[] _characteristicDetails;
}
[MemoryPackable]
public partial class MapColor
{
public float r;
public float g;
public float b;
[DefaultValue(1)]
public float a = 1f;
}
[MemoryPackable]
public partial class Contributor
{
public string _role;
public string _name;
public string _iconPath;
}
[MemoryPackable]
public partial class ColorScheme
{
public bool useOverride;
public string colorSchemeId;
public MapColor? saberAColor;
public MapColor? saberBColor;
public MapColor? environmentColor0;
public MapColor? environmentColor1;
public MapColor? obstaclesColor;
public MapColor? environmentColor0Boost;
public MapColor? environmentColor1Boost;
public MapColor? environmentColorW;
public MapColor? environmentColorWBoost;
}
[MemoryPackable]
public partial class RequirementData
{
public string[] _requirements;
public string[] _suggestions;
public string[] _warnings;
public string[] _information;
}
[MemoryPackable]
public partial class CharacteristicDetails
{
public string _beatmapCharacteristicName;
public string? _characteristicLabel;
public string? _characteristicIconFilePath = null;
}
[MemoryPackable]
public partial class DifficultyData
{
public string _beatmapCharacteristicName;
public BeatmapDifficulty _difficulty;
public string _difficultyLabel;
public RequirementData additionalDifficultyData;
public MapColor? _colorLeft;
public MapColor? _colorRight;
public MapColor? _envColorLeft;
public MapColor? _envColorRight;
public MapColor? _envColorWhite;
public MapColor? _envColorLeftBoost;
public MapColor? _envColorRightBoost;
public MapColor? _envColorWhiteBoost;
public MapColor? _obstacleColor;
public int? _beatmapColorSchemeIdx;
public int? _environmentNameIdx;
//PinkCore Port
public bool? _oneSaber;
public bool? _showRotationNoteSpawnLines;
//Tags
public string[] _styleTags;
}
}
Thank you for the reproduction code.
I have looked into it and it seems that your data has very long strings, over about 15KB.
(e.g. _difficultyLabel
.)
Since the default buffer size is 8KB, the buffer would overflow.
Check the argument for MemoryPackStreamingSerializer.DeserializeAsync.
The following seems to work:
MemoryPackStreamingSerializer.DeserializeAsync<ExtraSongData>(fileStream, bufferAtLeast: 65536, readMinimumSize: 65536))
bufferAtLeast
is the minimum buffer required. It must be longer than the maximum length of the utf8 encoding of your string. Try specifying a larger value.
I did try to specify a larger buffer already... but I guess it wasn't big enough. Thanks for taking the time to look into it, I appreciate it.
I am trying to implement MemoryPack into my existing serialization logic, but I'm getting this exception while I try to do so. I am really not sure if this is usage issue or lib issue, I based my code on what you did with your tests. It works when serializing only
string[]
orExtraSongData[]
.With this code (but do note that this also happens with
Dictionary<string, ExtraSongData>
):