Cysharp / MasterMemory

Embedded Typed Readonly In-Memory Document Database for .NET and Unity.
MIT License
1.52k stars 110 forks source link

OverflowException with large data #85

Closed rh-51662 closed 4 months ago

rh-51662 commented 1 year ago

OverflowException occurs when trying to add large data to DatabaseBuilder. Add large data using the ImmutableBuilder's Diff method does not throw OverflowException .

Is it possible to avoid this Exception by changing some settings?

Here is the source code I ran

var builder = new DatabaseBuilder();
var list = new List<Person>();
for (int i = 0; i < 100000000; i++)
{
    list.Add(new Person { PersonId = i, Age = i, Gender = (Gender)(i % 2), Name = "AAAAAAAAAAA" });
}
builder.Append(list);

ImmutableBuilder's Diff method does not throw OverflowException

var builder = new DatabaseBuilder();
var list = new List<Person>();

for (int i = 0; i < 100000000; i++)
{
    list.Add(new Person { PersonId = i, Age = i, Gender = (Gender)(i % 2), Name = "AAAAAAAAAAA" });
}

var firstData = list.First();
builder.Append(new List<Person>() { firstData });
byte[] dataA = builder.Build();
var db = new MemoryDatabase(dataA);

var builder1 = db.ToImmutableBuilder();
builder1.Diff(list.Skip(1).ToArray());
db = builder1.Build();

builder.Append(list);
MessagePack.MessagePackSerializationException   HResult=0x80131500   Message=Failed to serialize MasterMemorytest3.Model.Person[] value.   Source=MessagePack   スタック トレース:    場所 MessagePack.MessagePackSerializer.Serialize[T](MessagePackWriter& writer, T value, MessagePackSerializerOptions options)    場所 MessagePack.MessagePackSerializer.Serialize[T](IBufferWriter`1 writer, T value, MessagePackSerializerOptions options, CancellationToken cancellationToken)    場所 MasterMemory.DatabaseBuilderBase.AppendCore[T,TKey](IEnumerable`1 datasource, Func`2 indexSelector, IComparer`1 comparer)    場所 MasterMemorytest3.DatabaseBuilder.Append(IEnumerable`1 dataSource) (F:\\src\\source\\repos\\MasterMemorytest\\MasterMemorytest3\\MasterMemory\\DatabaseBuilder.cs):行 29    場所 MasterMemorytest3.Program.Test1() (F:\\src\\source\\repos\\MasterMemorytest\\MasterMemorytest3\\Program.cs):行 39    場所 MasterMemorytest3.Program.Main(String[] args) (F:\\src\\source\\repos\\MasterMemorytest\\MasterMemorytest3\\Program.cs):行 21    この例外は、最初にこの呼び出し履歴      [外部コード] でスローされました  内部例外 1: OverflowException: Arithmetic operation resulted in an overflow.
github-actions[bot] commented 1 year ago

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 30 days.

livwvil commented 11 months ago

I faced the same problem

github-actions[bot] commented 5 months ago

This issue is stale because it has been open 180 days with no activity. Remove stale label or comment or this will be closed in 30 days.