mbdavid / LiteDB

LiteDB - A .NET NoSQL Document Store in a single data file
http://www.litedb.org
MIT License
8.62k stars 1.25k forks source link

Causes of memory increase in LiteDB usage 5.0.16 [BUG] #2395

Open zangxq opened 11 months ago

zangxq commented 11 months ago

using LiteDB; using var db = new LiteDatabase(@"MyData.db");

try { int num = 0; while (true) { processData(db,num); if (num < 10000000) { num++; } else {

         num = 0;
    }
    Thread.Sleep(10);
}

} catch (Exception ex) { Console.WriteLine(ex.Message); Console.WriteLine(ex.StackTrace); } void processData(LiteDatabase? db,int num) { if (db == null) return; try { // 获取 Customers 集合 var col = db.GetCollection("customers"); // 创建一个对象 var customer = new Customer { Name = "John Doe"+ num.ToString(), Phones = new string[] { "8000-0000", "9000-0000" }, Age = 39, IsActive = true }; // 在 Name 字段上创建唯一索引 col.EnsureIndex(x => x.Name, true); // 数据插入 col.Insert(customer); Console.WriteLine("Insert ok:" + customer.Name); // 数据查询 List list = col.Find(x => x.Name==customer.Name).ToList(); Customer user = col.FindOne(x => x.Name == customer.Name);

    // Data deletion 
  //  col.Delete(user.Id);
    //col.DeleteMany(x => x.Id == user.Id);//The above deletion methods will not cause memory to rise.
    col.DeleteMany(Query.EQ("Name", user.Name));//Using this method will cause memory to rise.
    Console.WriteLine("Delete ok:" + user.Name);

    customer = null;
}
catch (Exception ex)
{
    Console.WriteLine(ex.Message);
    Console.WriteLine(ex.StackTrace);
}

} public class Customer { public int Id { get; set; } public string Name { get; set; } public int Age { get; set; } public string[] Phones { get; set; } public bool IsActive { get; set; } }

zangxq commented 11 months ago

DeleteMany(Query.EQ("Name", user.Name));//Using this method will cause memory to rise.

dethknite commented 9 months ago

The delete is done in memory, so that is expected. To control this one can do db transactions in chunks.. ie.

 while (min <= max)
 {
    foreach (objcet yourobj in col.FindAll().LongSkip(min).Take(step))
    {
        // Process record chunks here
    }

    //Update position
    min += step;
}

Then within the foreach, process the records (delete, update, etc.).

zangxq commented 9 months ago

col.DeleteMany(x => x.Id == user.Id);//The deletion methods will not cause memory to rise.