mbdavid / LiteDB

LiteDB - A .NET NoSQL Document Store in a single data file
http://www.litedb.org
MIT License
8.6k stars 1.25k forks source link

In version 5 of LiteDB, the file upload speed to the database has slowed down. #2544

Open BibaAndBoba23 opened 1 month ago

BibaAndBoba23 commented 1 month ago

Good day! We are using the fourth version of LiteDB as a file cache. Upon upgrading to version 5, we noticed a significant drop in performance. Example of data loading:

using LiteDatabase cache_database = new LiteDatabase(connectionString: this.connection_string); file_stream.Position = 0; LiteFileInfo lite_file_info = cache_database.FileStorage.Upload(id: $"{this.id_server}/{id_file}", filename: file_stream.Name, stream: file_stream);

maybe there are new ways to quickly download/upload files?

BibaAndBoba23 commented 1 month ago

You can see detailed information about the test results on this page: http://bug.sdpss.ru/issue/SDP-13037/Nizkaya-skorost-zagruzki-fajlov-pri-obnovlenii-versii-LiteDB-do-5

JKamsker commented 1 month ago

The detailed information is hidden behind an auth wall.

Lihoradit commented 1 month ago

Hello, we've encountered an issue with LiteDB5 where the file upload speed is slower than in version 4. The screenshot below shows the measurements for a 1GB file. image

Lihoradit commented 1 month ago

*Re-measurement with different file sizes: Calculation: minutes/seconds

5 files 1Gb
Cache size 0Gb 11 Gb
LiteDb 5 17:41,84 20:50,11
LiteDB 4 05:14,66 11:00.91
File tree size 76 Mb
Cache size 0Gb 11Gb
LiteDB 5 00:12,20 00:14,45
LiteDB 4 00:07,50 00:08,72
Image 135Mb
Cache size 0Gb 11Gb
LiteDB 5 00:26,05 00:33,30
LiteDB 4 00:05,11 00:05,56
10 files 20 Mb
Cache size 0Gb 11Gb
LiteDb 5 00:43,02 00:45,94
LiteDB 4 00:12,34 00:10,76

-On LiteDB 4, there seems to be a cap after which nothing is written to the cache, even though there is 65GB free on the C drive. -If you cancel the file copying process through document management, the temporary files remain in the cache folder and are not deleted. -The cache write limit is always different; on 11.09, my cap was 9.7GB, and on 12.09, it was 11.8GB. This might be slowing down the file copying speed to the disk.