search

IGITUGraz / SimRecorder

An high-performance library for recording and storing simulation data
GNU General Public License v3.0
4 stars 3 forks source link

Zarr backend - lmdb default size 1TB sparce #9

Open owilinux opened 1 year ago

owilinux commented 1 year ago

Following Problem: the data.mdb files are sparce with file size 1TB - this causes a lot of troubles when backuping data - rsync fails to recognise the sparce file type and reads 1TB to transfer, other file related tools show 1TB as real filesize - this is causing a lot of problems along the way - most software tools using lmdb files had to patch this problem later on. (see e.g. caffe https://github.com/BVLC/caffe/pull/3731)

please consider to fix this by using lmdb.open with writemap=True this avoids the wrong super large 1TB filesize for this normally small files. or limit map_size to much less than 1TB (see https://www.programcreek.com/python/example/106501/lmdb.open example 6)

regards oliver

owilinux commented 1 year ago

this fails also on 32bit Windows machines - not large enough address space...