search

kriszyp / lmdb-js

Simple, efficient, ultra-fast, scalable data store wrapper for LMDB
Other
520 stars 44 forks source link

Best Practices for Safely Using db.getRange #311

Open damartripamungkas opened 3 weeks ago

damartripamungkas commented 3 weeks ago

Hello! Are there any specific guidelines for retrieving a large dataset—such as 5 million records—using getRange? I am already utilizing the "start" and "end" options, but I need advice on other optional parameters to ensure safe and efficient data handling. Any guidance would be appreciated!

kriszyp commented 1 week ago

getRange should be quite efficient by default, as it will retrieve records on-demand for each iteration (doesn't hold the 5 million records in memory). However, if the query will continue for a long period of time and there are writes taking place and you aren't concerned about viewing an isolated snapshot of the data, you may want to consider disabling the snapshot flag (snapshot: false). This allows the read transaction to be continuously reset during iteration and avoids the cost of a long-lived read transactions (which causes database growth since it prevents in use old pages from being reclaimed/reused).