google / hat-backup

Backend-agnostic snapshotting backup system
Apache License 2.0
251 stars 39 forks source link

We should not have linear memory usage #39

Open brinchj opened 8 years ago

brinchj commented 8 years ago

Listing all our hashes or keys is using linear memory and could potentially be too large for RAM.

Whenver we list the contents of an index, the result should be streamed out the underlying SQLite database. There is currently no way to do this with Diesel (except manually with limits), so we will revisit this later when Diesel and our codebase have both matured a bit more.