search

mrworf / iceshelf

A simple tool to allow storage of signed, encrypted, incremental backups using Amazon's Glacier storage
GNU General Public License v2.0
32 stars 8 forks source link

JSON file will become too big #13

Open mrworf opened 5 years ago

mrworf commented 5 years ago

The way data is stored and processed today is using JSON. This works pretty well until you eventually have too many changes and JSON file gets too big. Since it's loaded into memory as one big block, this will become a scaling issue down the road.

Need to investigate other methods of storing the metadata or simply splitting it into multiple files to keep it simple (would prefer not to use sqlite or similar).

This issue is created so I can track thoughts and ideas as I experiment with different ways. Needless to say it must be backward compatible and not break existing backups.

securityfrog commented 5 years ago

I've had to complete some prelimnary investigation on this before @mrworf you may be interested in checking out BSON or YAML due to the size issues

mrworf commented 5 years ago

It's not so much the file size as holding it in memory. I like JSON because it's "human readable" meaning that it can be rescued/repaired/converted easier without having the reader. But hold it all in memory is less than ideal once the file gets big enough.