Provide tooling for local storage of remote service data

[Guibod]

Now that we can read data from remote services, and access HUGE repositories such as MTGJSON and Scryfall, it is now important to offer a way to store them efficiently.

This imply:

• Pick a storage engine (most probably mongo, and maybe sqlite ?), and then stick to it/them
• Basic collection maintenance tools (CLI) to flush / backup data
• Take in consideration model evolution, how should we handle that ? It is ok to ask for a database reset on new release. But this mean that we must store the mightstone version in the storage.
• Efficient tools, based on asyncio that can write MANY things all at once
• Synchronization support, eg: replace an object with the same unique identifier whenever required
• CLI tools for reading /accessing data
• Programmatic tools to access the data
• Documentation for the poor souls interested in Mightstone features
• We need to start adressing indexation in this story

Also:

• Storage is bound to a model, which is bound to a service. At the moment Mightstone does not have its own format for data, this will be addressed later

[Guibod]

This ticket was partly addressed by #29

[Guibod]

Let’s overview what’s still need to be implemented:

• [ ] A set of CLI tools to maintain the storage backend (this is rather important for mongita)
  • [ ] Get metrics from the database (number of collections, collections size and item count)
  • [ ] A feature to flush the database (by service)
• [ ] Models should be indexed
• [ ] Models should use beanie relationship Link see Relations
• [ ] Performance tests for download / write items from scryfall dumps

More complex matters that should be done in their own issues:

• [ ] Demonstrate a model migration solution to introduce a field name change or something like that
• [ ] Unified internal data model for key items such as cards, sets, decklist, people...