knightcrawler-stremio / knightcrawler

A selfhosted Stremio addon
Apache License 2.0
238 stars 39 forks source link

Compress the database #86

Open purple-emily opened 5 months ago

purple-emily commented 5 months ago

The database can get very large very quickly. We need to look at the data structure.

A summary of a suggestion from Discord: it's likely that the biggest amount of data we are storing is the magnet links.

In a magnet link the majority of the text is going to be the trackers.

If we make a trackers table for each tracker we encounter and then link back to it we can deconstruct and reconstruct the magnet links in the code.

Using this method we could see a significant reduction in the size of the stored data.