rr- / malgraph4

MALgraph: statistics service for MyAnimeList.net users.
http://mal.oko.im
Other
158 stars 39 forks source link

Optimize DB schema #63

Closed rr- closed 11 years ago

rr- commented 11 years ago

Right now DB has 400MB for 10k users. Some columns should be scaled down to make DB smaller.

rr- commented 11 years ago

Scaling columns is not possible in SQLite.

We can use custom INTEGER instead of VARCHAR(10) to store date times, but that's rather extreme approach. We can also "vacuum" the database sometimes (basically, it's like defragmentation).

oczki commented 11 years ago

Saving time in Unix format - is anything extreme about this?

rr- commented 11 years ago

The problem is that user can skip specifying month while specifying year. UNIX timestamps are not designed for this. So the solution would be doing something YEAR_13_32 + MONTH * 32 + DAY to store and doing proper divisions to get them back.

oczki commented 11 years ago

Ah, I see. Indeed, it sounds a bit extreme, but I'm all for this if it works toward smaller database.

rr- commented 11 years ago

There are also following workarounds:

rr- commented 11 years ago

We're going to delete old users occasionally when DB gets too large. Also see #72.