search

NigelCleland / lode

Scripts to automate getting data from electricity market sources
MIT License
1 stars 1 forks source link

Separate Database Tables by Year #16

Closed NigelCleland closed 10 years ago

NigelCleland commented 10 years ago

Currently Database Tables grow very very large, beyond the point of fast query times on standard desktop systems.

One way of solving this issue would be to separate the tables into individual years. This should limit the quantity of data involved to ~300-400mb as opposed to the 2GB currently. Then the query should be parsed initially to see which tables to query and if necessary broken down and ran across multiple tables.

This requires a wrapper to parse queries and then assign these to the appropriate tables.

In addition, it also requires smarter inserting into different tables. The insertion will have to be parsed to see which dates are in it so that it can be loaded into the appropriate table.

This is a bit of overhead but should speed things up considerably.

NigelCleland commented 10 years ago

Partially resolved here, at least some of the write side d7bf36499580e666a3504d789d4b1a783a5dde70

Still getting some strange issues for some of the tables, not being loaded in for some reason. Turns out I accidentally wiped some of my data...