jdemaeyer / brightsky

JSON API for DWD's open weather data.
https://brightsky.dev/
MIT License
300 stars 18 forks source link

Explore database structure optimizations #31

Closed jdemaeyer closed 4 years ago

jdemaeyer commented 4 years ago

Can we remove lat/lon from the weather table but accomodate station location changes?

jdemaeyer commented 4 years ago

Benchmarks with old database structure:

Database creation time: 0:39:10 h

Total database size:
   675 MB
Table sizes:
   657 MB  weather

MOSMIX Re-parse: 0:09:09 h

50 one-day queries, sequential : 0:00:10.580000 h
50 one-day queries, parallel   : 0:00:04.340000 h
50 one-week queries, sequential: 0:01:05.320000 h
50 one-week queries, parallel  : 0:00:37.580000 h
jdemaeyer commented 4 years ago

Benchmarks with new database structure:

Database creation time: 0:33:55 h

Total database size:
   429 MB
Table sizes:
   410 MB  weather
     1 MB  sources

MOSMIX Re-parse: 0:05:53 h

50 one-day queries, sequential : 0:00:09.630000 h
50 one-day queries, parallel   : 0:00:03.330000 h
50 one-week queries, sequential: 0:00:47.870000 h
50 one-week queries, parallel  : 0:00:15.890000 h