jdemaeyer / brightsky

JSON API for DWD's open weather data.
https://brightsky.dev/
MIT License
300 stars 18 forks source link

Investigate pre-selecting sources before querying weather #51

Closed jdemaeyer closed 4 years ago

jdemaeyer commented 4 years ago

Note: Deal with #52 first to see if this is actually worth it.


E.g. take three sources per observation type, then query weather with a source_id IN clause. In a second step, try caching the sources by lat/lon (rounded to .01 precision)

jdemaeyer commented 4 years ago

Query benchmarks before:

50 one-day queries,  by lat/lon, sequential:  10.57 s                     
50 one-day queries,  by lat/lon, parallel  :   3.59 s
50 one-week queries, by lat/lon, sequential:  54.56 s
50 one-week queries, by lat/lon, parallel  :  19.25 s
50 one-day queries,  by station, sequential:   0.89 s
50 one-day queries,  by station, parallel  :   0.40 s
50 one-week queries, by station, sequential:   2.42 s
50 one-week queries, by station, parallel  :   1.10 s
50 one-day queries,  by source,  sequential:   0.85 s
50 one-day queries,  by source,  parallel  :   0.37 s
50 one-week queries, by source,  sequential:   2.27 s
50 one-week queries, by source,  parallel  :   1.02 s

after:

50 one-day queries,  by lat/lon, sequential:   6.70 s
50 one-day queries,  by lat/lon, parallel  :   3.15 s
50 one-week queries, by lat/lon, sequential:  10.28 s
50 one-week queries, by lat/lon, parallel  :   5.71 s
50 one-day queries,  by station, sequential:   1.12 s
50 one-day queries,  by station, parallel  :   0.52 s
50 one-week queries, by station, sequential:   2.77 s
50 one-week queries, by station, parallel  :   1.25 s
50 one-day queries,  by source,  sequential:   0.52 s
50 one-day queries,  by source,  parallel  :   0.24 s
50 one-week queries, by source,  sequential:   1.93 s
50 one-week queries, by source,  parallel  :   0.89 s