Awesome: grab data by family rather than sensor

rcgsheffield / urban_flows

Research Computing work for the Urban Flows project

2 stars 2 forks source link

Closed Joe-Heffer-Shef closed 3 years ago

Joe-Heffer-Shef commented 3 years ago

more efficient due to partitioning of netcdf files (by family and date)

Joe-Heffer-Shef commented 3 years ago

Get list of families from metadata endpoint

Joe-Heffer-Shef commented 3 years ago

What are the overheads of a HTTP request?

Joe-Heffer-Shef commented 3 years ago

Benchmark retrieving a month's data by iterating over sensors vs. families https://github.com/rcgsheffield/urban_flows/blob/master/awesome/benchmarks/family_vs_sensor.py By family is much faster. Many sensors have 0 readings for a certain time period so there's hundreds of pointless HTTP requests

Joe-Heffer-Shef commented 3 years ago

ufdex response is partitioned by sensor and sorted by time