Closed Joe-Heffer-Shef closed 3 years ago
Get list of families from metadata endpoint
What are the overheads of a HTTP request?
Benchmark retrieving a month's data by iterating over sensors vs. families https://github.com/rcgsheffield/urban_flows/blob/master/awesome/benchmarks/family_vs_sensor.py By family is much faster. Many sensors have 0 readings for a certain time period so there's hundreds of pointless HTTP requests
ufdex response is partitioned by sensor and sorted by time
more efficient due to partitioning of netcdf files (by family and date)