When I query e.g. the past year from historical (2019-01/2019-12), which is within the 10-year time range of a single file, all historical files are loaded: Currently I count up to three, beginning in 1992. This doesn't seem efficient. Are all of these values loaded into the RAM?
I want to add that he was trying to mangle data from all stations, without any filtering on the station identifiers.
Details
We see the files representing historical data within the 10-minutes resolution [1] to be bundled into different files for each observation site, each one holding 10 years worth of data. I haven't been exactly aware of that detail, so thanks!
Observation
@wetterfrosch reported at https://github.com/earthobservations/wetterdienst/issues/250#issuecomment-734316263:
I want to add that he was trying to mangle data from all stations, without any filtering on the station identifiers.
Details
We see the files representing historical data within the 10-minutes resolution [1] to be bundled into different files for each observation site, each one holding 10 years worth of data. I haven't been exactly aware of that detail, so thanks!
Please note this is different with other data products like hourly resolution [2], where there is just one file per observation site.
[1] https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/10_minutes/air_temperature/historical/ [2] https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/hourly/air_temperature/historical/
Expected behavior
Only acquire historical data from the appropriate 10-year interval in order to improve efficiency and save resources.