search

CADWRDeltaModeling / dms_datastore

Data download and management tools for continuous data for Pandas. See documentation https://cadwrdeltamodeling.github.io/dms_datastore/
https://cadwrdeltamodeling.github.io/dms_datastore/
MIT License
1 stars 0 forks source link

variables flag not respected for agency ncro #30

Closed dwr-psandhu closed 6 months ago

dwr-psandhu commented 6 months ago 
populate_repo --agencies ncro --variables flow --dest raw

results in all ~6000 data streams to be downloaded.

water-e commented 6 months ago

I'll see what I can do. You have a use case for this?

At some point the cumulative issues with NCRO may motivate us to try to create a downloader based on web services.

water-e commented 6 months ago

Besides the fact that I don't have a use case for this, my concern here is that there are not a total of 6000 files in /raw for all the agencies, let alone NCRO. Should be 2000 across all the agencies in large time shards. Moreover, in my downloads NOAA is so dominant in its slowness (slightly USGS) that it would be the focus of any optimization.

dwr-psandhu commented 6 months ago

NOAA is fixed using the executor ( i followed your pattern). For NCRO it is not possible as it does all variables no matter what is specified.

water-e commented 6 months ago

Ready for testing