Update PM2.5 to use Dataset class

aiddata / geo-datasets

Scripts for preparing datasets in GeoQuery

http://geoquery.org

MIT License

20 stars 11 forks source link

Update PM2.5 to use Dataset class #104

Closed jacobwhall closed 1 year ago

jacobwhall commented 1 year ago

pm25 now has one main script with class definition

jacobwhall commented 1 year ago

Just force-pushed an update to this branch so that it can be merged into develop, and by extension pm25. It had originally branched off of master, which is why there were merge conflicts.

jacobwhall commented 1 year ago

It may be possible to download files concurrently, but too many and the connection pool dies. I've decided not to worry too much about it, since Box probably prefers it if we download things one-at-a-time anyways. Right now I'm running a test to make sure our connection isn't booted if we sequentially download the entire dataset. If this run succeeds, this PR will be ready for review!

jacobwhall commented 1 year ago

Even with sequential downloads, this is still having issues running with Prefect backend in HPC mode. Seems to work fine in all other modes. I'll revisit this shortly.