search

creare-com / podpac

Pipeline for Observational Data Processing Analysis and Collaboration
https://podpac.org
Apache License 2.0
44 stars 6 forks source link

LoadFileMixin always opens the whole file #459

Open jmilloy opened 3 years ago

jmilloy commented 3 years ago

Description

The LoadFileMixin is used by rasterio, csv, and xarray nodes for both local and remote files. As written, the it always loads the whole file. The Mixin should be updated to only load the whole file in certain cases, and otherwise open the dataset directly from the source path.

Describe the solution you'd like

Distinguish between local and remote sources. Only read the file into memory if necessary for caching.

Additional Notes

The Rasterio node should be modified to not use the mixin. This will make the read_from_source unnecessary.

jmilloy commented 3 years ago

I updated the Rasterio node for 3.0.0, and I'll do the rest of this for 3.1.0

jmilloy commented 3 years ago

I'm going to try to do this for 8/13