search

fatiando / rockhound

NOTICE: This library is no longer being developed. Use Ensaio instead (https://www.fatiando.org/ensaio). -- Download geophysical models/datasets and load them in Python
BSD 3-Clause "New" or "Revised" License
34 stars 15 forks source link

Optimize file access and out-of-core reading #20

Closed leouieda closed 5 years ago

leouieda commented 5 years ago

A lot of time and memory are wasted reading in whole large grids, like ETOPO1, even when only a small fraction is desired. The problem is that we store the compressed archives and need to unzip them in temporary folder every time. Since the files are temporary, we can't use xarray's lazy loading on the grids and must read them all into memory. This is not ideal.

This is a limitation of Pooch and need to think of a way of dealing with this. Ideas are welcome.

cc @santisoler

leouieda commented 5 years ago

fatiando/pooch#59 should allow us to do this.

santisoler commented 5 years ago

I agree! I think it's already solved.