search

allenai / rslearn

A tool for developing remote sensing datasets and models.
Apache License 2.0
5 stars 0 forks source link

Don't support storing bands in tile store in different combinations from source data #7

Closed favyen2 closed 8 months ago

favyen2 commented 8 months ago

Currently we read in all rasters associated with an item before warping them because the user can specify that tiles should be stored in combinations that span multiple raster files.

But this is memory-intensive for a data source like Sentinel-2 with many bands and large raw GeoTIFF files.

Instead I think we should drop support for this and instead only allow users to specify a subset of the bands available in the data source.

Then we can do the warping for each band separately, reducing memory usage for each process.

favyen2 commented 8 months ago

This was completed in 49fecc8fa6b005ccea73804a7f89915628c6dbe9