Applied-GeoSolutions / gips

Geospatial Image Processing System
GNU General Public License v3.0
17 stars 5 forks source link

performance enhancements for Sentinel-2 L1CGS #498

Closed ircwaves closed 5 years ago

ircwaves commented 5 years ago

Use a template for the base ref-toa (and 'allbands' in the fmask_geoimage method) VRT, instead of relying on gdalbuldvrt for creation. This is trivial and should shave ~100 seconds off the top. The VRT generated is an exceedingly simple one (an example), and the values to fill out the template could be snatched by inspection of just the first band of the image.

ircwaves commented 5 years ago

@sullamenashe -- also did a test, and narrowed the slowness further: jp2k over /vsicurl. Need to learn if there are strategies to make that more efficient, or to bail and use a local temp dir to store the bands.

ircwaves commented 5 years ago

remote reads are problematic, so for now we go back to a local temp-dir storing the bands, and this issue reverts back to templatizing the VRTs.

ircwaves commented 5 years ago

As of bfcb30c, the Sentinel-2 driver now using locally cached jp2 images, the vrt build takes ~0.05 seconds. Notice that the fmask VRT re-uses the ref-toa jp2s (if they exists), and hence the reported download time is < 0.01 seconds (see attached runlog).

At this point, the time spent breaks down as:

portion duration
gips init 0:13
download + VRT + 6s 1:41
fmask 6:41
ref 2:09
total 10:30