search

ubarsc / kealib

KEALib provides an implementation of the GDAL data model. The format supports raster attribute tables, image pyramids, meta-data and in-built statistics while also handling very large files and compression throughout.
http://kealib.org/
MIT License
12 stars 7 forks source link

add support for /vsi datasets in GDAL plugin #44

Closed gillins closed 10 months ago

gillins commented 10 months ago

@t-hackwood @petescarth this adds the mysterious hdf5vfl.h from GDAL (which weirdly I'm listed as one of the authors, must have been a long time ago). Conveniently this file was already under the same license as libkea.

Anyway, with this addition (plus a few tweaks) when building KEA as a GDAL plugin (ie against Ubuntu GDAL in a container) you now get access to /vsi filesystems. Which might be handy for those of us with large KEA files on S3....

gillins commented 10 months ago

Interesting: https://www.youtube.com/watch?v=bDH59YTXpkc. Might have to look into Paged Aggregation. @petebunting should we start tweaking some of the default chunk sizes etc to make the files run better on S3?

petebunting commented 10 months ago

@gillins, yeah, it looks like it would be a good idea to tweak the chunk sizes. That should also be for the RAT as well as the image data?

From a quick scan through the RFC for page buffering, it does not sound like it would be a negative thing when using KEA files locally to have paging enabled and would also help out people using them on a HPC system was well...

https://docs.hdfgroup.org/archive/support/HDF5/docNewFeatures/NewFeaturesPageBuffering.html

gillins commented 10 months ago

@petebunting yeah I guess RAT also. I didn't quite understand that bit about putting all the metadata in one chunk, but sounds like might be worth looking at. I'm hoping to put out a 1.5.3 shortly and maybe we can look at this for 1.6.0?

gillins commented 9 months ago

@petebunting have bumped the version to 1.6.0 in https://github.com/ubarsc/kealib/pull/50 so we can fiddle with these low level things...