search

ubarsc / kealib

KEALib provides an implementation of the GDAL data model. The format supports raster attribute tables, image pyramids, meta-data and in-built statistics while also handling very large files and compression throughout.
http://kealib.org/
MIT License
12 stars 7 forks source link

up chunk sizes #62

Closed gillins closed 6 months ago

gillins commented 8 months ago

@petebunting is this what you understood from that presentation?

I haven't done any testing with it yet....

cc: @neilflood

petebunting commented 8 months ago

Hi @gillins

Yes, I think this is was I understood from the presentation.

Will be interesting to see what difference it makes.

gillins commented 6 months ago

@petebunting just to close the loop on this. Turns out setting paged aggregation really increases the file size:

Size of file
Current kealib 736m
These Changes 715m
This plus pageSize=2048 952m
This plus pageSize=4096 1.7G
This plus pageSize=8192 3.9G

So I think it's worth changing the chunksize (all that this PR now does) but let's not bother with the paged aggregation. It increases the filesize by quite a lot (probably because we have lots of small datasets in a KEA file - ie each block) plus breaks the API.

With just changing the chunk size we do get smaller files.

What do you think?

petebunting commented 6 months ago

Hi @gillins

Yes, that sounds like a good way forward. I don't think we want file sizes growing that significantly.