search

TileDB-Inc / TileDB-CF-Py

TileDB interface with awareness of the CF metadata conventions
https://tiledb-inc.github.io/TileDB-CF-Py/
MIT License
19 stars 3 forks source link

Chunking option for converter from netCDF to tileDB #92

Closed eggio closed 3 years ago

eggio commented 3 years ago

Adding the option to do the converting operation chunk-wise to avoid a MemoryError. Something like creating the tdb array and then doing the writing of the data piece-wise.

jp-dark commented 3 years ago

Hi @eggio, this feature has come up a few times now,and will be implemented soon. The current design plan is to add a parameter directly into the converter (just like the tile size, compression filters, etc.) for the chunk size and/or maximum TileDB fragment size when copying NetCDF data.

jp-dark commented 3 years ago

Hi @eggio this feature is being implemented in PR #96 and will be included in the next release.

jp-dark commented 3 years ago

This is now available in release 0.5.0. To set the chunksize use the max_fragment_size parameter in the NetCDF4DomainCreator. Currently, you will need to set it for each TileDB array in the group you are creating.

See examples/netcdf_engine/netcdf-to-tiledb-set-max-fragment-size.ipynb for an example on setting the chunk size.

I am closing this now, but if you have any issues please feel free to reopen and/or comment here.