Guide on how to add Geo(Parquet) and Zarr (BLOSC) support to OTBTF

[daspk04]

Hi @remicres !

I was testing with the latest docker. As it is GDAL 3.7.1, and OTB 9.0, which seems interesting! I tested a bit with 'ZARR' and it seems that it OTB can read it.

1) Compression method doesn't support BLOSC compressor

gdalinfo --format 'ZARR'

2) Similarly Arrow format is not supported.

ogrinfo --format 'ARROW'

Do you think it can added (we copy libraries from this docker image to otbtf image) or somehow recompile GDAL with

• libblosc1 ? GDAL Ubuntu Docker
• arrow ? GDAL Ubuntu Docker

[remicres]

Hi @Pratyush1991 ,

We could add these apt packages in the docker build. Then, when building GDAL from the OTB SuperBuild (the cmake script that builds all dependencies), we can provide GDAL build options. So if that is required, I guess that we could do the necessary to include the support of ZARR and ARROW.

I add this in otbtf roadmap! Will keep you updated

[remicres]

I have created a MR here it is a work in progress

[daspk04]

I have created a MR here it is a work in progress

Thank you @remicres !

[remicres]

For now I am failing to make GDAL see arrow and blosc, maybe something on the otb/superbuild side... or related to the GDAL version used in the otb branch (3.7.1), do you know if these supports have been added recently in GDAL?

[daspk04]

For now I am failing to make GDAL see arrow and blosc, maybe something on the otb/superbuild side... or related to the GDAL version used in the otb branch (3.7.1), do you know if these supports have been added recently in GDAL?

Thank you @remicres for the question. As I checked on the GDAL documentation ARROW support was added from 3.5 onwards and ZARR from 3.4 onwards.

Arrow:

Regarding building it from the source., I checked the the coda recipe for gdal 3.7, I don't completely understand the process but as much as I can understand.

• Seems like first GDAL is built without it
• Then it seems like it is again rebuilt using Arrow support
• Finally python binding

ZARR:

• Possibly needs to be build from source
• Then during GDAL build we just configure
• Some reference from conda-forge

Regarding OTB may be above process can be adapted into the Superbuild, although I might be wrong. Seems like there is a branch for GDAL 3.8 🤔

On a side note I cannot access the merge request may be some issue with sever, I was able to access it 3 weeks earlier thou ! 😄

[remicres]

Thanks for this insight, I realise that I did not properly configure gdal before build. I'll try that as soon as our CI is working again...

[daspk04]

Hi @remicres !

I was able to build OTB 9.0.0 with Geo(Parquet) and Zarr (Blosc). I used Nix to build OTB from source, basically it's a native build with all the process similar to SuperBuild. Nix already has Gdal with Arrow and Blosc enabled by default so it was easy to add them to build process. I built ITK (4.13) and Shark from source as these were not available in Nix packages.

I have added OTB nix package to my repo with readme and how to use it. Currently this doesn't build the remote modules. Except for OTBTF (as it requires TF), other's I think I can possibly add in coming weeks. I also have added some examples on how to configure python environment, incase one needs to configure python environment with pyotb + gdal + otb + rasterio . Your suggestion or comments on this pull request will also help. Incase you have any questions please feel free to open an issue.

[remicres]

Hi @daspk04 , that is great news. We will definitely take a look in your nix package after the hollidays. I guess that you have already done some heavy work building otb and itk.