search

FAIRiCUBE / FAIRiCUBE-Hub-issue-tracker

FAIRiCUBE HUB issue tracker
Creative Commons Zero v1.0 Universal
0 stars 1 forks source link

Python packages to bridge between standard EO types such as zarr and xarray and rasdaman WCPS requests? #30

Closed robknapen closed 10 months ago

robknapen commented 1 year ago

Are there any operational/maintained Python packages available that bridge between the rasdaman WCPS queries and the standard types and classes used in EO, such as xarray and zarr? Or any known documentation / best practices?

Ultimately I would need something that works with PyTorch Datasets and DataLoaders, or TorchGeo, to get the data into machine learning format.

pebau commented 1 year ago

no, I am not aware of anything out there, and we do not provide any packages in that direction either.

robknapen commented 1 year ago

Ok, will have to figure out something then.

robknapen commented 1 year ago

My workaround will probably (have to) be to (pre)download (using WCS or WCPS requests) the required data from the data cubes as (sliced) geotiff or netcdf files, which I can then import into xarrays or acces as a TorchGeo Dataset for machine learning as needed.

pebau commented 1 year ago

not necessary - you get the data results as generic python arrays, see the Jupyter notebooks we have provided. That can be numpy, but I would expect that they can be transformed into xarray etc. easily.

robknapen commented 1 year ago

Thanks, I will have a look at that example Notebook.

jetschny commented 10 months ago

is this issue closed maybe? @robknapen ?

robknapen commented 10 months ago

The question has been answered indeed. When we get to it in our use case I will write the code that requests data from rasdaman via WCPS and then gets it into PyTorch for model training. For performance reasons (to keep the GPU busy) perhaps it has to be a download script first to get all training/test data (tiles) and write it to local files.

I will close the issue with this.