search

inaos / iron-array

2 stars 0 forks source link

Proxy for ironArray arrays #605

Open FrancescAlted opened 2 years ago

FrancescAlted commented 2 years ago

For the Spark interface it would be nice to implement a proxy for accessing remote ironArray arrays so that we don't need a shared filesystem.

This can be achieved quite easily with a similar interface than the current proxy for Zarr objects. We just need to implement a network interface to our ironArray objects and ask for metainfo (shape, dtype...) and slices remotely.

As a proof of concept, we can use FastAPI (for the server) and requests (for the client).

martaiborra commented 2 years ago

This work is being done in https://github.com/inaos/iron-array-python/pull/189. This work is very preliminary and it doesn't let compute a reduction of a remote viewed array, because it doesn't copy the array before computing the reduction.