search

d70-t / ipldstore

zarr backend store using IPLD datastructures
MIT License
6 stars 7 forks source link

adding illustrative example of loading a Zarr into Xarray from an IPLD mapper #6

Closed rbanick closed 2 years ago

rbanick commented 2 years ago

Adding an example use case we’ve pieced together that we believe is repeatable, easy, and of value to others.

d70-t commented 2 years ago

Cool, thanks for the PR! That's a very nice (and missing example). I'm just wondering where exactly to put it, and probably I'd put it just after "storing on IPFS". The latter parts only cover usage independent of IPFS and I like to have both, but with a clear visual distinction.

Then there's the CI, which has some critics about the examples... Can you have a look at those? You can run the tests locally using tox as well.

rbanick commented 2 years ago

Agreed on the placement of the example, it’s nice to have it immediately following the to_zarr. Feel free to swap in the example hash you used and the corresponding Xarray — I used one of mine just to show a more “real life” Xarray dataset.

On the tests — hmm. Will check.

rbanick commented 2 years ago

print(cid_obj) just gives back the hash — I need to call cid_obj directly to see the base32 representation. Oops! Tests now pass and the example has been moved under “Storing on IPFS”

d70-t commented 2 years ago

So I wasn't able to retrieve that referenced dataset using IPFS (my node couldn't find it). It would be very cool if that's possible with the demonstrated dataset, but the example is nice anyways.

As there is currently no IPFS gateway running in CI, I've marked the problematic lines as to be SKIPed. It's of course cheating, but...

If you are happy, I'll go merge it.

rbanick commented 2 years ago

So we are in the midst of pushing up and pulling down datasets to IPFS as we figure out how continuously updating Zarrs can work on our stack. In ~2 months I will absolutely have some "stable" Zarrs we can point to and would love to include them. Can we use this hash as a placeholder until then?

d70-t commented 2 years ago

Sure, that's fine!