search

zarr-developers / community

An open community with an interest in developing and using new technologies for tensor data storage.
19 stars 1 forks source link

Connecting with the Julia Community #17

Open jakirkham opened 6 years ago

jakirkham commented 6 years ago

Appears there are Julia users, who are discussing how to get a Zarr implementation going for Julia. Would be good to connect with them and see how we can help them get a Julia implementation going.

hearnsj commented 6 years ago

I will help with testing.

alimanfoo commented 6 years ago

Thank you @hearnsj. FWIW I was thinking it might be useful to have a go at creating a notebook using a Julia kernel that tries to adapt the code examples in the zarr tutorial to Julia, with calls to zarr via pycall. This should throw up any compatibility or usability issues. Another useful activity would be to benchmark some simple read and write operations in both Python and in Julia via pycall, to get a sense of if there are any major performance issues.

meggart commented 6 years ago

This sounds great and I would be highly interested and would like to support this initiative. I have started a very rudimentary wrapper using PyCall which allows me to read from Zarr arrays as Julia arrays, https://github.com/meggart/Zarr.jl . This was basically just for my own reading benchmarks to decide if the format would work for me, so sorry for missing tests/docs. Basically you can just run zopen(path) on any zarr group or variable and then use Julia indexing (supports ranges and colons) to read slices from the dataset.

What is really missing because I did not need it is the creation and writing of datasets. I will see if I can add a few functions soon, so that one can repeat the tutorial.

alimanfoo commented 6 years ago

Fabian, very cool! Look forward to hearing more.

On Fri, 10 Aug 2018, 12:24 Fabian Gans, notifications@github.com wrote:

This sounds great and I would be highly interested and would like to support this initiative. I have started a very rudimentary wrapper using PyCall which allows me to read from Zarr arrays as Julia arrays, https://github.com/meggart/Zarr.jl . This was basically just for my own reading benchmarks to decide if the format would work for me, so sorry for missing tests/docs. Basically you can just run zopen(path) on any zarr group or variable and then use Julia indexing (supports ranges and colons) to read slices from the dataset.

What is really missing because I did not need it is the creation and writing of datasets. I will see if I can add a few functions soon, so that one can repeat the tutorial.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/zarr-developers/zarr/issues/284#issuecomment-412054853, or mute the thread https://github.com/notifications/unsubscribe-auth/AAq8QraTS-Ig26WPSdhRDh-vwGwI5KAbks5uPW1XgaJpZM4Vo9eR .

jakirkham commented 6 years ago

Indeed very cool. Please let us know if there is any way we can help.

rabernat commented 5 years ago

Should also point out that @meggart has a newer, julia-native implementation of zarr: https://github.com/meggart/ZarrNative.jl

rabernat commented 5 years ago

I'm pinging this issue to check in on the status of zarr + julia. We now have julia kernels working in pangeo environments (see e.g. https://github.com/pangeo-data/pangeo_ocean_examples). The next step for interoperability is to be able to load data from zarr stores in Google Cloud Storage.

Could someone (probably @meggart) give an update on where the zarr julia packages stand? I checked the repos and didn't see much recent activity. Are there any roadblocks we can help overcome?

visr commented 5 years ago

Regarding ZarrNative.jl, there is a basis there, but it still needs more work. I think the main thing holding it back is simply developer time. Thanks for the offer for help though, I'm sure we will be needing it.

There is no Google Cloud Storage support yet, it's probably best to get to a more solid base, testing with local stores first.

alimanfoo commented 5 years ago

@meggart, @visr, I just wanted to let you know that we've started up a regular conference call as a way of building connections within the zarr and n5 community, it's currently fairly small but includes folks from groups implementing libraries in various languages as well as folks representing user communities. I know bandwidth is short but if you had any time to attend you would be very welcome. Details of the call are over on #315.

meggart commented 5 years ago

Thanks for the info, I have already followed that thread and I think I will join one of the next meetings.

meggart commented 5 years ago

Just to let you know, we have released a first version of Zarr.jl and registered them in the official package channels. It does not yet implement the full Zarr spec, many compressors and backends are missing, but this may come over time as needed. The main focus for now will be on extending documentation, in particular finish translating the Tutorial.

alimanfoo commented 5 years ago

Fantastic news, and perfect timing!

On Tue, 9 Jul 2019, 09:55 Fabian Gans, notifications@github.com wrote:

Just to let you know, we have released a first version of Zarr.jl https://github.com/meggart/Zarr.jl and registered them in the official package channels. It does not yet implement the full Zarr spec, many compressors and backends are missing, but this may come over time as needed. The main focus for now will be on extending documentation, in particular finish translating the Tutorial https://meggart.github.io/Zarr.jl/latest/tutorial/.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/zarr-developers/community/issues/17?email_source=notifications&email_token=AAFLYQTYEV2ZGZRAWU32BF3P6RHBLA5CNFSM4H5MBZ52YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZPTAWY#issuecomment-509554779, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFLYQTCBYEXM5ASUQRYQMTP6RHBLANCNFSM4H5MBZ5Q .