search

Unidata / netcdf-c

Official GitHub repository for netCDF-C libraries and utilities.
BSD 3-Clause "New" or "Revised" License
520 stars 262 forks source link

ZARR implementation status #1677

Open durack1 opened 4 years ago

durack1 commented 4 years ago

Hi folks, I read https://www.unidata.ucar.edu/blogs/news/entry/netcdf-and-native-cloud-storage with interest and am curious about how this project is progressing.

Would it be possible to create a new project for this task, so that it's more transparent as to how work is progressing (and where help is needed)?

WardF commented 4 years ago

Having a separate project might be problematic, as it is part of the C library and not a stand-alone project/executable/library. I'll create a tag, however, that we can use to filter on as a stop-gap; I'm open to being pursuaded with counter arguments or other solutions we could adopt. In terms of the progress and such, I'll tag @DennisHeimbigner in explicitly.

DennisHeimbigner commented 4 years ago

I am close to releasing an experimental version so I am not sure that a separate project is warranted. The "caveats" are as follows:

dopplershift commented 4 years ago

@WardF I think @durack1 is referring to a GitHub "project management" project that lists all the relevant issues and shows their status.

lesserwhirls commented 4 years ago

I think by project, @durack1 means a github project for tracking issues/milestones (much like the "Thread Safety" project at https://github.com/Unidata/netcdf-c/projects/6), and not a separate github repository. As testing and feedback are provided, tracking those new issues as a project might result in having extra fingers on keyboards to help tackle the issues, as well as provide a roadmap on where things stand for those actively watching.

WardF commented 4 years ago

Thank you both @dopplershift and @lesserwhirls I see that now. We can probably do that, I will discuss and coordinate with @DennisHeimbigner tomorrow.

DennisHeimbigner commented 4 years ago

Sure we can do that. My above message should perhaps be a first entry for that project.

durack1 commented 4 years ago

@dopplershift @lesserwhirls thanks for expanding on my comment and clarifying, it is exactly what I meant. I understand the code will be adapted, but the discussion in https://github.com/Unidata/netcdf-c/issues/1677#issuecomment-602886363 was exactly my motivation. You will have many eyes starting to watch this as it evolves

DennisHeimbigner commented 4 years ago

I now have a S3 storage driver that compile and builds. Adding a c++ file to the build is a real poison pill because it requires that the c++ standard libraries be loaded (-lstdc++).

ethanrd commented 4 years ago

Have you looked at the AWS SDK for C? I stumbled across the aws-c-common just yesterday. Looks like there are other components (e.g., aws-c-io and aws-c-compression) but I haven't found a full list and not sure of the status of the overall project.

DennisHeimbigner commented 4 years ago

Yes, it is because I am using th aws s3 sdk (which is c++ onky) that I am having the poison pill problem.

dopplershift commented 4 years ago

Is there actually a problem with defining CC=g++ ? What about manually linking in libstdc++?

WardF commented 4 years ago

Looking at the project, Ethan listed (that appear to be C not C++), could that work for us? At a glance it appears to be active and cross platform. I’m on mobile at the moment but will look closer.

DennisHeimbigner commented 4 years ago

I am looking at it now. One question is: do we trust this organization to be around for a while?

DennisHeimbigner commented 4 years ago

I do not see any evidence that this supports S3. Am I missing something?

ethanrd commented 4 years ago

I didn't see S3 support either but wasn't sure if I had found all the aws-c-* projects.

Also, given there appears to be at least some level of Amazon connection (the AWSLabs GitHub organization is "verified to control the amazon.com domain"), it seemed like it might be worth some digging.