As the roadmap stands now it strongly emphasizes the download -> analyze workflow.
Would it be possible to spell out more distinctly the use case of streaming access to the data?
For context I have been part of the Pangeo / ESGF Cloud Data Working Group and born out of that am currently running the ongoing ingestion of CMIP6 datasets into public cloud storage repo which is currently funded by my involvement with LEAP.
This effort enables users to stream CMIP6 data from any computation environment on the public internet, and we have had a tremendous amount of positive responses from users, crucially from early career researchers and students and researchers from the global south.
I very much believe this data access pattern presents a tremendous opportunity to both extend and diversify the users of CMIP data and would like it to be represented in a more prominent way in this roadmap if that is possible.
My suggestion is to not 'inline' the Download/User Interaction points, but to generalize the User Interaction into two subpoints:
Download/Transfer Data --> Analyze
Stream Data on demand (e.g. via xarray/dask lazy loading on top of zarr or virtualized zarr)
I am very happy to provide more details on the lessons learned, the user experience, or any other questions that might arise.
scollis
commented
5 months ago
As the roadmap stands now it strongly emphasizes the download -> analyze workflow.
Would it be possible to spell out more distinctly the use case of streaming access to the data?
For context I have been part of the Pangeo / ESGF Cloud Data Working Group and born out of that am currently running the ongoing ingestion of CMIP6 datasets into public cloud storage repo which is currently funded by my involvement with LEAP.
This effort enables users to stream CMIP6 data from any computation environment on the public internet, and we have had a tremendous amount of positive responses from users, crucially from early career researchers and students and researchers from the global south.
I very much believe this data access pattern presents a tremendous opportunity to both extend and diversify the users of CMIP data and would like it to be represented in a more prominent way in this roadmap if that is possible.
My suggestion is to not 'inline' the Download/User Interaction points, but to generalize the User Interaction into two subpoints:
I am very happy to provide more details on the lessons learned, the user experience, or any other questions that might arise.
cc @RobertPincus @scollis