search

icenet-ai / icenet

The icenet library is a pip installable python package containing the commands and code you need to produce forecasts
MIT License
21 stars 7 forks source link

Better analytics/trial direct NetCDF loads #35

Open JimCircadian opened 2 years ago

JimCircadian commented 2 years ago

https://www.noahbrenowitz.com/post/loading_netcdfs/

JimCircadian commented 2 years ago

This ideally should be trialled with better benchmarks for performance from the existing pipeline runs, so consider implementing better analytics

JimCircadian commented 1 year ago

There's not a lot of call to do this without performant infrastructure to support it, which we are still in the process of acquiring, so unassigned until we can look at it sensibly.

bnubald commented 6 months ago

Data loading related:

https://docs.nvidia.com/deeplearning/dali/user-guide/docs/index.html

bnubald commented 5 months ago

Leaving here visibility, not planning to use atm.

A batch generator for Tensorflow/PyTorch from xarray DataArrays/datasets:

https://github.com/xarray-contrib/xbatcher

and, a relevant discussion thread:

https://discourse.pangeo.io/t/favorite-way-to-go-from-netcdf-xarray-to-torch-tf-jax-et-al/2663/2