search

Cloud-Drift / clouddrift

CloudDrift accelerates the use of Lagrangian data for atmospheric, oceanic, and climate sciences.
https://clouddrift.org/
MIT License
33 stars 8 forks source link

Research memory efficient ragged array netCDF file generation #367

Open kevinsantana11 opened 5 months ago

kevinsantana11 commented 5 months ago

When aggregating the gdp1h trajectories into one file the RAM memory utilized during this process requires a machine with greater capacity than 16GB of RAM.

Research needs to be done in regards to understanding whether there's memory efficient method of generating the aggregated trajectories.

milancurcic commented 5 months ago

May be a solution: https://stackoverflow.com/questions/73769633/initialize-larger-than-memory-xarray-dataset

selipot commented 5 months ago

That's exactly what I was thinking about!