search

NCAR / xtar

Reading netCDF tar archives with xarray/zarr
MIT License
0 stars 2 forks source link

dummy netcdf test data #5

Open tinaok opened 4 years ago

tinaok commented 4 years ago

hello @kmpaul @andersy005

I was wondering if it would be possible to have dummy netcdf test data, like the one used on Pangeo-data/benchmarking;

  1. With flex input size for total lattitude, longitude and time, and time interval (for example if we say lat=360, lon=180, time=120 hourly, it will create lat: -180, 180 divided in 360 data points, lon: -90 , 90 is divided in 90 points, time: makes hourly data, starts year 2000 Jan, 1st 0:00 and ends on 5th
  2. Then separated the file in flex chunk sized netcdf file; in lat_chunk, lon_chunk and time_chunk. for example if we say 10,10,10, which make 1000 netcdf file. 0.0.0.nc: corresponds to lat: -180 to -145 degree, lon: -90, -73 degree time: 2000 Jan 1st 0:00 to 9:00

This way we can bench what is the best tar, and netcdf chunked format to be stored at the mpi computation on oceanophysics or atmospheric computation; to perform Pangeo post-analysis computation later on.

kmpaul commented 4 years ago

I'm in favor of this, but with our limitations on Cheyenne with ratarmount, I don't think we can do this and test. I'd be open to a PR from someone else, though.