What to use for test dataset?

[jreadey]

Thoughts:

• Should be large enough to provide realistic test case for things like time series extraction (say >100GB)
• Source data should be in NetCDF4/HDF5 (though specific drivers, say zarr, may need to transform to something else)
• Should comprise of 100's to 1000's of individual files (to capture potential overhead of file open/file close operations)
• Data that is already in S3/GCS is a plus

[jhamman]

@rabernat has suggested the GHRSST dataset.

I'll list a few other options:

• The GMET/Newman Meteorologic Ensemble. In zarr and netcdf format on GCP and on Cheyenne. ~5TB and 100 files (1 per ensemble member).
• Anything in NASA-NEX on s3. I've been using the LOCA dataset from NASA-NEX so I have a real science use case for this dataset. This dataset is about 10TB in size and is probably about 50k files.

[rabernat]

An example I really like is from Copernicus:

GLOBAL OCEAN GRIDDED L4 SEA SURFACE HEIGHTS AND DERIVED VARIABLES REPROCESSED (1993-ONGOING)

http://marine.copernicus.eu/services-portfolio/access-to-products/?option=com_csw&view=details&product_id=SEALEVEL_GLO_PHY_L4_REP_OBSERVATIONS_008_047

Here is a notebook showing how to push it to zarr on GCS. However, this is not already on S3, so that is a downside

https://nbviewer.jupyter.org/gist/rabernat/311faea2695370feb50ef51e6c0f0d22

[rabernat]

So :+1: to the LOCA dataset.

[kaipak]

@jreadey provided LOCA dataset on S3 so I think we can call this issue closed. Will reopen if anyone has objections.