Run comparison of different retrieval patterns from object store

[agstephens]

@mjones07 please describe the plan for this work here and tag @agstephens and @philipkershaw for comment.

[mjones07]

@philipkershaw @agstephens the plan for this is to:

Read from the 3 different file types

• netcdf4,
• s3netcdf,
• zarr

Have 3 (maybe more) read patterns

• map: [0,0,:,:] slice
• time series [:,0,0,0] slice
• "serial" where the whole file is read [i,j,:,:], or the optimal read patterns for the chunking strategy

This is timed and saved into a pandas json file with current headers:

• bytes_read
• config_file
• config_modified_at
• config_name
• rateMB
• read_pattern
• repeats
• run_at
• test
• total_time

Current issues are:

• caching:
  • reading such a "small" part of the files with the map and time series could hit caching problems. One of the reasons for doing the "serial" read is to get a rough comparison of what the optimal read rate is, to account for any overhead reading small parts of the file, and to try and avoid caching somewhere (although the file could fit into machine memory...)
  • potential solutions are
    1. multiple files (this is what I do for the EC testing)
    2. read a random map or time series, i.e. [x,y,:,:] where x and y are randomly generated
    3. manually clear the cache. This can be done as root to clear the cache on the VM the code is running on, but running as root and manually clearing the cache might be labour intensive, and would only clear the VM cache, not the filesystem or network cache. I don't think this is viable and/or useful
    4. A very large number of repeats
  • I think a combination of b and d are most viable

[agstephens]

@mjones07: thanks for providing details. This all looks good. I agree that combining (b) and (d) should increase the robustness of the tests.

Another potential issue that might interfere is:

• Sam/Ag/Neil doing significant read/writes to Caringo

I suspect that many repeats are the best option for reducing the noise that might be created by contention with others who are accessing Caringo.

[mjones07]

Testing code is done and map and timeseries reads are in progress for S3netCDF4, netCDF4 (byte range), and xarray/zarr

[philipkershaw]

@mjones07, from conversation with Pangeo people, there's a newer version of xarray/zarr that integrates an update to fspec that uses asyncio. This could change the results significantly. I don't know how far you've got with the tests and whether it would be possible to investigate this new release.