Implement dataset & workload based on presentation from meeting on Thu 28th Sept.

[JackKelly]

Data:

• Total dataset size = 100k traces x 70k samples = 14 GBytes
• Data type = int16.
• Zarr chunks:
  • Size = 5k traces x 1k samples
  • 10 MBytes per chunk (uncompressed. Or ~7 MB using LZ4 clevel 3)
  • 1,400 chunks
• Uncompressed.

Workload psuedocode:

# Profile 1
SAMPLES_SLICE = slice(0, 70_000)
TRACES_CHUNK_SIZE = 5000
for i in range(20):
    traces_slice = slice(i * TRACES_CHUNK_SIZE, (i+1) * TRACES_CHUNK_SIZE)
    batch = array.get_orthogonal_selection((traces_slice, SAMPLES_SLICE))