JackKelly
commented
5 months ago More info about high-speed IO from Google Cloud Storage:
For cloud buckets, re-implement the cloud storage API using io_uring. Although, cloud storage buckets can be severely limited in terms of read IOPs. For example, the Google Cloud Storage docs state that "buckets have an initial IO capacity of.. Approximately 5,000 object read requests per second, which includes listing objects, reading object data, and reading object metadata." Although the documentation continues on to say "A longer randomized prefix provides more effective auto-scaling when ramping to very high read and write rates. For example, a 1-character prefix using a random hex value provides effective auto-scaling from the initial 5,000/1,000 reads/writes per second up to roughly 80,000/16,000 reads/writes per second, because the prefix has 16 potential values." Which suggests that a longer random prefix could increase read IOPs indefinitely. The "quotas" page says the "Maximum rate of object reads in a bucket" is "unlimited". Bandwidth is limited to 200 Gbps (about 20 GB/s). AWS S3 has similar behaviour (also see this Stack Overflow question). Although this does raise a problem with Zarr sharding: you might actually want to randomly take chunks which are nearby in n-dim space, and spread them randomly across many shards, where each shard has a random prefix in its filename (although that forces you to make lots of requests. It may be better to keep nearby chunks close together, and coalesce reads).
Some notes (in no particular order) about speeding up reading many chunks of data from cloud storage.
General info about cloud storage buckets
In general, cloud storage buckets are highly distributed storage systems. They can be capable of very high bandwidth. But - crucially - they also have very high latencies of 100-200 milliseconds for first-byte-out.
Contrast this with read latencies of locally attached storage: HDD: ~5 ms; SSD: ~80 µs; DDR5 RAM: 90 ns. In other words: cloud storage latencies are two orders of magnitude higher than a local HDD; and cloud storage latencies are four orders of magnitude higher than a local SSD!
How many IOPS can we get from cloud storage to a single machine?
AWS docs say:
(Also see this Stack Overflow discusion)
So... it sounds like we could achieve arbitrarily high IOPS by using lots of prefixes.
How fast can a single VM go? A 100 Gbps NIC could submit 20 million GET requests per second (assuming each GET request is 500 bytes). And a 100 Gbps NIC could receive 2 million 4 kB chunks per second (assuming a 20% overhead, so each request is 5 kB); or 2,000 x 4 MB chunks per second.
To hit 2 million IOPS, you'd need to evenly distribute your reads across 364 prefixes.
2 million requests per second per machine is a lot! It wasn't that long ago that it was considered very impressive if a single HTTP server could handle 10,000 RPS. I'm not even sure if 2 million RPS is actually possible! Here's a benchmark of a proxy server handling 2 million RPS, on a single machine.
I'd assume that, in order for the operating system to handle anything close to 2 million requests per second, we'd have to submit our GET requests using io_uring.
Some "tricks" that LSIO could use to improve read bandwidth from cloud storage buckets to a single VM:
Rangeheader, orpartNumber), and download the parts in parallel (already planned)