Elasticity: How to elastically scale resources to fit users' workloads?
Multi-user: How to protect customer's data (i.e. isolation) AND enable high resource utilization?
high resource utilization is desirable because it reduces the need for overprovisioning -> lower cost for customer.
Performance: How to make the approaches used fast?
Key Ideas:
Elasticity is achieved through separation of compute and (persistent) storage.
enables scaling compute and storage independently of each other.
compute refers to worker nodes; each consisting of CPU, local memory, ephemeral storage. (persistent) storage refers to object stores (e.g. S3, ADFS, GCS)
Perf:
Use local ephemeral storage as a write-through cache fronting the remote persistent storage, whenever possible
main usecase of local storage is still for intermediate query execution data
reasonably high cache hit rate (60-80%) due to skewed access distributions and temporal access patterns common in data warehouses, even if only a small amount of local storage capacity is utilized.
Use consistent hashing for both persistent data caching and task (query) scheduling. This increases data locality. Enabled by having a centralized control plane.
Use lazy consistent hashing to avoid reshuffling of intermediate data, at the cost of an extra remote persistent data read.
Isolation is trivially provided in the current design due to machines allocated exclusively for a customer (via Virtual Warehouse abstraction). If Snowflake were to increase resource utilization even more via sharing machines between customers, they will need to rethink the isolation guarantees and performance guarantees (e.g. when evicting data from local ephemeral storage).
Problem addressed:
Key Ideas:
Related: