gaffer01
commented
1 year ago Performance of t3.xlarge-based compactions (t3.xlarge has 4vcpu, 16GiB, cost is $0.1888 in London region - note 4 tasks can run on one of these at a time):
- Compaction 174,000 records per second per task = 4 * 3,300,000,000 records per $ = 13,200,000,000 per $
- Splitting compaction 232,000 records per second = 4 * 4,400,000,000 records per $ = 17,600,000,000 per $
Performance of m6i.xlarge-based compactions (m6i.xlarge has 4vcpu, 16Gib, cost is $0.222 in London region - note 4 tasks can run on one of these at a time):
- Compaction 211,500 records per second = 4 * 3,400,000,000 records per $ = 13,600,000,000 per $
- Splitting compaction 290,000 records per second = 4 * 4,700,000,000 records per $ = 18,800,000,000 per $
For comparison, on Fargate the rates are (each Fargate task has 1vcpu and 4GiB and costs 0.04656 + 4 * 0.00511 = $0.067):
- Compaction 249,000 records per second = 13,000,000,000 per $
- Splitting compaction 357,000 records per second = 19,000,000,000 per $
This PR https://github.com/gchq/sleeper/pull/316 adds the ability to run compactions on EC2 instances. The current default EC2 type is T3 - they're "designed for applications with moderate CPU usage that experience temporary spikes in use" so probably aren't ideal for compactions. Experiment to find a better (performance and performance per $) type.