Estimate Hardware requirements, CPU, RAM and NVMe disks

[a-saksena]

Known questions:

• [x] Main disk requirements, RAID/non-RAID options, maybe ZFS or btrfs?
  • [x] How much disk space gets us how many years storage?
• [x] What is the hash rate needed?
  • [x] Can we use GPU or NPU resources?
  • [x] What hardware accelerators are available?
• [x] What is the networking required, and what are the options?
• [x] How much RAM do we require, what are the primary drivers of RAM usage?
• [x] How much parallelism do we require, how many cores?
• [x] Can we offload some functions to separate machines, possibly ephemeral?
• [x] Can we use "typical" cost reduction approaches?
  • [x] Can we reduce cost with several small/cheap machines?
  • [x] Can we lower cost with networked storage for the large-volume historical data?
  • [x] Can we use separate "worker" processes (K8S containers) to handle large/long processes like reconnect or state snapshots?
  • [x] Should we be using ZFS, btrfs, LVM, etc...?
  • [x] Do we need RAID (5/6) or is multiple-machine redundancy enough/better?
• [x] What uptime requirements do we have?
• [ ] What is the impact of 1/3/6/12 simultaneous block node failures?
• [ ] What level of "spare" capacity do we need to avoid failure amplification?
• [x] What is the disk read/write rate required for "recent" blocks, live state, state snapshots, "old" blocks, etc...?
• [x] What data volume is required for "fast" data versus "slow" data?
• [x] Do we need/want any level of CXL 2.0 support?

Notes:

• Record stream can be estimated by listing the files/folders/subfolders at https://console.cloud.google.com/storage/browser/hedera-mainnet-streams/recordstreams?project=mainnet-exports&pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))
  • This must be done with care, costs climb very fast in requester-pays buckets.
  • Command line is likely to be the most efficient option
  • Suggested process
    1. First part compute existing record stream size when deduplicated and compressed.
       1. Run some gsutil "ls -l" commands on a couple hosts that have been around a long time LG,IBM, and maybe swirlds. Pipe to text file. can do on GCP VM or laptop
       2. Take those text files and match them up to get a size for every record file. Either in Java or SED/BASH what every is quickest
       3. For that we should be able to compute what the record stream deduplicated up to today takes.
       4. We need one copy of every record and sidecar file and 32 copies of signature files.
       5. We could look at the compression ratio we get on record files and apply that ratio to all uncompress early files
    2. Then next try and extrapolate for block streams
       1. Look at some sample data from latest services running with record stream and block stream turned on. Alex K can get you data if you don't have any. Estimate a what % larger block streams are than record streams.
       2. Plot a chat showing record stream growth tragectory and then multiply by % bigger block streams are.
       3. Extrapolate out 1yr, 2yr, 5 and 10
       4. Maybe look at 10K TPS size and see what more worst case would look like
       5. Might be fun to add event stream + record stream size to show that block streams are larger than record streams alone but smaller than the two we have today

[jsync-swirlds]

Some responses:

• What is the hash rate, can we use GPU or NPU resources, and what accelerators are available?
  • GPU/NPU resources can be used, but it would require custom native code, and GPU, in particular, requires processing data in large batches, so it's possible, but not easy.
  • Custom accelerators are available, but only if we specify custom servers and the added cost is justifiable. This does not currently appear to be the case.
    • It may be worthwhile to research this option for "execution" nodes in the future.
• How much RAM do we require, what are the primary drivers of RAM usage?
  • The primary driver is the MerkleDB usage for state.
  • Total RAM requires is not yet determined, but we are estimating a value similar to current consensus nodes.
• How much parallelism do we require, how many cores?
  • We have very good parallelism opportunities, and could fully utilize as many as 64 cores, possibly more.
• Can we offload some functions to separate machines, possibly ephemeral?
  • TBD, this will be addressed as a future effort.
• Can we use "typical" cost reduction approaches?
  • Can we reduce cost with several small/cheap machines?
    • Not for "state managing" block nodes, which require the MerkleDB, at least not currently.
  • Can we lower cost with networked storage for the large-volume historical data?
    • Not initially, but perhaps in future work.
  • Can we use separate "worker" processes (K8S containers) to handle large/long processes like reconnect or state snapshots?
    • Possibly, this is an item for future work.
  • Do we need RAID (5/6) or is multiple-machine redundancy enough/better?
    • RAID increases availability in a single error scenario, but we already have much higher availability via multiple nodes. The tradeoff cost of long rebuild times, read-only array during rebuild, and very high CPU impact of rebuild make this unlikely to improve real-world availability.
• Do we need/want any level of CXL 2.0 support?
  • Not at this time.

[jsync-swirlds]

More answers:

• What is the hash rate needed?
  • On the order of 1.5 billion/sec; dominated by live state rehash.
  • Without live state rehash (if that's even possible), roughly 400 million/sec.
• What is the networking required, and what are the options?
  • We might be able to make do with 1Gb/s, but 10Gb/s (RJ45) hardware is recommended, regardless of available ingress/egress due to obsolescence of 1Gb/s hardware in the data center.
• Can we use "typical" cost reduction approaches?
  • Can we use separate "worker" processes (K8S containers) to handle large/long processes like reconnect or state snapshots?
    • Perhaps, but this will need to be addressed in future phases.
  • Should we be using ZFS, btrfs, LVM, etc...?
    • Yes. Btrfs or ZFS are recommended. If RAM is limited, btrfs is more memory efficient, particularly in a remote JBOD setup (disk chassis connected via high bandwidth SAS over Ethernet).
  • Do we need RAID (5/6) or is multiple-machine redundancy enough/better?
    • RAID is not recommended. It does not improve practical availability due to performance impacts when degraded/rebuilding, and without additional hardware support, the overall performance cost is not sustainable above 200TB/array.
• What uptime requirements do we have?
  • 5 nines or better is desired.
• What data volume is required for "fast" data versus "slow" data?
  • A relatively small amount of "fast" data, perhaps 8-12 TB, is needed for live state and "recent" blocks and snapshots. The rest is archival and can be less performant.

[jsync-swirlds]

Here is a recommended generalized (almost no specifics) hardware specification Consolidated source-agnostic specification. Cost estimate is total and represents retail pricing.

THIS IS NOT FINAL and still requires review and approval.

Total Cost Estimate: ~$28,100 (retail pricing)

---

CPU (Est. $3,000)

• Any Intel 4th Gen or later Xeon, or AMD 9004/9005 series with at least 32 cores
• Recommended: 32 core, 3 GHz, 280-320W, Xeon 4th Gen / EPYC 9004 series or later
  • Examples
    • Intel 6430/6530, AMD 9354/9355, or better

    • Intel 6952P (6th Gen Xeon, 72 "P" cores, the smallest all-P-core configuration)

      Core Networking (Est. $100 or $0 if included on Motherboard)

• 2 X 10Gb/s Ethernet (SFP+ or RJ45) or better (25Gb/s ports may be less expensive)

• Actual network speed may vary, but hardware should match specified speed.

  O/S Storage (Est. <$200)

• Recommended: 2 X 1 TB (or larger) NVMe drives, RAID 1
  • M.2, PCIe 4 x 3.0 or better
• Required: 2500 MB/s read, >100,000 IOPS write, 500,000 IOPS read
  • Example

    • Samsung 980 PRO or better

      "Fast" Storage (Est. $1,800)

• Recommended: 2 X 3.8 TB NVMe drives, RAID 0
  • PCIe 4 x 3.0 or better
• Required: 3000 MB/s write (per drive), >250,000 IOPS write, 800,000 IOPS read

  • Durability: Minimum 0.1 TDW/day

    Bulk Storage (Est. $11,000)

• Recommended: 450 TB (usable), 30 X 16 TB SAS 12Gb/s drives, JBOD
  • Minimum internal 16-lane SAS connect, or external 4 X 25Gb/s SoE

  • Configuration Note: At least 30 drives recommended to ensure adequate throughput and IOPS with magnetic storage

    Chassis, Power, Motherboard, etc. (Est. $12,000)

• Chassis: Datacenter-class with redundant power supplies
• Motherboard: Datacenter-class, compatible with CPU
  • Memory: At least 384 GB RAM, with all channels populated if possible
  • PCIe: PCIe 3.0 or later, at least 64 lanes (PCIe 4.0 recommended)

• Communication: Suitable SAS and network interfaces from a major vendor

  Caveats:

  1. We are not yet certain about the data required for state snapshots. This may significantly impact storage required. If snapshots are in the lower predicted range, a 450TB configuration is acceptable. If snapshots are in the upper predicted range, then a 900TB configuration is required.
  2. Prices here represent a custom build option. A pre-built configuration that is close to this specification may be sufficient.
  3. Price estimate is without access to corporate or volume discounts
  4. We are assuming that host systems are installed "network close" to consensus nodes and can take advantage of local 10Gb/s network links in most cases. The hardware for 10Gb/s is essentially the same cost as 1Gb/s in this configuration (difference is <$1/month)
  5. We are assuming a lower bandwidth requirement for access to JBOD storage based on primarily archive access, it may be necessary to upgrade to a faster SAS ethernet controller at an estimated cost of $50/month if demand for data more than 24 hours old is higher than expected.