production checklist: add ext4 and xfs file system mount options guidance

[sheaffej]

John Sheaffer (sheaffej) commented:

I am seeing many customers struggling with IO performance on servers with locally attached disks, only to find they have the default file system mount options which are suboptimal for any RDBMS that stores data as files in a file system.

We should consider adding guidance on how to mount filesystems to hold the CockroachDB data files (i.e. the location specified by --store or the default of ./cockroach-data).

Suggestion: In the Production Checklist consider adding verbiage such as the following: https://www.cockroachlabs.com/docs/v21.1/recommended-production-settings.html#storage

---

• The recommended Linux filesystems are ext4 and XFS. <---- this list item already exists, add the sublist items)
  • Mount ext4 or XFS file systems with the defaults,noatime,nodiratime,nobarrier options
  • Examples:
    • On-demand mounting: mount -t ext4 -o defaults,noatime,nodiratime,nobarrier /dev/nvme0n1 /data
    • Mount on reboot: sudo sh -c "echo '/dev/nvme0n1 /data ext4 defaults,noatime,nodiratime,nobarrier 0 2' >> /etc/fstab"
  • The defaults, noatime and nodiratime options are recommended in all installations
  • The nobarrier option is recommended in the following situations
  • When the storage is enterprise-class and has battery-backed cache

  • When ephemeral SSD storage (aka instance storage) is used in AWS, GCP and Azure

References:

• https://linux.die.net/man/8/mount (describes the mount options)
• https://www.cockroachlabs.com/blog/2018_cloud_report/ (see "Why nobarrier matters")
• https://cloud.google.com/compute/docs/disks/optimizing-local-ssd-performance#disable_flush
• https://www.slideshare.net/AmazonWebServices/maximizing-ec2-and-elastic-block-store-disk-performance-32781088 (slide 35)
• https://docs.microsoft.com/en-us/azure/virtual-machines/premium-storage-performance#optimize-performance-on-linux-vms (Azure don't use nobarrier)

Jira Issue: DOC-1577

[andrewbaptist]

If we are providing guidance to customers on setting up ext4, we should tune 2 additional settings to save over 5% of disk space at virtually no risk.

1) Reduce the reserved space. Setting it to 0 might be risky, but setting it to 1% should be fine. Cloudera similarly recommends this: https://docs.cloudera.com/cloudera-manager/7.4.2/managing-clusters/topics/cm-decrease-reserved-space.html. This change will save 4% of disk space.

2) Use a smaller bytes-per-inode parameter. Storage team would have the best value to use here, but something like -i 131072 would be very conservative and save almost 1.25%.