search

openstreetmap / operations

OSMF Operations Working Group issue tracking
https://operations.osmfoundation.org/
98 stars 13 forks source link

Resize NVMe sector size on snap-03 #1141

Open Firefishy opened 1 month ago

Firefishy commented 1 month ago

The NVMe disks in snap-03 are using 512 sectors instead of 4096 byte sectors. The sector size should likely be changed if boosts performance.

LBA Format  0 : Metadata Size: 0   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good (in use)
LBA Format  1 : Metadata Size: 8   bytes - Data Size: 512 bytes - Relative Performance: 0x2 Good
LBA Format  2 : Metadata Size: 0   bytes - Data Size: 4096 bytes - Relative Performance: 0x2 Good
LBA Format  3 : Metadata Size: 8   bytes - Data Size: 4096 bytes - Relative Performance: 0x2 Good
LBA Format  4 : Metadata Size: 64  bytes - Data Size: 4096 bytes - Relative Performance: 0x2 Good

Changing the disk sector size formats the disk, the sector size needs to be carefully coordinated with mdadm.

Outline steps:

  1. Confirm array idle via /proc/mdstats
  2. mdadm --fail
  3. mdadm --remove
  4. nvme format --lbaf=2
  5. mdadm --add

(repeat disk by disk across all disks)