search

letsencrypt / openzfs-nvme-databases

Creative Commons Zero v1.0 Universal
572 stars 36 forks source link

Is the ZFS compression actually worth it? #9

Closed jdvorak001 closed 3 years ago

jdvorak001 commented 3 years ago

Thank you for sharing the details of your setup and the thoughts behind it.

I'm wondering a little how big, in your case, the benefits of using the compression are. From #2 I understood it actually saves space only if a 16KB block is compressed to 8KB or less, while the CPU cycles are burnt in any case. So the effect of the compression is kind of just binary and it can't get better than 50%. On the other hand, the InnoDB reads & writes (actually happening on an mmap()-ed file) would be much more straightforward for the filesystem & the kernel if the ZFS compression were not used.

A couple of questions if you allow:

  1. What is the compression ratio you are actually experiencing?
  2. Based on the demand surge you are expecting in the future, when would you be out of storage space (a) with your current setup, and (b) without using the ZFS compression?
  3. Could you possibly estimate the effect of the compression being on/off on your server's CPU utilisation and on its energy consumption?
  4. Should compression be justifiable, wouldn't it be more efficient to partition the big transaction table(s) into a live partition and a series of read-only, archival ones, with a more aggresive compression applied on the latter?
jprenken commented 3 years ago

Thanks for this feedback! I'm adding a caveat to the document, based on your comments.

  1. We currently have a 1.33x compression ratio. (Our database schema is moderately compressible; we store plenty of integers and varchars.)
  2. Given that ratio, and our usage and growth, compression doesn't make any real difference for our storage capacity timelines.
  3. I don't have data for that, unfortunately. The database engine is a much heavier CPU user than the I/O layer, so I'd expect no more than a single-digit percentage. I'll test this if we ever have the spare hardware to do benchmarks.
  4. Definitely. I hinted at that in the "storage priorities" section where I said we ran out of improvements "...in the short term;" we've since made a lot of progress on partitioning. Multi-tiered storage (i.e. a separate set of hardware) wouldn't yet be efficient at our scale, but we're likely to explore it as we grow.