search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.42k stars 1.72k forks source link

Request: compress ZIL writes #8896

Open romanshein opened 5 years ago

romanshein commented 5 years ago

Case description: Proxmox Raidz2 made of 6 disks VD is a raw on zvol w zlv compression. zfs is v. 0.7.13

I'm performing free space zeroing. The combination of sync=always and logbias=latency provides for maximum zero-write performance (200-400 MB/s within the guest).

I observe an interesting thing at "zpool iostat -v" :
Normally w (sync always and logbias latency) the amount of data written to slog is equal to amount of data written to pool, slog = pool/1.5 (the data written to pool in 1.5 times higher due to parity overhead). In this particular case the situation is completely different. While slog devices write at maximum possible speed (200 MB/s each), the total pool writes are 10-20 times less of slog's ones (whole pool writes are in the region of 5MB/s, including overhead + pool reads (probably metadata) approximately at the same rate). Again, this is not a burst activity, but a steady state (and I have just few gigs), no caching can explain this. I believe it is looks like pool writes are getting properly compressed (as it is zle, the pool writing essentially metadata), while slog writes without any compression whatsoever. This is especially ridiculous as it writes zeros. With proper compression the write speed would increase by an order of magnitude I suppose.

romanshein commented 5 years ago

At the same time sync=always w logbias=throughput works as I would expect (slog writes at a speed similar to the pool, which implies slog writes are getting compressed now).

romanshein commented 5 years ago

It looks like logbias=latency is the main factor. Sync can be always or standard, but as soon as you switch logbias to latency the slog compression is getting switched off.

scineram commented 5 years ago

Why would you waste time compressing for the slog?

ahrens commented 5 years ago

Is it normal that slog writes do not undergo compression?

Yes.

sync=always w logbias=throughput

logbias=throughput means that ZFS will write the data directly to the main pool, in its final form (including compression). The ZIL blocks (written to the slog device) will only contain metadata.

devZer0 commented 4 years ago

Why would you waste time compressing for the slog?

because it could make your ssd last longer, as it has limited lifetime (tbw/pbw) ?

romanshein commented 4 years ago

because it could make your ssd last longer, as it has limited lifetime (tbw/pbw) ?

Although it will be a significant workload for CPU and RAM, because it will have to compress->decompress ->compress again or at least run 2 independent data compression streams, because a slog and a pool rarely have the same layout (the same layout, like mirror slog and mirror vdev is rare indeed). The in-memory buffer will have to be split to allocate memory to keep compressed slog data. Compression is known for causing latency.

On the other hand I was thinking mostly about compression for zero only data stream. It is not rare, because of virtualization. Theoretically ZLE is not taxing. It shouldn't cause a significant latency for non-compressible data, while dramatically increase throughput (above physical specs of a slog device) and decrease latency for at zero-only streams (f.i.,a case of virtual disk shrinking). The arrival of trim probably make my case a bit less appealing, yet TRIM (even if it make it correctly through all levels of virtualization to vdev), will not kill zero streams completely. There are other scenarios for zero only streams.

Some SSDs perform compression themselves anyway, yet SSD's on-board processor cannot compete with a CPU anyway.

The bottom line: I propose OpenZFS to implement at least slog ZLE compression for compressed file system. It will deliver simultaneously the significant performance gain with zero only streams (higher throughput, lower pool latency) and lower slog wear speed.

stale[bot] commented 3 years ago

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

devZer0 commented 3 years ago

so let's conclude, there are two significant advantages:

  1. saving ssd lifetime
  2. speedup for zero only data streams

i think the user should decide to have compression for slog, so this should be configurable