koverstreet / bcachefs

Other
643 stars 71 forks source link

ZSTD compression doesn't work [45ebe8bf0e6d] #614

Closed marcin-github closed 8 months ago

marcin-github commented 8 months ago
# mkfs.bcachefs -L testkompresji   /dev/vg-ssd-1833E14DA86F/testkompresji
External UUID:                              631360b6-9d4a-4062-99f1-e2ddd40e9b25
Internal UUID:                              0ae76b1b-0e98-426c-9fa2-a2d45f1b1d62
Device index:                               0
Label:                                      testkompresji
Version:                                    1.3: rebalance_work
Version upgrade complete:                   0.0: (unknown version)
Oldest version on disk:                     1.3: rebalance_work
Created:                                    Wed Nov 22 14:44:08 2023
Sequence number:                            0
Superblock size:                            952
Clean:                                      0
Devices:                                    1
Sections:                                   members_v1,members_v2
Features:                                   new_siphash,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,new_varint,journal_no_flush,alloc_v2,extents_acr
oss_btree_nodes
Compat features:

Options:
  block_size:                               4.00 KiB
  btree_node_size:                          256 KiB
  errors:                                   continue [ro] panic
  metadata_replicas:                        1
  data_replicas:                            1
  metadata_replicas_required:               1
  data_replicas_required:                   1
  encoded_extent_max:                       64.0 KiB
  metadata_checksum:                        none [crc32c] crc64 xxhash
  data_checksum:                            none [crc32c] crc64 xxhash
  compression:                              none
  background_compression:                   none
  str_hash:                                 crc32c crc64 [siphash]
  metadata_target:                          none
  foreground_target:                        none
  background_target:                        none
  promote_target:                           none
  erasure_code:                             0
  inodes_32bit:                             1
  shard_inode_numbers:                      1
  inodes_use_key_cache:                     1
  gc_reserve_percent:                       8
  gc_reserve_bytes:                         0 B
  root_reserve_percent:                     0
  wide_macs:                                0
  acl:                                      1
  usrquota:                                 0
  grpquota:                                 0
  prjquota:                                 0
  journal_flush_delay:                      1000
  journal_flush_disabled:                   0
  journal_reclaim_delay:                    100
  journal_transaction_names:                1
  version_upgrade:                          [compatible] incompatible none
  nocow:                                    0

members_v2 (size 136):
  Device:                                   0
    Label:                                  (none)
    UUID:                                   e33e4fbd-98cc-4c8b-af46-152c2baa5f9d
    Size:                                   100 GiB
    read errors:                            0
    write errors:                           0
    checksum errors:                        0
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            256 KiB
    First bucket:                           0
    Buckets:                                409600
    Last mount:                             (never)
    State:                                  rw
    Data allowed:                           journal,btree,user
    Has data:                               (none)
    Discard:                                0
    Freespace initialized:                  0
mounting version 1.3: rebalance_work
initializing new filesystem
going read-write
initializing freespace
# cat /sys/fs/bcachefs/631360b6-9d4a-4062-99f1-e2ddd40e9b25/{compression_stats,options/compression,options/background_compression}
uncompressed:
        nr extents:             0
        size:                   0 B
compressed:
        nr extents:             0
        compressed size:        0 B
        uncompressed size:      0 B
incompressible:
        nr extents:             0
        size:                   0 B
none
none

Lets'go:

echo lz4 > /sys/fs/bcachefs/631360b6-9d4a-4062-99f1-e2ddd40e9b25/options/compression
dd if=/dev/zero of=/mnt/test/zero1 bs=1M count=1024 oflag=direct
cat /sys/fs/bcachefs/631360b6-9d4a-4062-99f1-e2ddd40e9b25/{compression_stats,options/compression,options/background_compression}
uncompressed:
        nr extents:             0
        size:                   0 B
compressed:
        nr extents:             16384
        compressed size:        64.0 MiB
        uncompressed size:      1.00 GiB
incompressible:
        nr extents:             0
        size:                   0 B
lz4
none

^- looks correctly Now:

echo zstd > /sys/fs/bcachefs/631360b6-9d4a-4062-99f1-e2ddd40e9b25/options/compression
dd if=/dev/zero of=/mnt/test/zero1 bs=1M count=1024 oflag=direct
cat /sys/fs/bcachefs/631360b6-9d4a-4062-99f1-e2ddd40e9b25/{compression_stats,options/compression,options/background_compression}
uncompressed:
        nr extents:             0
        size:                   0 B
compressed:
        nr extents:             0
        compressed size:        0 B
        uncompressed size:      0 B
incompressible:
        nr extents:             16384
        size:                   1.00 GiB
zstd
none

^ now is bad

koverstreet commented 8 months ago

@.***/T/#u

On Wed, Nov 22, 2023 at 8:56 AM Marcin @.***> wrote:

mkfs.bcachefs -L testkompresji /dev/vg-ssd-1833E14DA86F/testkompresji

External UUID: 631360b6-9d4a-4062-99f1-e2ddd40e9b25 Internal UUID: 0ae76b1b-0e98-426c-9fa2-a2d45f1b1d62 Device index: 0 Label: testkompresji Version: 1.3: rebalance_work Version upgrade complete: 0.0: (unknown version) Oldest version on disk: 1.3: rebalance_work Created: Wed Nov 22 14:44:08 2023 Sequence number: 0 Superblock size: 952 Clean: 0 Devices: 1 Sections: members_v1,members_v2 Features: new_siphash,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,new_varint,journal_no_flush,alloc_v2,extents_acr oss_btree_nodes Compat features:

Options: block_size: 4.00 KiB btree_node_size: 256 KiB errors: continue [ro] panic metadata_replicas: 1 data_replicas: 1 metadata_replicas_required: 1 data_replicas_required: 1 encoded_extent_max: 64.0 KiB metadata_checksum: none [crc32c] crc64 xxhash data_checksum: none [crc32c] crc64 xxhash compression: none background_compression: none str_hash: crc32c crc64 [siphash] metadata_target: none foreground_target: none background_target: none promote_target: none erasure_code: 0 inodes_32bit: 1 shard_inode_numbers: 1 inodes_use_key_cache: 1 gc_reserve_percent: 8 gc_reserve_bytes: 0 B root_reserve_percent: 0 wide_macs: 0 acl: 1 usrquota: 0 grpquota: 0 prjquota: 0 journal_flush_delay: 1000 journal_flush_disabled: 0 journal_reclaim_delay: 100 journal_transaction_names: 1 version_upgrade: [compatible] incompatible none nocow: 0

members_v2 (size 136): Device: 0 Label: (none) UUID: e33e4fbd-98cc-4c8b-af46-152c2baa5f9d Size: 100 GiB read errors: 0 write errors: 0 checksum errors: 0 seqread iops: 0 seqwrite iops: 0 randread iops: 0 randwrite iops: 0 Bucket size: 256 KiB First bucket: 0 Buckets: 409600 Last mount: (never) State: rw Data allowed: journal,btree,user Has data: (none) Discard: 0 Freespace initialized: 0 mounting version 1.3: rebalance_work initializing new filesystem going read-write initializing freespace

cat /sys/fs/bcachefs/631360b6-9d4a-4062-99f1-e2ddd40e9b25/{compression_stats,options/compression,options/background_compression}

uncompressed: nr extents: 0 size: 0 B compressed: nr extents: 0 compressed size: 0 B uncompressed size: 0 B incompressible: nr extents: 0 size: 0 B none none

Lets'go:

echo lz4 > /sys/fs/bcachefs/631360b6-9d4a-4062-99f1-e2ddd40e9b25/options/compression dd if=/dev/zero of=/mnt/test/zero1 bs=1M count=1024 oflag=direct cat /sys/fs/bcachefs/631360b6-9d4a-4062-99f1-e2ddd40e9b25/{compression_stats,options/compression,options/background_compression} uncompressed: nr extents: 0 size: 0 B compressed: nr extents: 16384 compressed size: 64.0 MiB uncompressed size: 1.00 GiB incompressible: nr extents: 0 size: 0 B lz4 none

^- looks correctly Now:

echo zstd > /sys/fs/bcachefs/631360b6-9d4a-4062-99f1-e2ddd40e9b25/options/compression dd if=/dev/zero of=/mnt/test/zero1 bs=1M count=1024 oflag=direct cat /sys/fs/bcachefs/631360b6-9d4a-4062-99f1-e2ddd40e9b25/{compression_stats,options/compression,options/background_compression} uncompressed: nr extents: 0 size: 0 B compressed: nr extents: 0 compressed size: 0 B uncompressed size: 0 B incompressible: nr extents: 16384 size: 1.00 GiB zstd none

^ now is bad

— Reply to this email directly, view it on GitHub https://github.com/koverstreet/bcachefs/issues/614, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAPGX3TALVNGDA73QHCZUCLYFYACXAVCNFSM6AAAAAA7WI6PZ2VHI2DSMVQWIX3LMV43ASLTON2WKOZSGAYDMNBQGYYDGOA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

koverstreet commented 8 months ago

This is fixed now - zstd lies about the size of the compression workspace we need, doubling the allocation size fixed it.