search

koverstreet / bcachefs

Other
662 stars 70 forks source link

Resize changes device's capacity but does not use the newly available space - b56e84bccab0 #596

Closed ticpu closed 1 month ago

ticpu commented 11 months ago

Version Kernel is at 6.5.1.arch1.r1204752.b56e84bccab0 bcachefs tool version v0.1-736-ga053ebf

Generic info Provide the output of:

# bcachefs fs usage
btree:          1/3             [dm-21 dm-22 dm-19]         2.01 GiB
btree:          1/3             [dm-21 dm-22 dm-20]         20.2 GiB
btree:          1/3             [dm-21 dm-22 dm-18]         1.09 GiB
user:           1/1             [dm-18]                     24.7 GiB
user:           1/2             [dm-20 dm-19]                577 GiB
user:           1/1             [dm-21]                     39.1 GiB
user:           1/2             [dm-21 dm-18]               1.62 MiB
user:           1/1             [dm-20]                      158 GiB
user:           1/2             [dm-21 dm-20]               8.44 MiB
user:           1/2             [dm-22 dm-19]               11.1 MiB
user:           1/2             [dm-19 dm-18]               98.6 GiB
user:           1/1             [dm-22]                     39.7 GiB
user:           1/1             [dm-19]                      160 GiB
user:           1/2             [dm-21 dm-22]                387 MiB
user:           1/2             [dm-21 dm-19]               11.0 MiB
user:           1/2             [dm-22 dm-20]               11.5 MiB
user:           1/2             [dm-22 dm-18]               3.01 MiB
user:           1/2             [dm-20 dm-18]               89.7 GiB
cached:         1/1             [dm-22]                     28.0 MiB
cached:         1/1             [dm-21]                      572 KiB
hdd.1 (device 2):              dm-20              rw
  free:                          0 B               4
  btree:                    6.74 GiB           27624
  user:                      491 GiB         2012167
  cached:                        0 B               0
  need_discard:                  0 B               0
  capacity:                  800 GiB         3276800
hdd.2 (device 3):              dm-19              rw
  free:                          0 B               6
  btree:                     685 MiB            2741
  user:                      497 GiB         2037048
  cached:                        0 B               0
  need_discard:                  0 B               0
  capacity:                  800 GiB         3276800
hdd.3 (device 4):              dm-18              rw
  free:                          0 B               1
  btree:                     373 MiB            1491
  user:                      119 GiB          486815
  cached:                        0 B               0
  need_discard:                  0 B               0
  capacity:                  120 GiB          491520
nvme.1 (device 0):             dm-21              rw
  free:                          0 B          134388
  btree:                    7.78 GiB           31856
  user:                     39.3 GiB          160779       408 KiB
  cached:                    572 KiB               4
  need_discard:                  0 B               0
  capacity:                  200 GiB          819200
nvme.2 (device 1):             dm-22              rw
  free:                          0 B          131757
  btree:                    7.78 GiB           31856
  user:                     39.9 GiB          163295       880 KiB
  cached:                   28.0 MiB             119
  need_discard:                  0 B               0
  capacity:                  100 GiB          409600

# bcachefs show-super
External UUID:                              d0337b38-cbe7-4c3c-90f7-4305fbe05171
Internal UUID:                              3b2af93d-5b21-47e1-8abd-15d2a46ff64a
Device index:                               0
Label:                                      
Version:                                    1.2: deleted_inodes
Version upgrade complete:                   1.2: deleted_inodes
Oldest version on disk:                     1.1: snapshot_skiplists
Created:                                    Sat Aug 26 16:58:08 2023
Sequence number:                            274
Superblock size:                            6952
Clean:                                      0
Devices:                                    5
Sections:                                   members,crypt,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters
Features:                                   lz4,zstd,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:                            alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                               4.00 KiB
  btree_node_size:                          256 KiB
  errors:                                   continue [ro] panic 
  metadata_replicas:                        3
  data_replicas:                            2
  metadata_replicas_required:               2
  data_replicas_required:                   1
  encoded_extent_max:                       64.0 KiB
  metadata_checksum:                        none [crc32c] crc64 xxhash 
  data_checksum:                            none [crc32c] crc64 xxhash 
  compression:                              none
  background_compression:                   zstd
  str_hash:                                 crc32c crc64 [siphash] 
  metadata_target:                          nvme
  foreground_target:                        nvme
  background_target:                        hdd
  promote_target:                           nvme
  erasure_code:                             0
  inodes_32bit:                             1
  shard_inode_numbers:                      1
  inodes_use_key_cache:                     1
  gc_reserve_percent:                       8
  gc_reserve_bytes:                         0 B
  root_reserve_percent:                     0
  wide_macs:                                0
  acl:                                      1
  usrquota:                                 0
  grpquota:                                 0
  prjquota:                                 0
  journal_flush_delay:                      1000
  journal_flush_disabled:                   0
  journal_reclaim_delay:                    100
  journal_transaction_names:                1
  version_upgrade:                          [compatible] incompatible none 
  nocow:                                    0

members (size 288):
  Device:                                   0
    UUID:                                   600a073b-417a-4f27-a8ed-e8959a3cb13c
    Size:                                   200 GiB
    Bucket size:                            256 KiB
    First bucket:                           0
    Buckets:                                819200
    Last mount:                             Wed Sep 27 00:06:19 2023
    State:                                  rw
    Label:                                  1 (1)
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                1
    Freespace initialized:                  1
  Device:                                   1
    UUID:                                   f106c284-7a5c-4828-8111-fdf0f2d63228
    Size:                                   100 GiB
    Bucket size:                            256 KiB
    First bucket:                           0
    Buckets:                                409600
    Last mount:                             Wed Sep 27 00:06:19 2023
    State:                                  rw
    Label:                                  2 (2)
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                1
    Freespace initialized:                  1
  Device:                                   2
    UUID:                                   97b5abdf-9563-4ab8-9fff-a1e86a6ae4a6
    Size:                                   800 GiB
    Bucket size:                            256 KiB
    First bucket:                           0
    Buckets:                                3276800
    Last mount:                             Wed Sep 27 00:06:19 2023
    State:                                  rw
    Label:                                  1 (4)
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   3
    UUID:                                   14a64f04-883f-4cc9-8fa2-2c4b1a131aff
    Size:                                   800 GiB
    Bucket size:                            256 KiB
    First bucket:                           0
    Buckets:                                3276800
    Last mount:                             Wed Sep 27 00:06:19 2023
    State:                                  rw
    Label:                                  2 (5)
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   4
    UUID:                                   72182ee5-f08e-4a03-8c35-2143d249b56d
    Size:                                   120 GiB
    Bucket size:                            256 KiB
    First bucket:                           0
    Buckets:                                491520
    Last mount:                             Wed Sep 27 00:06:19 2023
    State:                                  rw
    Label:                                  3 (6)
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                0
    Freespace initialized:                  1

Kernel bugs Compile the kernel with these flags:

CONFIG_PREEMPT=y
CONFIG_BCACHEFS_DEBUG is not set <<
CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_DEBUG_FS=y
CONFIG_DYNAMIC_FTRACE=y
CONFIG_FTRACE=y

Output of dmesg when bch-copygc is stuck in a loop trying to find space et rebalance is also stuck: https://pastebin.com/V8DZqa7Q

I'm compressing a perf record at the moment that I can distribute if anything. Here's a screenshot in the meantime. image

To reproduce, create a bcachefs with small device size, fill the device to more than 50%. Grow one or more device, use bcachefs device resize on each grown device. Fill the filesystem, it'll get stuck to the original maximum size and "fs usage" will report 0 free, user+btree+cached will not equal capacity.

Remounting the filesystem will allow bcachefs to use the newly available space.

ticpu commented 11 months ago

As recommended by @koverstreet I'll try to reproduce the issue in ktest