Evacuating a device moves data into foreground permanently [e2cd10f254dd]

Please search for duplicates

Found no other problem with "evacuate" which describes my problem.

Version I am using bcachefs.git master commit e2cd10f254dd bcachefs-tools.git master commit c6e69549288

Generic info Provide the output of:

bcachefs show-super
External UUID:                              f0605888-9e17-4fec-bef6-92e885adc371
Internal UUID:                              3edf84f6-82fa-4eeb-9376-47005224497e
Device index:                               3
Label:                                      
Version:                                    1.3: rebalance_work
Version upgrade complete:                   1.3: rebalance_work
Oldest version on disk:                     1.1: snapshot_skiplists
Created:                                    Thu Aug 10 09:03:52 2023
Sequence number:                            987
Superblock size:                            13536
Clean:                                      0
Devices:                                    12
Sections:                                   members_v1,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters,members_v2,errors
Features:                                   zstd,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes
Compat features:                            alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done

Options:
  block_size:                               4.00 KiB
  btree_node_size:                          256 KiB
  errors:                                   continue [ro] panic 
  metadata_replicas:                        2
  data_replicas:                            2
  metadata_replicas_required:               1
  data_replicas_required:                   1
  encoded_extent_max:                       64.0 KiB
  metadata_checksum:                        none [crc32c] crc64 xxhash 
  data_checksum:                            none [crc32c] crc64 xxhash 
  compression:                              zstd:1
  background_compression:                   zstd:9
  str_hash:                                 crc32c crc64 [siphash] 
  metadata_target:                          ssd
  foreground_target:                        ssd
  background_target:                        hdd
  promote_target:                           ssd
  erasure_code:                             0
  inodes_32bit:                             1
  shard_inode_numbers:                      1
  inodes_use_key_cache:                     1
  gc_reserve_percent:                       8
  gc_reserve_bytes:                         0 B
  root_reserve_percent:                     0
  wide_macs:                                0
  acl:                                      1
  usrquota:                                 0
  grpquota:                                 0
  prjquota:                                 0
  journal_flush_delay:                      1000
  journal_flush_disabled:                   0
  journal_reclaim_delay:                    100
  journal_transaction_names:                1
  version_upgrade:                          [compatible] incompatible none 
  nocow:                                    0

replicas_v0 (size 376):
  user: 2 [5 9] user: 2 [1 11] user: 2 [9 11] btree: 2 [7 11] user: 2 [3 8] user: 2 [7 9] cached: 1 [5] journal: 2 [7 11] user: 2 [1 3] user: 2 [2 9] user: 2 [4 8] user: 2 [6 9] user: 2 [8 10] cached: 1 [1] cached: 1 [9] journal: 2 [5 7] btree: 2 [5 7] user: 1 [8] user: 2 [1 7] user: 2 [2 5] user: 2 [3 4] user: 2 [3 12] user: 2 [4 12] user: 2 [6 7] user: 2 [6 11] user: 2 [7 11] user: 2 [8 12] user: 2 [10 11] cached: 1 [3] cached: 1 [7] cached: 1 [11] journal: 2 [4 10] journal: 2 [6 7] journal: 2 [9 10] btree: 2 [6 7] user: 1 [3] user: 1 [10] user: 2 [1 5] user: 2 [1 9] user: 2 [2 3] user: 2 [2 7] user: 2 [2 11] user: 2 [3 6] user: 2 [3 10] user: 2 [4 6] user: 2 [4 10] user: 2 [5 7] user: 2 [5 11] user: 2 [6 8] user: 2 [6 10] user: 2 [7 8] user: 2 [7 10] user: 2 [8 9] user: 2 [8 11] user: 2 [9 10] user: 2 [9 12] user: 2 [10 12] cached: 1 [2] cached: 1 [4] cached: 1 [6] cached: 1 [8] cached: 1 [10] cached: 1 [12] journal: 2 [4 8] journal: 2 [5 6] journal: 2 [5 11] journal: 2 [6 11] journal: 2 [8 9] btree: 2 [5 6] btree: 2 [5 11] btree: 2 [6 11] user: 1 [1] user: 1 [4] user: 1 [9] user: 2 [1 2] user: 2 [1 4] user: 2 [1 6] user: 2 [1 8] user: 2 [1 10] user: 2 [1 12] user: 2 [2 4] user: 2 [2 6] user: 2 [2 8] user: 2 [2 10] user: 2 [2 12] user: 2 [3 5] user: 2 [3 7] user: 2 [3 9] user: 2 [3 11] user: 2 [4 5] user: 2 [4 7] user: 2 [4 9] user: 2 [4 11] user: 2 [5 6] user: 2 [5 8] user: 2 [5 10]

members_v2 (size 1576):
  Device:                                   1
    Label:                                  hdd2 (2)
    UUID:                                   c9b36005-d887-4967-a98f-bf0a74303f88
    Size:                                   3.64 TiB
    read errors:                            0
    write errors:                           0
    checksum errors:                        0
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                7630895
    Last mount:                             Sun Nov 19 16:07:59 2023
    State:                                  rw
    Data allowed:                           journal,btree,user
    Has data:                               user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   2
    Label:                                  hdd3 (3)
    UUID:                                   f13d99b0-a4d8-4d23-a205-6e70b05d76e7
    Size:                                   3.64 TiB
    read errors:                            0
    write errors:                           0
    checksum errors:                        0
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                7630895
    Last mount:                             Sun Nov 19 16:07:59 2023
    State:                                  ro
    Data allowed:                           journal,btree,user
    Has data:                               user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   3
    Label:                                  hdd4 (4)
    UUID:                                   7da308d8-deaf-4f17-babf-5c8fa4dca33d
    Size:                                   3.64 TiB
    read errors:                            0
    write errors:                           0
    checksum errors:                        0
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                7630895
    Last mount:                             Sun Nov 19 16:07:59 2023
    State:                                  rw
    Data allowed:                           journal,btree,user
    Has data:                               user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   4
    Label:                                  hdd5 (5)
    UUID:                                   c1f38d5c-d588-49d8-9713-7234d09c2be7
    Size:                                   7.28 TiB
    read errors:                            0
    write errors:                           0
    checksum errors:                        0
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            1.00 MiB
    First bucket:                           0
    Buckets:                                7630855
    Last mount:                             Sun Nov 19 16:07:59 2023
    State:                                  rw
    Data allowed:                           journal,btree,user
    Has data:                               journal,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   5
    Label:                                  ssd1 (9)
    UUID:                                   f32a73fb-91a3-4034-b6e9-df126718dbd7
    Size:                                   932 GiB
    read errors:                            0
    write errors:                           0
    checksum errors:                        0
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                1907739
    Last mount:                             Sun Nov 19 16:07:59 2023
    State:                                  rw
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                1
    Freespace initialized:                  1
  Device:                                   6
    Label:                                  ssd4 (12)
    UUID:                                   2154f2e4-49d5-435f-9947-e876e31d1791
    Size:                                   932 GiB
    read errors:                            0
    write errors:                           0
    checksum errors:                        0
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                1907739
    Last mount:                             Sun Nov 19 16:07:59 2023
    State:                                  rw
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   7
    Label:                                  ssd2 (10)
    UUID:                                   25306296-aa14-49fe-b0e4-5e4a46527af7
    Size:                                   466 GiB
    read errors:                            0
    write errors:                           0
    checksum errors:                        0
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                953880
    Last mount:                             Sun Nov 19 16:07:59 2023
    State:                                  rw
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                1
    Freespace initialized:                  1
  Device:                                   8
    Label:                                  hdd6 (6)
    UUID:                                   4b2ef818-7a27-421d-a99f-7192139e7a70
    Size:                                   5.46 TiB
    read errors:                            0
    write errors:                           0
    checksum errors:                        0
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            1.00 MiB
    First bucket:                           0
    Buckets:                                5723136
    Last mount:                             Sun Nov 19 16:07:59 2023
    State:                                  rw
    Data allowed:                           journal,btree,user
    Has data:                               journal,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   9
    Label:                                  hdd7 (7)
    UUID:                                   ef6b86f4-d94e-4b34-8eb8-4c1ad2ce5edb
    Size:                                   7.28 TiB
    read errors:                            0
    write errors:                           0
    checksum errors:                        0
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            1.00 MiB
    First bucket:                           0
    Buckets:                                7630855
    Last mount:                             Sun Nov 19 16:07:59 2023
    State:                                  rw
    Data allowed:                           journal,btree,user
    Has data:                               journal,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   10
    Label:                                  hdd8 (17)
    UUID:                                   0660728f-deef-4420-a2a9-f1e02b88ef63
    Size:                                   5.46 TiB
    read errors:                            0
    write errors:                           0
    checksum errors:                        0
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            1.00 MiB
    First bucket:                           0
    Buckets:                                5723136
    Last mount:                             Sun Nov 19 16:07:59 2023
    State:                                  rw
    Data allowed:                           journal,btree,user
    Has data:                               journal,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   11
    Label:                                  ssd3 (11)
    UUID:                                   53bb3640-5f0f-4d24-9c4a-77e4963e17b9
    Size:                                   932 GiB
    read errors:                            298
    write errors:                           3215
    checksum errors:                        0
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                1907739
    Last mount:                             Sun Nov 19 16:07:59 2023
    State:                                  rw
    Data allowed:                           journal,btree,user
    Has data:                               journal,btree,user,cached
    Discard:                                0
    Freespace initialized:                  1
  Device:                                   12
    Label:                                  ssd5 (18)
    UUID:                                   c50de25d-4124-4445-8f9c-f2dea8296e12
    Size:                                   932 GiB
    read errors:                            13
    write errors:                           157231
    checksum errors:                        0
    seqread iops:                           0
    seqwrite iops:                          0
    randread iops:                          0
    randwrite iops:                         0
    Bucket size:                            512 KiB
    First bucket:                           0
    Buckets:                                1907739
    Last mount:                             Mon Nov 20 06:18:01 2023
    State:                                  rw
    Data allowed:                           journal,btree,user
    Has data:                               user,cached
    Discard:                                0
    Freespace initialized:                  1

Problem

While evacuating a disk with 2TB of user data I saw that all disks are getting filled regardless whether they are background or foreground devices.

bcachefs evacuate /dev/sdk -> hdd.hdd3 (device 2): sdk

After some minutes i got the following usage:

bcachefs fs usage -h /srv/bcachefs
Filesystem: f0605888-9e17-4fec-bef6-92e885adc371
Size:                       37.2 TiB
Used:                       24.0 TiB
Online reserved:             424 KiB

Data type       Required/total  Devices
reserved:       1/1                    [] 16.6 MiB
btree:          1/2             [sdg sde]                   8.70 GiB
btree:          1/2             [sdh sdg]                   23.7 GiB
btree:          1/2             [sdd sdg]                   26.0 GiB
btree:          1/2             [sdh sdd]                   53.7 GiB
btree:          1/2             [sdh sde]                   15.7 GiB
btree:          1/2             [sdd sde]                   75.1 GiB
user:           1/2             [sdh sdj]                    411 GiB
user:           1/2             [sdb sde]                   17.9 GiB
user:           1/2             [sdj sde]                   52.2 GiB
user:           1/2             [sda sdn]                    959 GiB
user:           1/2             [sdg sdj]                    170 GiB
user:           1/2             [sdb sda]                    573 GiB
user:           1/2             [sdk sdj]                   1.20 TiB
user:           1/2             [sdi sdn]                   1.94 TiB
user:           1/2             [sdd sdj]                   50.3 GiB
user:           1/2             [sdn sdm]                    275 GiB
user:           1/2             [sdb sdg]                   42.7 GiB
user:           1/2             [sdk sdh]                   93.7 GiB
user:           1/2             [sda sdi]                   1.14 TiB
user:           1/2             [sda sdc]                   5.77 MiB
user:           1/2             [sdi sdc]                   1.99 MiB
user:           1/2             [sdd sdg]                   1.70 GiB
user:           1/2             [sdd sde]                   45.9 MiB
user:           1/2             [sdg sde]                   1.71 GiB
user:           1/2             [sdn sdc]                   1.77 MiB
user:           1/2             [sdm sde]                    330 MiB
user:           1/1             [sda]                        140 KiB
user:           1/1             [sdm]                        836 KiB
user:           1/2             [sdb sdh]                    105 GiB
user:           1/2             [sdb sdj]                   1.42 TiB
user:           1/2             [sdk sda]                    566 GiB
user:           1/2             [sdk sdg]                   37.7 GiB
user:           1/2             [sdk sde]                   16.8 GiB
user:           1/2             [sda sdd]                   17.1 GiB
user:           1/2             [sda sdm]                    165 GiB
user:           1/2             [sdi sdd]                   35.2 GiB
user:           1/2             [sdi sdm]                    353 GiB
user:           1/2             [sdh sdg]                   22.4 GiB
user:           1/2             [sdh sde]                   4.45 GiB
user:           1/2             [sdd sdn]                   27.0 GiB
user:           1/2             [sdd sdm]                    334 MiB
user:           1/2             [sdg sdn]                   64.8 GiB
user:           1/2             [sdg sdm]                   11.7 GiB
user:           1/2             [sdn sdj]                   2.06 TiB
user:           1/2             [sdn sde]                   28.0 GiB
user:           1/2             [sdj sdm]                    494 GiB
user:           1/2             [sdj sdc]                   2.38 GiB
user:           1/2             [sdm sdc]                    432 KiB
user:           1/1             [sdb]                        436 KiB
user:           1/1             [sdi]                        428 KiB
user:           1/1             [sdj]                        232 KiB
user:           1/2             [sdb sdk]                    554 GiB
user:           1/2             [sdb sdi]                   1.14 TiB
user:           1/2             [sdb sdd]                   17.3 GiB
user:           1/2             [sdb sdn]                    958 GiB
user:           1/2             [sdb sdm]                    166 GiB
user:           1/2             [sdb sdc]                   5.59 MiB
user:           1/2             [sdk sdi]                   1.10 TiB
user:           1/2             [sdk sdd]                   16.2 GiB
user:           1/2             [sdk sdn]                    952 GiB
user:           1/2             [sdk sdm]                    158 GiB
user:           1/2             [sdk sdc]                   8.18 MiB
user:           1/2             [sda sdh]                    107 GiB
user:           1/2             [sda sdg]                   43.0 GiB
user:           1/2             [sda sdj]                   1.43 TiB
user:           1/2             [sda sde]                   17.7 GiB
user:           1/2             [sdi sdh]                    238 GiB
user:           1/2             [sdi sdg]                   99.7 GiB
user:           1/2             [sdi sdj]                   4.29 TiB
user:           1/2             [sdi sde]                   36.6 GiB
user:           1/2             [sdh sdd]                   4.24 GiB
user:           1/2             [sdh sdn]                    159 GiB
user:           1/2             [sdh sdm]                   29.6 GiB
cached:         1/1             [sdh]                        200 GiB
cached:         1/1             [sdb]                        790 GiB
cached:         1/1             [sdj]                        734 GiB
cached:         1/1             [sda]                        776 GiB
cached:         1/1             [sdg]                       89.4 GiB
cached:         1/1             [sde]                       96.5 GiB
cached:         1/1             [sdk]                        941 GiB
cached:         1/1             [sdi]                       1.08 TiB
cached:         1/1             [sdd]                       93.6 GiB
cached:         1/1             [sdn]                       1.00 TiB
cached:         1/1             [sdm]                       8.26 MiB
cached:         1/1             [sdc]                       56.0 KiB

hdd.hdd2 (device 1):             sdb              rw
                                data         buckets    fragmented
  free:                          0 B          298106
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                         0 B               0
  user:                     2.47 TiB         5181638      4.28 GiB
  cached:                    790 GiB         2142951
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               1
  erasure coded:                 0 B               0
  capacity:                 3.64 TiB         7630895

hdd.hdd3 (device 2):             sdk              ro
                                data         buckets    fragmented
  free:                          0 B          301132
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                         0 B               0
  user:                     2.32 TiB         4882436      12.0 GiB
  cached:                    941 GiB         2439128
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                 3.64 TiB         7630895

hdd.hdd4 (device 3):             sda              rw
                                data         buckets    fragmented
  free:                          0 B          298106
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                         0 B               0
  user:                     2.48 TiB         5216089      4.30 GiB
  cached:                    776 GiB         2108499
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               2
  erasure coded:                 0 B               0
  capacity:                 3.64 TiB         7630895

hdd.hdd5 (device 4):             sdi              rw
                                data         buckets    fragmented
  free:                          0 B          298092
  sb:                       3.00 MiB               4      1020 KiB
  journal:                  8.00 GiB            8192
  btree:                         0 B               0
  user:                     5.18 TiB         5444890      14.1 GiB
  cached:                   1.08 TiB         1879675
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               2
  erasure coded:                 0 B               0
  capacity:                 7.28 TiB         7630855

hdd.hdd6 (device 8):             sdn              rw
                                data         buckets    fragmented
  free:                          0 B          223572
  sb:                       3.00 MiB               4      1020 KiB
  journal:                  8.00 GiB            8192
  btree:                         0 B               0
  user:                     3.67 TiB         3860604      8.41 GiB
  cached:                   1.00 TiB         1630763
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               1
  erasure coded:                 0 B               0
  capacity:                 5.46 TiB         5723136

hdd.hdd7 (device 9):             sdj              rw
                                data         buckets    fragmented
  free:                          0 B          298093
  sb:                       3.00 MiB               4      1020 KiB
  journal:                  8.00 GiB            8192
  btree:                         0 B               0
  user:                     5.78 TiB         6075520      16.9 GiB
  cached:                    734 GiB         1249045
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               1
  erasure coded:                 0 B               0
  capacity:                 7.28 TiB         7630855

hdd.hdd8 (device 10):            sdm              rw
                                data         buckets    fragmented
  free:                          0 B         4867560
  sb:                       3.00 MiB               4      1020 KiB
  journal:                  8.00 GiB            8192
  btree:                         0 B               0
  user:                      826 GiB          847364      1.04 GiB
  cached:                   8.26 MiB              16
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                 5.46 TiB         5723136

ssd.ssd1 (device 5):             sdh              rw
                                data         buckets    fragmented
  free:                          0 B          169839
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                    46.6 GiB          104454      4.45 GiB
  user:                      587 GiB         1205401      1.20 GiB
  cached:                    200 GiB          419846
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                  932 GiB         1907739

ssd.ssd2 (device 7):             sdg              rw
                                data         buckets    fragmented
  free:                          0 B          182267
  sb:                       3.00 MiB               7       508 KiB
  journal:                  3.64 GiB            7452
  btree:                    29.2 GiB           66124      3.12 GiB
  user:                      248 GiB          509577      1.15 GiB
  cached:                   89.4 GiB          188452
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               1
  erasure coded:                 0 B               0
  capacity:                  466 GiB          953880

ssd.ssd3 (device 11):            sde              rw
                                data         buckets    fragmented
  free:                          0 B         1398029
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                    49.7 GiB          114315      6.08 GiB
  user:                     87.9 GiB          180194      99.8 MiB
  cached:                   96.5 GiB          207001
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               1
  erasure coded:                 0 B               0
  capacity:                  932 GiB         1907739

ssd.ssd4 (device 6):             sdd              rw
                                data         buckets    fragmented
  free:                          0 B         1351134
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                    77.4 GiB          173359      7.25 GiB
  user:                     84.7 GiB          173754      94.2 MiB
  cached:                   93.6 GiB          201293
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                  932 GiB         1907739

ssd.ssd5 (device 12):            sdc              rw
                                data         buckets    fragmented
  free:                          0 B         1897073
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                         0 B               0
  user:                     1.20 GiB            2467      1.39 MiB
  cached:                   56.0 KiB               0
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                  932 GiB         1907739

See below for fs usage after removal / evacuation of the device.

Seeing that ssd.ssd[1-5] is getting user data (The filesystem was idle except for the evacuation) Looking into iostat it seems like every device gets data based on free space (as all were background devices)

The user data on ssd.ssd1 and ssd.ssd2 was from another evacuation where they were the only two ssd.* devices. The user data on these devices is not getting rebalanced into the background after finish of the evacuation as well as remounting / relabeling the devices

I think this behaviour started when upgrading to v1.3 the rebalance work. I would argue that it was working fine with an earlier version.

EDIT: removed the fs usage and posted a new one below after complete evacuation

Evacuation + remove of the device hdd.hdd3 (device 2): sdk is done.

As you can see the foreground devices have now even more user data which are not getting rebalanced or moved. Rebalance is idle as well.

New data written to the filesystem is handled correctly.

~#  bcachefs fs usage -h /srv/bcachefs
Filesystem: f0605888-9e17-4fec-bef6-92e885adc371
Size:                       33.9 TiB
Used:                       24.0 TiB
Online reserved:             496 KiB

Data type       Required/total  Devices
reserved:       1/1                    [] 16.4 MiB
btree:          1/2             [sde sda]                   14.1 GiB
btree:          1/2             [sdl sdf]                   7.81 GiB
btree:          1/2             [sdm sdl]                   20.9 GiB
btree:          1/2             [sde sdf]                   14.9 GiB
btree:          1/2             [sda sdl]                   20.7 GiB
btree:          1/2             [sde sdm]                   4.04 GiB
btree:          1/2             [sde sdl]                   2.48 GiB
btree:          1/2             [sdm sda]                   42.4 GiB
btree:          1/2             [sdm sdf]                   13.7 GiB
btree:          1/2             [sda sdf]                   60.5 GiB
user:           1/2             [sdm sda]                   8.71 GiB
user:           1/2             [sde sdh]                   46.6 GiB
user:           1/2             [sdj sdi]                    637 GiB
user:           1/2             [sdb sdj]                   1.10 TiB
user:           1/2             [sda sdh]                   94.6 GiB
user:           1/2             [sdc sdj]                   1.10 TiB
user:           1/2             [sdg sdj]                   2.30 TiB
user:           1/2             [sdm sdi]                   60.1 GiB
user:           1/2             [sdl sdh]                    185 GiB
user:           1/2             [sdi sdf]                   12.3 GiB
user:           1/2             [sde sdm]                   4.09 GiB
user:           1/2             [sdc sdg]                   1.34 TiB
user:           1/2             [sdb sdg]                   1.34 TiB
user:           1/2             [sdg sdm]                    268 GiB
user:           1/2             [sdg sdi]                    796 GiB
user:           1/2             [sdm sdj]                    184 GiB
user:           1/2             [sda sdl]                   3.54 GiB
user:           1/2             [sda sdf]                   1.62 GiB
user:           1/2             [sdl sdf]                   3.64 GiB
user:           1/2             [sdh sdi]                    948 GiB
user:           1/2             [sde sdb]                   25.6 GiB
user:           1/2             [sde sdl]                   1.64 GiB
user:           1/2             [sde sdf]                    948 MiB
user:           1/2             [sdc sda]                   38.7 GiB
user:           1/2             [sdc sdi]                    370 GiB
user:           1/2             [sdb sda]                   39.1 GiB
user:           1/2             [sdb sdi]                    373 GiB
user:           1/2             [sdg sdl]                    115 GiB
user:           1/2             [sdg sdh]                   4.68 TiB
user:           1/2             [sdg sdf]                   81.4 GiB
user:           1/2             [sdm sdl]                   24.0 GiB
user:           1/2             [sdm sdh]                    439 GiB
user:           1/2             [sdm sdf]                   9.13 GiB
user:           1/2             [sda sdj]                   62.7 GiB
user:           1/2             [sda sdi]                   12.1 GiB
user:           1/2             [sdl sdj]                   77.4 GiB
user:           1/2             [sdl sdi]                   24.6 GiB
user:           1/2             [sdj sdh]                   2.37 TiB
user:           1/2             [sdj sdf]                   64.9 GiB
user:           1/2             [sdh sdf]                   98.0 GiB
user:           1/2             [sde sdc]                   25.0 GiB
user:           1/2             [sde sdg]                   50.9 GiB
user:           1/2             [sde sda]                    920 MiB
user:           1/2             [sde sdj]                   45.9 GiB
user:           1/2             [sde sdi]                   8.77 GiB
user:           1/2             [sdc sdb]                    666 GiB
user:           1/2             [sdc sdm]                    119 GiB
user:           1/2             [sdc sdl]                   50.2 GiB
user:           1/2             [sdc sdh]                   1.59 TiB
user:           1/2             [sdc sdf]                   39.9 GiB
user:           1/2             [sdb sdm]                    122 GiB
user:           1/2             [sdb sdl]                   50.6 GiB
user:           1/2             [sdb sdh]                   1.61 TiB
user:           1/2             [sdb sdf]                   40.2 GiB
user:           1/2             [sdg sda]                   78.5 GiB
cached:         1/1             [sdg]                        848 GiB
cached:         1/1             [sdj]                        801 GiB
cached:         1/1             [sdc]                        628 GiB
cached:         1/1             [sda]                        135 GiB
cached:         1/1             [sdi]                       18.2 MiB
cached:         1/1             [sde]                       24.7 GiB
cached:         1/1             [sdb]                        618 GiB
cached:         1/1             [sdm]                        214 GiB
cached:         1/1             [sdl]                       98.7 GiB
cached:         1/1             [sdh]                        598 GiB
cached:         1/1             [sdf]                        139 GiB

hdd.hdd2 (device 1):             sdc              rw
                                data         buckets    fragmented
  free:                          0 B          298109
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                         0 B               0
  user:                     2.65 TiB         5577701      4.77 GiB
  cached:                    628 GiB         1746886
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                 3.64 TiB         7630895

hdd.hdd4 (device 3):             sdb              rw
                                data         buckets    fragmented
  free:                          0 B          298109
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                         0 B               0
  user:                     2.67 TiB         5606137      4.77 GiB
  cached:                    618 GiB         1718450
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                 3.64 TiB         7630895

hdd.hdd5 (device 4):             sdg              rw
                                data         buckets    fragmented
  free:                          0 B          298094
  sb:                       3.00 MiB               4      1020 KiB
  journal:                  8.00 GiB            8192
  btree:                         0 B               0
  user:                     5.51 TiB         5795569      14.9 GiB
  cached:                    848 GiB         1528996
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                 7.28 TiB         7630855

hdd.hdd6 (device 8):             sdj              rw
                                data         buckets    fragmented
  free:                          0 B          223574
  sb:                       3.00 MiB               4      1020 KiB
  journal:                  8.00 GiB            8192
  btree:                         0 B               0
  user:                     3.96 TiB         4161843      9.14 GiB
  cached:                    801 GiB         1329523
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                 5.46 TiB         5723136

hdd.hdd7 (device 9):             sdh              rw
                                data         buckets    fragmented
  free:                          0 B          298096
  sb:                       3.00 MiB               4      1020 KiB
  journal:                  8.00 GiB            8192
  btree:                         0 B               0
  user:                     6.01 TiB         6317470      17.5 GiB
  cached:                    598 GiB         1007093
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                 7.28 TiB         7630855

hdd.hdd8 (device 10):            sdi              rw
                                data         buckets    fragmented
  free:                          0 B         4052403
  sb:                       3.00 MiB               4      1020 KiB
  journal:                  8.00 GiB            8192
  btree:                         0 B               0
  user:                     1.58 TiB         1662522      2.19 GiB
  cached:                   18.2 MiB              15
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                 5.46 TiB         5723136

ssd.ssd1 (device 5):             sdm              rw
                                data         buckets    fragmented
  free:                          0 B           79505
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                    40.5 GiB           99164      7.89 GiB
  user:                      619 GiB         1270162      1.25 GiB
  cached:                    214 GiB          450709
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                  932 GiB         1907739

ssd.ssd2 (device 7):             sdl              rw
                                data         buckets    fragmented
  free:                          0 B          124249
  sb:                       3.00 MiB               7       508 KiB
  journal:                  3.64 GiB            7452
  btree:                    25.9 GiB           63153      4.90 GiB
  user:                      268 GiB          550696      1.19 GiB
  cached:                   98.7 GiB          208322
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               1
  erasure coded:                 0 B               0
  capacity:                  466 GiB          953880

ssd.ssd3 (device 11):            sdf              rw
                                data         buckets    fragmented
  free:                          0 B         1118365
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                    48.5 GiB          123285      11.7 GiB
  user:                      176 GiB          361112       244 MiB
  cached:                    139 GiB          296778
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                  932 GiB         1907739

ssd.ssd4 (device 6):             sda              rw
                                data         buckets    fragmented
  free:                          0 B         1089934
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                    68.8 GiB          172112      15.2 GiB
  user:                      170 GiB          349195       239 MiB
  cached:                    135 GiB          288299
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                  932 GiB         1907739

ssd.ssd5 (device 0):             sde              rw
                                data         buckets    fragmented
  free:                          0 B         1588931
  sb:                       3.00 MiB               7       508 KiB
  journal:                  4.00 GiB            8192
  btree:                    17.8 GiB           44061      3.75 GiB
  user:                      105 GiB          215612      83.8 MiB
  cached:                   24.7 GiB           50936
  parity:                        0 B               0
  stripe:                        0 B               0
  need_gc_gens:                  0 B               0
  need_discard:                  0 B               0
  erasure coded:                 0 B               0
  capacity:                  932 GiB         1907739
~# cat /sys/fs/bcachefs/f0605888-9e17-4fec-bef6-92e885adc371/internal/rebalance_status 
waiting
  io wait duration:  14.6 MiB
  io wait remaining: 11.4 MiB
  duration waited:   13634 h

koverstreet / bcachefs

Evacuating a device moves data into foreground permanently [e2cd10f254dd] #613