Dikay900
commented
11 months ago It seems i have data on non existing devices now based on the metadata.
available devices are currently ids: 0 1 2 3 4 8 9 11
It seems as device 5 is getting checked although I do not see any data on this id.
replicas_v0:
user: 2 [2 11] user: 2 [0 2] cached: 1 [0] user: 1 [1] user: 2 [1 4] user: 2 [4 11] cached: 1 [4] btree: 2 [0 1] user: 1 [8] user: 2 [0 9] user: 2 [2 3] user: 2 [3 11] user: 2 [8 11] cached: 1 [2] cached: 1 [9] journal: 2 [4 9] btree: 2 [0 3] user: 1 [3] user: 1 [11] user: 2 [0 4] user: 2 [1 2] user: 2 [1 9] user: 2 [2 8] user: 2 [3 8] user: 2 [4 9] user: 2 [8 9] user: 2 [9 11] cached: 1 [1] cached: 1 [3] cached: 1 [8] cached: 1 [11] journal: 2 [0 2] journal: 2 [8 9] btree: 2 [0 2] user: 1 [0] user: 1 [2] user: 1 [4] user: 1 [9] user: 2 [0 1] user: 2 [0 3] user: 2 [0 8] user: 2 [0 11] user: 2 [1 3] user: 2 [1 8] user: 2 [1 11] user: 2 [2 4] user: 2 [2 9] user: 2 [3 4] user: 2 [3 9] user: 2 [4 8]With the following trace output added in bch2_have_enough_devs: replicas.c:932
bool bch2_have_enough_devs(struct bch_fs *c, struct bch_devs_mask devs,
unsigned flags, bool print)
{
struct bch_replicas_entry *e;
bool ret = true;
percpu_down_read(&c->mark_lock);
for_each_cpu_replicas_entry(&c->replicas, e) {
unsigned i, nr_online = 0, nr_failed = 0, dflags = 0;
bool metadata = e->data_type < BCH_DATA_user;
if (e->data_type == BCH_DATA_cached)
continue;
for (i = 0; i < e->nr_devs; i++) {
+ bch_err(c, "checking dev index (%u) of replicas (%u) with type (%u)",
+ e->devs[i], e->nr_devs, e->data_type);
struct bch_dev *ca = bch_dev_bkey_exists(c, e->devs[i]);
nr_online += test_bit(e->devs[i], devs.d);
nr_failed += ca->mi.state == BCH_MEMBER_STATE_failed;
}
I am getting the following output:
checking dev index (2) of replicas (2) with type (4)
checking dev index (8) of replicas (2) with type (4)
checking dev index (11) of replicas (1) with type (4)
checking dev index (0) of replicas (2) with type (3)
checking dev index (3) of replicas (2) with type (3)
checking dev index (1) of replicas (2) with type (4)
checking dev index (2) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (4)
checking dev index (9) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (2)
checking dev index (9) of replicas (2) with type (2)
checking dev index (3) of replicas (1) with type (4)
checking dev index (0) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (4)
checking dev index (1) of replicas (2) with type (4)
checking dev index (9) of replicas (2) with type (4)
checking dev index (3) of replicas (2) with type (4)
checking dev index (8) of replicas (2) with type (4)
checking dev index (8) of replicas (2) with type (4)
checking dev index (11) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (2)
checking dev index (5) of replicas (2) with type (2)
bcachefs: libbcachefs/sb-members.h:160: bch_dev_bkey_exists: Assertion `!(idx >= c->sb.nr_devices || !c->devs[idx])' failed.looking into bch2_dev_remove I see the following comment, which could actually be the case here
/*
* At this point the device object has been removed in-core, but the
* on-disk journal might still refer to the device index via sb device
* usage entries. Recovery fails if it sees usage information for an
* invalid device. Flush journal pins to push the back of the journal
* past now invalid device index references before we update the
* superblock, but after the device object has been removed so any
* further journal writes elide usage info for the device.
*/
bch2_journal_flush_all_pins(&c->journal);
actually I see that the device 5 was added during the journal replay. Since the checking dev index trace is printed at the beginning of the fsck where no dev 5 is there:
~/bcachefs-tools# bcachefs fsck -pv /dev/sdl /dev/sda /dev/sdd /dev/sdb /dev/sdg /dev/sdh /dev/sdj /dev/sdf
checking dev index (2) of replicas (2) with type (4)
checking dev index (11) of replicas (2) with type (4)
checking dev index (0) of replicas (2) with type (4)
checking dev index (2) of replicas (2) with type (4)
checking dev index (1) of replicas (1) with type (4)
checking dev index (1) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (4)
checking dev index (11) of replicas (2) with type (4)
checking dev index (0) of replicas (2) with type (3)
checking dev index (1) of replicas (2) with type (3)
checking dev index (8) of replicas (1) with type (4)
checking dev index (0) of replicas (2) with type (4)
checking dev index (9) of replicas (2) with type (4)
checking dev index (2) of replicas (2) with type (4)
checking dev index (3) of replicas (2) with type (4)
checking dev index (3) of replicas (2) with type (4)
checking dev index (11) of replicas (2) with type (4)
checking dev index (8) of replicas (2) with type (4)
checking dev index (11) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (2)
checking dev index (9) of replicas (2) with type (2)
checking dev index (0) of replicas (2) with type (3)
checking dev index (3) of replicas (2) with type (3)
checking dev index (3) of replicas (1) with type (4)
checking dev index (11) of replicas (1) with type (4)
checking dev index (0) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (4)
checking dev index (1) of replicas (2) with type (4)
checking dev index (2) of replicas (2) with type (4)
checking dev index (1) of replicas (2) with type (4)
checking dev index (9) of replicas (2) with type (4)
checking dev index (2) of replicas (2) with type (4)
checking dev index (8) of replicas (2) with type (4)
checking dev index (3) of replicas (2) with type (4)
checking dev index (8) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (4)
checking dev index (9) of replicas (2) with type (4)
checking dev index (8) of replicas (2) with type (4)
checking dev index (9) of replicas (2) with type (4)
checking dev index (9) of replicas (2) with type (4)
checking dev index (11) of replicas (2) with type (4)
checking dev index (0) of replicas (2) with type (2)
checking dev index (2) of replicas (2) with type (2)
checking dev index (8) of replicas (2) with type (2)
checking dev index (9) of replicas (2) with type (2)
checking dev index (0) of replicas (2) with type (3)
checking dev index (2) of replicas (2) with type (3)
checking dev index (0) of replicas (1) with type (4)
checking dev index (2) of replicas (1) with type (4)
checking dev index (4) of replicas (1) with type (4)
checking dev index (9) of replicas (1) with type (4)
checking dev index (0) of replicas (2) with type (4)
checking dev index (1) of replicas (2) with type (4)
checking dev index (0) of replicas (2) with type (4)
checking dev index (3) of replicas (2) with type (4)
checking dev index (0) of replicas (2) with type (4)
checking dev index (8) of replicas (2) with type (4)
checking dev index (0) of replicas (2) with type (4)
checking dev index (11) of replicas (2) with type (4)
checking dev index (1) of replicas (2) with type (4)
checking dev index (3) of replicas (2) with type (4)
checking dev index (1) of replicas (2) with type (4)
checking dev index (8) of replicas (2) with type (4)
checking dev index (1) of replicas (2) with type (4)
checking dev index (11) of replicas (2) with type (4)
checking dev index (2) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (4)
checking dev index (2) of replicas (2) with type (4)
checking dev index (9) of replicas (2) with type (4)
checking dev index (3) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (4)
checking dev index (3) of replicas (2) with type (4)
checking dev index (9) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (4)
checking dev index (8) of replicas (2) with type (4)
mounting version 1.3: rebalance_work opts=metadata_replicas=2,data_replicas=2,metadata_target=fast,foreground_target=fast,background_target=hdd,promote_target=fast,degraded,fsck,fix_errors=yes
recovering from unclean shutdown
starting journal read
journal read done on device 0x55fc71bd15c0g, ret 0
journal read done on device 0x55fc71bca5e0g, ret 0
journal read done on device 0x55fc71bc27f0g, ret 0
journal read done on device 0x55fc71bc3e60g, ret 0
journal read done on device 0x55fc71bd1770g, ret 0
journal read done on device 0x55fc71bdd400g, ret 0
journal read done on device 0x55fc71bca770g, ret 0
journal read done on device 0x55fc71bdd770g, ret 0
journal read done, replaying entries 25734944-25735261
invalid journal entry, version=1.3: rebalance_work type=dev_usage seq=25734944 offset=2009/2151: bad dev, fixing
[..]
invalid journal entry, version=1.3: rebalance_work type=dev_usage seq=25735261 offset=67279/67421: bad dev, fixing
Journal keys: 27066 read, 18675 after sorting and compacting
checking dev index (2) of replicas (2) with type (4)
checking dev index (8) of replicas (2) with type (4)
checking dev index (11) of replicas (1) with type (4)
checking dev index (0) of replicas (2) with type (3)
checking dev index (3) of replicas (2) with type (3)
checking dev index (1) of replicas (2) with type (4)
checking dev index (2) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (4)
checking dev index (9) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (2)
checking dev index (9) of replicas (2) with type (2)
checking dev index (3) of replicas (1) with type (4)
checking dev index (0) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (4)
checking dev index (1) of replicas (2) with type (4)
checking dev index (9) of replicas (2) with type (4)
checking dev index (3) of replicas (2) with type (4)
checking dev index (8) of replicas (2) with type (4)
checking dev index (8) of replicas (2) with type (4)
checking dev index (11) of replicas (2) with type (4)
checking dev index (4) of replicas (2) with type (2)
checking dev index (5) of replicas (2) with type (2)
bcachefs: libbcachefs/sb-members.h:160: bch_dev_bkey_exists: Assertion `!(idx >= c->sb.nr_devices || !c->devs[idx])' failed.
data type 2 is journal so in the beginning there was no journal entry for dev 5 but after the replay there is. Probably the scenario described in the comment above during dev remove.
So now the question would be how can i just ignore the readded device or properly remove it after journal replay or how to skip the journal replay here.
koverstreet
Version
bcachefs master e2cd10f254d
Generic info
I am currently tearing down my old array and moving the data to a new one, removing some devices from the existing array. After the latest removal I cannot mount the array anymore. Trying to fsck results into a crash see below: Are there more options available i can get into read only mode of this array without replaying the journal or dropping the journal keys? Since I am reading only from this array anyway I do not need write access anymore.
fsck with debug usertools gives me
bcachefs: libbcachefs/sb-members.h:160: bch_dev_bkey_exists: Assertion '!(idx >= c->sb.nr_devices || !c->devs[idx])' failed.