akiradeveloper
commented
4 years ago I am afaid after you fsck the filesystem and change the state, we can't track the root cause any more but I wish you used this tool to check whether the dm-writeboost is broken or not.
https://github.com/akiradeveloper/dm-writeboost-tools
And of course you should have checked the dmesg if there is some message from dm-writeboost. At least, the configuration must be shared.
About flawlessness of the software, in my opinion, dm-writeboost will not be broken in a usual operation because there are users that operate more harder more longer days. So the typical situation, dm-writeboost may go insane is that you shut down the system in unusual manner. I design the dm-writeboost so it survives even in this situation but I can't prove that the isn't anything wrong in code.
When dm-writeboost is broken, it is typically seen that the checksum is broken. The tool wbcheck is the one to check this. (checksum is broken means writing the segment is ended in unanticipated consequence)
If you ever deconstructed the dm-writeboost device after your system went wrong and the device was rebuilt and then sawthe dmesg, there were chances we could check if the checksums are not broken because the kernel code check all the segments. This code means, dm-writeboost discards segments after failure (e.g. partial writes)
/*
* Compare the checksum
* if they don't match we discard the subsequent logs.
*/
actual = calc_checksum(rambuf, header->length);
expected = le32_to_cpu(header->checksum);
if (actual != expected) {
DMWARN("Checksum incorrect id:%llu checksum: %u != %u",
(long long unsigned int) le64_to_cpu(header->id),
actual, expected);
break;
}However, since you are telling that most of the data are gone, I don't believe only the some final writes have failed but the entire data is broken. I don't know it is due to dm-writeboost or ext4 or your operation.
nhocchi007
Dear akira, After 13 days running dm-writeboost on my disk something wrong happen, i cannot interact with the folder /data/ anymore using ls, mkdir, add file.... dmesg log fault: [3341551.692063] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm ls: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3341552.701259] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm ls: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3341553.339575] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm ls: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3341554.576510] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm ls: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3341653.508665] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm bash: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3341653.731893] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm bash: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3341663.437200] EXT4-fs error (device dm-5): ext4_find_dest_de:1653: inode #2: block 14654: comm mkdir: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3341665.889757] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm ls: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3341666.823193] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm ls: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3341671.900039] EXT4-fs error (device dm-5): ext4_find_dest_de:1653: inode #2: block 14654: comm mkdir: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3341742.492629] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm ls: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3341953.555294] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm find: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3341977.426021] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm find: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3341981.320515] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm find: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3341983.834547] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm find: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3342035.975328] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm lsof: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3342038.377535] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm lsof: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3342045.639684] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm lsof: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3342046.804830] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm lsof: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3342047.532793] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm lsof: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3342194.645129] EXT4-fs (dm-5): warning: mounting fs with errors, running e2fsck is recommended [3342195.055847] EXT4-fs (dm-5): mounted filesystem with ordered data mode. Opts: (null) [3342199.564925] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm ls: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3342200.426116] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm ls: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3342203.563006] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm ls: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3342221.592676] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm find: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3342226.812250] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm find: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3342229.270561] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm find: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3342242.942348] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm find: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4 [3342254.850072] EXT4-fs error (device dm-5): htree_dirblock_to_tree:914: inode #2: block 14654: comm find: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=677963375, rec_len=28910, name_len=4
but dm -writeboost block still running with lsblock: NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 2.6T 0 disk └─sda1 8:1 0 2.6T 0 part └─ssddisk-ssd 253:4 0 2.6T 0 lvm
└─wbdev 253:5 0 98.2T 0 dm /data sdb 8:16 0 98.2T 0 disk └─sdb1 8:17 0 98.2T 0 part └─hdddisk-hdd 253:3 0 98.2T 0 lvm
└─wbdev 253:5 0 98.2T 0 dm /data sdd 8:48 0 447.1G 0 disk ├─sdd1 8:49 0 200M 0 part /boot/efi ├─sdd2 8:50 0 1G 0 part /boot └─sdd3 8:51 0 445.9G 0 part ├─centos-root 253:0 0 217.9G 0 lvm / ├─centos-swap 253:1 0 128G 0 lvm [SWAP] └─centos-home 253:2 0 100G 0 lvm /home I'm afraid that when writeback threshold is meet something glitch or bug between my software and dm-writeboost. Is that possible? I have try umount and mount that data folder again but nothing changed, still can not interact with the folder. Then i use fsck trying to fix file system, lost+found folder created but all the other folder and data had been removed. I know it's not your responsibility to this fault but can you tell me how can i debug it to the root cause? Thank you akira