dattobd fails on system with pagesize=64k

[lishuai-ujs]

dattobd fails on system with pagesize=64k

After transition-to-incremental, kernel carsh. Messages as follows:

Jun 18 10:39:21 localhost kernel: [71355.704277] WARNING: CPU: 48 PID: 15140 at drivers/scsi/scsi_lib.c:1195 scsi_init_io+0x128/0x1b0 Jun 18 10:39:21 localhost kernel: [71355.713455] Modules linked in: dattobd(O) ip_set nfnetlink ib_isert iscsi_target_mod ib_srpt vfat fat target_core_mod ib_srp scsi_transport_srp ib_ipoib ib_umad dm_multipath ipmi_ssif rpcrdma sunrpc rdma_ucm ib_iser rdma_cm iw_cm ib_cm libiscsi scsi_transport_iscsi aes_ce_blk crypto_simd cryptd aes_ce_cipher ghash_ce sha2_ce sha256_arm64 sha1_ce sbsa_gwdt ses enclosure hns_roce_hw_v2 hns_roce ib_uverbs ib_core ofpart cmdlinepart ipmi_si hi_sfc ipmi_devintf mtd ipmi_msghandler spi_dw_mmio sch_fq_codel realtek hclge hibmc_drm hns3 hisi_sas_v3_hw hnae3 megaraid_sas ttm hisi_sas_main host_edma_drv [last unloaded: dattobd] Jun 18 10:39:21 localhost kernel: [71355.768362] CPU: 48 PID: 15140 Comm: dbdctl Kdump: loaded Tainted: G O 4.19.90-21.2.ky10.aarch64 #1 Jun 18 10:39:21 localhost kernel: [71355.779091] Hardware name: Huawei TaiShan 200 (Model 2280)/BC82AMDGA, BIOS 1.38 07/04/2020 Jun 18 10:39:21 localhost kernel: [71355.787747] pstate: 20400009 (nzCv daif +PAN -UAO) Jun 18 10:39:21 localhost kernel: [71355.792949] pc : scsi_init_io+0x128/0x1b0 Jun 18 10:39:21 localhost kernel: [71355.797374] lr : sd_setup_read_write_cmnd+0x64/0x868 Jun 18 10:39:21 localhost kernel: [71355.892257] Call trace: Jun 18 10:39:21 localhost kernel: [71355.895124] scsi_init_io+0x128/0x1b0 Jun 18 10:39:21 localhost kernel: [71355.899201] sd_setup_read_write_cmnd+0x64/0x868 Jun 18 10:39:21 localhost kernel: [71355.904229] sd_init_command+0x1f0/0x478 Jun 18 10:39:21 localhost kernel: [71355.908567] scsi_setup_cmnd+0x78/0x140 Jun 18 10:39:21 localhost kernel: [71355.912817] scsi_queue_rq+0x4e0/0x678 Jun 18 10:39:21 localhost kernel: [71355.916985] blk_mq_dispatch_rq_list+0xa0/0x5f8 Jun 18 10:39:21 localhost kernel: [71355.921926] blk_mq_do_dispatch_sched+0x50/0xd8 Jun 18 10:39:21 localhost kernel: [71355.926867] blk_mq_sched_dispatch_requests+0x118/0x1f0 Jun 18 10:39:21 localhost kernel: [71355.932500] blk_mq_run_hw_queue+0x9c/0x120 Jun 18 10:39:21 localhost kernel: [71355.937270] blk_mq_delay_run_hw_queue+0x198/0x1d8 Jun 18 10:39:21 localhost kernel: [71355.942644] blk_mq_run_hw_queue+0x60/0x108 Jun 18 10:39:21 localhost kernel: [71355.947240] blk_mq_sched_insert_requests+0x9c/0x158 Jun 18 10:39:21 localhost kernel: [71355.952615] blk_mq_flush_plug_list+0x1a0/0x2d8 Jun 18 10:39:21 localhost kernel: [71355.957557] blk_flush_plug_list+0xd4/0x270 Jun 18 10:39:21 localhost kernel: [71355.962154] blk_finish_plug+0x40/0x50 Jun 18 10:39:21 localhost kernel: [71355.966321] _xfs_buf_ioapply+0x31c/0x3f8 Jun 18 10:39:21 localhost kernel: [71355.970745] __xfs_buf_submit+0xb0/0x250 Jun 18 10:39:21 localhost kernel: [71355.975087] xlog_bdstrat+0x40/0x88 Jun 18 10:39:21 localhost kernel: [71355.978991] xlog_sync+0x2c8/0x3e0 Jun 18 10:39:21 localhost kernel: [71355.982810] xlog_state_release_iclog+0x94/0xc0 Jun 18 10:39:21 localhost kernel: [71355.987751] xfs_log_force_lsn.isra.10+0x204/0x330 Jun 18 10:39:21 localhost kernel: [71355.993125] xfs_log_force_lsn+0xd8/0x190 Jun 18 10:39:21 localhost kernel: [71355.997549] xfs_trans_commit+0x2a8/0x388 Jun 18 10:39:21 localhost kernel: [71356.002145] xfs_trans_commit+0x24/0x30 Jun 18 10:39:21 localhost kernel: [71356.006396] xfs_sync_sb+0x68/0x78 Jun 18 10:39:21 localhost kernel: [71356.010214] xfs_log_sbcount+0x68/0x88 Jun 18 10:39:21 localhost kernel: [71356.014379] xfs_quiesce_attr+0x64/0xc8 Jun 18 10:39:21 localhost kernel: [71356.018629] xfs_fs_freeze+0x34/0x50 Jun 18 10:39:21 localhost kernel: [71356.022623] freeze_super+0xcc/0x1a8 Jun 18 10:39:21 localhost kernel: [71356.026618] freeze_bdev+0xf0/0xf8 Jun 18 10:39:21 localhost kernel: [71356.030439] tracer_transition_tracing+0x58/0x1c8 [dattobd] Jun 18 10:39:21 localhost kernel: [71356.036590] tracer_setup_tracing+0x98/0x138 [dattobd] Jun 18 10:39:21 localhost kernel: [71356.042309] ioctl_transition_inc+0x164/0x4a8 [dattobd] Jun 18 10:39:21 localhost kernel: [71356.047942] ctrl_ioctl+0x840/0xdb8 [dattobd] Jun 18 10:39:21 localhost kernel: [71356.052712] do_vfs_ioctl+0xb0/0x898 Jun 18 10:39:21 localhost kernel: [71356.056703] ksys_ioctl+0x8c/0xa0 Jun 18 10:39:21 localhost kernel: [71356.060433] __arm64_sys_ioctl+0x28/0x38 Jun 18 10:39:21 localhost kernel: [71356.064776] el0_svc_common+0x84/0x140 Jun 18 10:39:21 localhost kernel: [71356.068941] el0_svc_handler+0x80/0xa0 Jun 18 10:39:21 localhost kernel: [71356.076407] ---[ end trace 325eb1812a1633fc ]--- Jun 18 10:39:21 localhost kernel: [71356.081448] print_req_error: I/O error, dev sda, sector 10495688 Jun 18 10:39:21 localhost kernel: [71356.087862] datto: error reading from base device for copy on write: -5 Jun 18 10:39:21 localhost kernel: [71356.094880] datto: error during bio read complete callback: -5

I add a log to dattobd.c:

@@ -3003,6 +3004,7 @@ static int snap_trace_bio(struct snap_device *dev, struct bio *bio){
        start_sect = ROUND_DOWN(bio_sector(bio) - dev->sd_sect_off, SECTORS_PER_BLOCK) + dev->sd_sect_off;
        end_sect = ROUND_UP(bio_sector(bio) + (bio_size(bio) / SECTOR_SIZE) - dev->sd_sect_off, SECTORS_PER_BLOCK) + dev->sd_sect_off;
        pages = (end_sect - start_sect) / SECTORS_PER_PAGE;
+        LOG_WARN("start_sect %lu, end_sect %lu, pages %u", start_sect, end_sect, pages);

Jun 18 11:02:57 localhost kernel: [ 1256.454234] datto: start_sect 10495984, end_sect 10495992, pages 0

pagesize > COW_BLOCK_SIZE cause pagse=0, then cause crash.

follow patch works well. Any other better ideas?

diff --git a/src/dattobd.c b/src/dattobd.c
index c6867ff..0874971 100644
--- a/src/dattobd.c
+++ b/src/dattobd.c
@@ -718,7 +718,8 @@ static inline void dattobd_bio_copy_dev(struct bio *dst, struct bio *src){

 //macros for defining sector and block sizes
 #define SECTORS_PER_PAGE (PAGE_SIZE / SECTOR_SIZE)
-#define COW_SECTION_SIZE 4096
+#define COW_SECTION_LOG_SIZE 16
+#define COW_SECTION_SIZE (1 << COW_SECTION_LOG_SIZE)
 #define SECTORS_PER_BLOCK (COW_BLOCK_SIZE / SECTOR_SIZE)
 #define SECTOR_TO_BLOCK(sect) ((sect) / SECTORS_PER_BLOCK)
 #define BLOCK_TO_SECTOR(block) ((block) * SECTORS_PER_BLOCK)
@@ -755,7 +756,7 @@ static inline void dattobd_bio_copy_dev(struct bio *dst, struct bio *src){
 #define DATTOBD_MAX_SNAP_DEVICES 255

diff --git a/src/dattobd.h b/src/dattobd.h
index 66487a2..1dc4264 100644
--- a/src/dattobd.h
+++ b/src/dattobd.h
@@ -44,7 +44,7 @@ struct reconfigure_params{
 };

 #define COW_UUID_SIZE 16
-#define COW_BLOCK_LOG_SIZE 12
+#define COW_BLOCK_LOG_SIZE 16
 #define COW_BLOCK_SIZE (1 << COW_BLOCK_LOG_SIZE)
 #define COW_HEADER_SIZE 4096
 #define COW_MAGIC ((uint32_t)4776)

[nixomose]

so yeah, we never built or tested dattobd to work with anything but 4k pages. Our customers tend not to have equipment in that configuration.

I forget all the places where we make assumptions about the page size being 4k but you may also want to watch for cases where alignment problems come up. Like if you have a partition aligned to 63 sectors or raid stripes that overlap disks in a non 64k way, you may run into trouble there. Just something to look out for.

[nixomose]

but off the top of my head, yeah if you make all of the 4k hardcoded things 64k, it seems like it should work. to play nice you might want to base the size changes on the PAGE_SIZE rather than hardcoding 16. Not that our hardcoding of 12 is any better. :-)