search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.45k stars 1.73k forks source link

When you remove single HDD, the raidz3 pool could be import #8175

Closed homerl closed 3 years ago

homerl commented 5 years ago

System information

Type Version/Name
Distribution Name CentOS
Distribution Version 7.4
Linux Kernel 3.10.0-693.5.2.el7_lustre.x86_64
Architecture AMD64
ZFS Version 0.7.3-1
SPL Version 0.7.3-1

Describe the problem you're observing

When I reboot the server, single zpool could not be mounted.
When I remove single HDD (echo 1 > /sys/class/block/sddi/device/delete) in this zpool , Why the zpool import success ? Thank you.

$ zpool import -f -o  cachefile=none -o readonly=on ost_61
cannot import 'ost_61': I/O error Destroy and re-create the pool from a backup source.

zpool import show 

     pool: ost_61
     id: 15498746923132605816
  state: FAULTED
 status: The pool metadata is corrupted.
 action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
   see: http://zfsonlinux.org/msg/ZFS-8000-72
 config:        ost_61                      FAULTED  corrupted data
          raidz3-0                  ONLINE
            sddd                    ONLINE
            sdde                    ONLINE
            sddf                    ONLINE
            sddg                    ONLINE
            sddh                    ONLINE
            sddi                    ONLINE
            scsi-35000c500a66c86f7  ONLINE
            scsi-35000c500a675b907  ONLINE
            scsi-35000c500a665afe7  ONLINE
            scsi-35000c500a670c7a3  ONLINE
            scsi-35000c500a66d665f  ONLINE
            scsi-35000c500a65a86b7  ONLINE
            scsi-35000c500a664a617  ONLINE
            scsi-35000c500a665b353  ONLINE
            scsi-35000c500a670c9fb  ONLINE
            scsi-35000c500a670b10f  ONLINE
            scsi-35000c500a65a9027  ONLINE
            scsi-35000c500a65a02af  ONLINE
            scsi-35000c500a670b213  ONLINE
            scsi-35000c500a670b157  ONLINE
            scsi-35000c500a665b1b3  ONLINE
            scsi-35000c500a670c5fb  ONLINE

I found the input / ouput error in sddi , the others is OK.

zdb -e ost_61 -d -t 1274825
Dataset mos [META], ID 0, cr_txg 4, 492M, 284 objects
Dataset ost_61/ost_61 [ZPL], ID 155, cr_txg 1744, 59.3T, 662232 objects
Dataset ost_61 [ZPL], ID 51, cr_txg 1, 68.1K, 6 objects
Verified large_blocks feature refcount of 1 is correct
Verified large_dnode feature refcount of 0 is correct
Verified sha512 feature refcount of 0 is correct
Verified skein feature refcount of 0 is correct
Verified edonr feature refcount of 0 is correct
Verified userobj_accounting feature refcount of 2 is correct
----sddh--finish-----
----sddi-----
zdb -e ost_61 -d -t 1414018
zdb: can't open 'ost_61': Input/output error
zdb -e ost_61 -d -t 1414019
zdb: can't open 'ost_61': Input/output error
zdb -e ost_61 -d -t 1414020
zdb: can't open 'ost_61': Input/output error
zdb -e ost_61 -d -t 1414021
zdb: can't open 'ost_61': Input/output error
zdb -e ost_61 -d -t 1414022
zdb: can't open 'ost_61': Input/output error
zdb -e ost_61 -d -t 1414023
zdb: can't open 'ost_61': Input/output error
zdb -e ost_61 -d -t 1414025
zdb: can't open 'ost_61': Input/output error

......
zdb -e ost_61 -d -t 1274813
Dataset mos [META], ID 0, cr_txg 4, 492M, 284 objects
Dataset ost_61/ost_61 [ZPL], ID 155, cr_txg 1744, 59.3T, 662232 objects
Dataset ost_61 [ZPL], ID 51, cr_txg 1, 68.1K, 6 objects
Verified large_blocks feature refcount of 1 is correct
Verified large_dnode feature refcount of 0 is correct
Verified sha512 feature refcount of 0 is correct
Verified skein feature refcount of 0 is correct
Verified edonr feature refcount of 0 is correct
Verified userobj_accounting feature refcount of 2 is correct
zdb -e ost_61 -d -t 1274814
Dataset mos [META], ID 0, cr_txg 4, 492M, 284 objects
Dataset ost_61/ost_61 [ZPL], ID 155, cr_txg 1744, 59.3T, 662232 objects
Dataset ost_61 [ZPL], ID 51, cr_txg 1, 68.1K, 6 objects
Verified large_blocks feature refcount of 1 is correct
Verified large_dnode feature refcount of 0 is correct
Verified sha512 feature refcount of 0 is correct
Verified skein feature refcount of 0 is correct
Verified edonr feature refcount of 0 is correct
Verified userobj_accounting feature refcount of 2 is correct
zdb -e ost_61 -d -t 1414007
zdb: can't open 'ost_61': Input/output error
.....

----sddi--finish-----
----scsi-35000c500a66c86f7-----
zdb -e ost_61 -d -t 1274826
Dataset mos [META], ID 0, cr_txg 4, 492M, 284 objects
Dataset ost_61/ost_61 [ZPL], ID 155, cr_txg 1744, 59.3T, 662232 objects
Dataset ost_61 [ZPL], ID 51, cr_txg 1, 68.1K, 6 objects
Verified large_blocks feature refcount of 1 is correct
Verified large_dnode feature refcount of 0 is correct
Verified sha512 feature refcount of 0 is correct
Verified skein feature refcount of 0 is correct
Verified edonr feature refcount of 0 is correct
Verified userobj_accounting feature refcount of 2 is correct
zdb -e ost_61 -d -t 1274827
Dataset mos [META], ID 0, cr_txg 4, 492M, 284 objects
Dataset ost_61/ost_61 [ZPL], ID 155, cr_txg 1744, 59.3T, 662232 objects

......
Dataset mos [META], ID 0, cr_txg 4, 492M, 284 objects
Dataset ost_61/ost_61 [ZPL], ID 155, cr_txg 1744, 59.3T, 662232 objects
Dataset ost_61 [ZPL], ID 51, cr_txg 1, 68.1K, 6 objects
Verified large_blocks feature refcount of 1 is correct
Verified large_dnode feature refcount of 0 is correct
Verified sha512 feature refcount of 0 is correct
Verified skein feature refcount of 0 is correct
Verified edonr feature refcount of 0 is correct
Verified userobj_accounting feature refcount of 2 is correct

Describe how to reproduce the problem

Sorry, I just reboot the server, no other operate, after reboot the zpool could not be imported. because it 's the production env, I have to recover the service ASAP. I 'm just save first 100MB for /dev/sddi

Include any warning/errors/backtraces from the system logs

# zdb -e ost_61 -p /dev/disk/by-idConfiguration for import:
        vdev_children: 1
        version: 5000
        pool_guid: 15498746923132605816
        name: 'ost_61'
        state: 0
        hostid: 3219179115
        hostname: 'cngb-oss-c25-4.cngb.sz.hpc'
        vdev_tree:
            type: 'root'
            id: 0
            guid: 15498746923132605816
            children[0]:
                type: 'raidz'
                id: 0
                guid: 18120440563767675975
                nparity: 3
                metaslab_array: 65
                metaslab_shift: 40
                ashift: 9
                asize: 215529714352128
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 691489041564037318
                    whole_disk: 1
                    DTL: 277
                    create_txg: 4
                    path: '/dev/disk/by-id/wwn-0x5000c500a670b7bf-part1'
                    devid: 'scsi-35000c500a670b7bf-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a670b7bd-lun-0'
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 2793954157375116890
                    whole_disk: 1
                    DTL: 276
                    create_txg: 4
                    path: '/dev/disk/by-id/wwn-0x5000c500a65a9a73-part1'
                    devid: 'scsi-35000c500a65a9a73-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a65a9a71-lun-0'
                children[2]:
                    type: 'disk'
                    id: 2
                    guid: 11324950091598464835
                    whole_disk: 1
                    DTL: 275
                    create_txg: 4
                    path: '/dev/disk/by-id/wwn-0x5000c500a670cd0f-part1'
                    devid: 'scsi-35000c500a670cd0f-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a670cd0d-lun-0'
                children[3]:
                    type: 'disk'
                    id: 3
                    guid: 14639020071569065218
                    whole_disk: 1
                    DTL: 274
                    create_txg: 4
                    path: '/dev/disk/by-id/wwn-0x5000c500a66cc9c7-part1'
                    devid: 'scsi-35000c500a66cc9c7-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a66cc9c5-lun-0'
                children[4]:
                    type: 'disk'
                    id: 4
                    guid: 7471169359584585262
                    whole_disk: 1
                    DTL: 273
                    create_txg: 4
                    path: '/dev/disk/by-id/wwn-0x5000c500a663e3d7-part1'
                    devid: 'scsi-35000c500a663e3d7-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a663e3d5-lun-0'
                children[5]:
                    type: 'disk'
                    id: 5
                    guid: 2256939013648686921
                    whole_disk: 1
                    DTL: 525
                    create_txg: 4
                    path: '/dev/disk/by-id/wwn-0x5000c500a670c8eb-part1'
                    devid: 'scsi-35000c500a670c8eb-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a670c8e9-lun-0'
                children[6]:
                    type: 'disk'
                    id: 6
                    guid: 16663238592269080088
                    whole_disk: 1
                    DTL: 271
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a66c86f7-part1'
                    devid: 'scsi-35000c500a66c86f7-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a66c86f5-lun-0'
                children[7]:
                    type: 'disk'
                    id: 7
                    guid: 6001772952532611291
                    whole_disk: 1
                    DTL: 270
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a675b907-part1'
                    devid: 'scsi-35000c500a675b907-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a675b905-lun-0'
                children[8]:
                    type: 'disk'
                    id: 8
                    guid: 8880497111011200992
                    whole_disk: 1
                    DTL: 269
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a665afe7-part1'
                    devid: 'scsi-35000c500a665afe7-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a665afe5-lun-0'
                children[9]:
                    type: 'disk'
                    id: 9
                    guid: 16220702193743177328
                    whole_disk: 1
                    DTL: 268
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a670c7a3-part1'
                    devid: 'scsi-35000c500a670c7a3-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a670c7a1-lun-0'
                children[10]:
                    type: 'disk'
                    id: 10
                    guid: 16720204657942376868
                    whole_disk: 1
                    DTL: 267
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a66d665f-part1'
                    devid: 'scsi-35000c500a66d665f-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a66d665d-lun-0'
                children[11]:
                    type: 'disk'
                    id: 11
                    guid: 4688300806090570331
                    whole_disk: 1
                    DTL: 266
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a65a86b7-part1'
                    devid: 'scsi-35000c500a65a86b7-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a65a86b5-lun-0'
                children[12]:
                    type: 'disk'
                    id: 12
                    guid: 10290898539518718447
                    whole_disk: 1
                    DTL: 265
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a664a617-part1'
                    devid: 'scsi-35000c500a664a617-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a664a615-lun-0'
                children[13]:
                    type: 'disk'
                    id: 13
                    guid: 7807535298729944426
                    whole_disk: 1
                    DTL: 264
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a665b353-part1'
                    devid: 'scsi-35000c500a665b353-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a665b351-lun-0'
                children[14]:
                    type: 'disk'
                    id: 14
                    guid: 15544210171960321421
                    whole_disk: 1
                    DTL: 263
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a670c9fb-part1'
                    devid: 'scsi-35000c500a670c9fb-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a670c9f9-lun-0'
                children[15]:
                    type: 'disk'
                    id: 15
                    guid: 12606861152537374219
                    whole_disk: 1
                    DTL: 262
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a670b10f-part1'
                    devid: 'scsi-35000c500a670b10f-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a670b10d-lun-0'
                children[16]:
                    type: 'disk'
                    id: 16
                    guid: 2608076077830433444
                    whole_disk: 1
                    DTL: 261
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a65a9027-part1'
                    devid: 'scsi-35000c500a65a9027-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a65a9025-lun-0'
                children[17]:
                    type: 'disk'
                    id: 17
                    guid: 12656340520652708264
                    whole_disk: 1
                    DTL: 260
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a65a02af-part1'
                    devid: 'scsi-35000c500a65a02af-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a65a02ad-lun-0'
                children[18]:
                    type: 'disk'
                    id: 18
                    guid: 3158634311724628854
                    whole_disk: 1
                    DTL: 259
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a670b213-part1'
                    devid: 'scsi-35000c500a670b213-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a670b211-lun-0'
                children[19]:
                    type: 'disk'
                    id: 19
                    guid: 7462459450120653037
                    whole_disk: 1
                    DTL: 258
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a670b157-part1'
                    devid: 'scsi-35000c500a670b157-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a670b155-lun-0'
                children[20]:
                    type: 'disk'
                    id: 20
                    guid: 3278057294213455851
                    whole_disk: 1
                    DTL: 257
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a665b1b3-part1'
                    devid: 'scsi-35000c500a665b1b3-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a665b1b1-lun-0'
                children[21]:
                    type: 'disk'
                    id: 21
                    guid: 10749955888611927037
                    whole_disk: 1
                    DTL: 256
                    create_txg: 4
                    path: '/dev/disk/by-id/scsi-35000c500a670c5fb-part1'
                    devid: 'scsi-35000c500a670c5fb-part1'
                    phys_path: 'pci-0000:b3:00.0-sas-0x5000c500a670c5f9-lun-0'
        rewind-policy:
            rewind-request-txg: 18446744073709551615
            rewind-request: 2
zdb: can't open 'ost_61': Input/output error
ZFS_DBGMSG(zdb):
rincebrain commented 5 years ago

Dupe of #7184 maybe?

behlendorf commented 5 years ago

It's hard to say for certain, but yes this could definitely be a duplicate of#7184 and should be resolved in master by the improvements to zpool import.

samuelxhu commented 5 years ago

@homerl Recently I've got a pool not being able to import, and i suspect the case may be similar to your case. The pool reported Input/output error but I have no idea which disk is the culprit.

I am curious that by running which command you get to know the disk sddi is exactly the troubling disk? what is the command you run to get the following messages?

zdb -e ost_61 -d -t 1274825 Dataset mos [META], ID 0, cr_txg 4, 492M, 284 objects Dataset ost_61/ost_61 [ZPL], ID 155, cr_txg 1744, 59.3T, 662232 objects Dataset ost_61 [ZPL], ID 51, cr_txg 1, 68.1K, 6 objects Verified large_blocks feature refcount of 1 is correct Verified large_dnode feature refcount of 0 is correct Verified sha512 feature refcount of 0 is correct Verified skein feature refcount of 0 is correct Verified edonr feature refcount of 0 is correct Verified userobj_accounting feature refcount of 2 is correct ----sddh--finish----- ----sddi----- zdb -e ost_61 -d -t 1414018 zdb: can't open 'ost_61': Input/output error zdb -e ost_61 -d -t 1414019 zdb: can't open 'ost_61': Input/output error zdb -e ost_61 -d -t 1414020

homerl commented 5 years ago

@samuelxhu Thank you, Sorry, the resilvering has finished. because it 's 10TB HDD. I just save 0~100M form the issue HDD. Next time if the issue come back, Could I dd some important part from the issue HDD? I'm not sure whether there is a way to take out the important parts of the issue HDD.

homerl commented 5 years ago

@samuelxhu Sorry, I reply wrong. You could get it from zdb man page

 -t transaction
         Specify the highest transaction to use when searching for uberblocks.  See also the -u and -l options for a means to see the
         available uberblocks and their associated transaction numbers.
stale[bot] commented 4 years ago

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.