search

LINBIT / linstor-server

High Performance Software-Defined Block Storage for container, cloud and virtualisation. Fully integrated with Docker, Kubernetes, Openstack, Proxmox etc.
https://docs.linbit.com/docs/linstor-guide/
GNU General Public License v3.0
954 stars 76 forks source link

New resource is marked as UpToDate without data synchronization #145

Closed mrakopes closed 2 years ago

mrakopes commented 4 years ago

Hi. I've just created new resource for my volume, which had one diskfull resource and one diskless resource before. Just after resource was created, it was marked as UpToDate but it was not synced with already existing data resource:

before:

linstor r lv -r pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656
+-------------------------------------------------------------------------------------------------------------------------------------------+
| Node  | Resource                                 | StoragePool          | VolNr | MinorNr | DeviceName    | Allocated | InUse  |    State |
|===========================================================================================================================================|
| m7c2  | pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656 | thindata             |     0 |    1025 | /dev/drbd1025 | 31.86 GiB | Unused | UpToDate |
| m9c25 | pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656 | DfltDisklessStorPool |     0 |    1025 | /dev/drbd1025 |           | InUse  | Diskless |
+-------------------------------------------------------------------------------------------------------------------------------------------+

created new resource:

# linstor r c -s thindata m12c1 pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656
SUCCESS:
    Successfully set property key(s): StorPoolName
INFO:
    Resource-definition property 'DrbdOptions/Resource/quorum' updated from 'off' to 'majority' by auto-quorum
INFO:
    Resource-definition property 'DrbdOptions/Resource/on-no-quorum' updated from 'off' to 'io-error' by auto-quorum
SUCCESS:
Description:
    New resource 'pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656' on node 'm12c1' registered.
Details:
    Resource 'pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656' on node 'm12c1' UUID is: 6400edcd-357b-45a5-9f0d-389eccc4585b
SUCCESS:
Description:
    Volume with number '0' on resource 'pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656' on node 'm12c1' successfully registered
Details:
    Volume UUID is: 6c26552b-7844-44fe-91d7-da9cc001d112
SUCCESS:
    Added peer(s) 'm12c1' to resource 'pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656' on 'm9c25'
SUCCESS:
    Added peer(s) 'm12c1' to resource 'pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656' on 'm7c2'
SUCCESS:
    Created resource 'pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656' on 'm12c1'
SUCCESS:
Description:
    Resource 'pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656' on 'm12c1' ready
Details:
    Node(s): 'm12c1', Resource: 'pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656'

State right after creation (the new resource is already UpToDate:

# linstor r lv -r pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656 -a
+--------------------------------------------------------------------------------------------------------------------------------------------+
| Node  | Resource                                 | StoragePool          | VolNr | MinorNr | DeviceName    |  Allocated | InUse  |    State |
|============================================================================================================================================|
| m12c1 | pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656 | thindata             |     0 |    1025 | /dev/drbd1025 | 143.39 MiB | Unused | UpToDate |
| m7c2  | pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656 | thindata             |     0 |    1025 | /dev/drbd1025 |  31.86 GiB | Unused | UpToDate |
| m9c25 | pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656 | DfltDisklessStorPool |     0 |    1025 | /dev/drbd1025 |            | InUse  | Diskless |
+--------------------------------------------------------------------------------------------------------------------------------------------+

drbd status on node with new resource:

m12c1# drbdsetup -s status pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656
pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656 role:Secondary
  disk:UpToDate
      read:0 written:496
  m7c2 role:Secondary
    peer-disk:UpToDate
        received:0 sent:0
  m9c25 role:Primary
    peer-disk:Diskless
        received:496 sent:0

Then i verified resource state and it was in fact out-of-sync:

m12c1# drbdadm verify pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656
(..)

m12c1# drbdsetup -s status pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656
pvc-9414168a-a8ce-4c2a-b836-8fa1471ea656 role:Secondary
  disk:UpToDate
      read:3257088 written:652
  m7c2 role:Secondary
    replication:VerifyS peer-disk:UpToDate done:0.84
        received:0 sent:0 out-of-sync:2947336
  m9c25 role:Primary
    peer-disk:Diskless
        received:652 sent:0

kernel log from node with new resource during creation: m12c1-drbd.txt

kernel log from node with drbd diskfull resource already present: m7c2-drbd.txt

kernel log from node with diskless resource: m9c25-drbd.txt

# linstor --version
linstor 1.1.0; GIT-hash: 2397766389b01ca535ebac0cc58d90be15fc655d

# drbdadm -V
DRBDADM_BUILDTAG=GIT-hash:\ 63092751e76e1fba397e53df4be5c1161b83a223\ build\ by\ buildd@lgw01-amd64-038\,\ 2019-10-16\ 11:17:20
DRBDADM_API_VERSION=2
DRBD_KERNEL_VERSION_CODE=0x090015
DRBD_KERNEL_VERSION=9.0.21
DRBDADM_VERSION_CODE=0x090b00
DRBDADM_VERSION=9.11.0

I tried the same operation later with different volume, this time i've unmounted the volume and remove diskless replica before creating new data resource. Result was the same - Imediately marked as UpToDate - this time there was 300+ GB of used space so it should took some time to sync.

# linstor r lv -r pvc-b43abdf4-5de1-4a0c-a175-94fadfaf1b55
+-----------------------------------------------------------------------------------------------------------------------------------+
| Node  | Resource                                 | StoragePool | VolNr | MinorNr | DeviceName    |  Allocated | InUse  |    State |
|===================================================================================================================================|
| m12c6 | pvc-b43abdf4-5de1-4a0c-a175-94fadfaf1b55 | thindata    |     0 |    3245 | /dev/drbd3245 | 143.39 MiB | Unused | UpToDate |  <-- new resource
| m7c7  | pvc-b43abdf4-5de1-4a0c-a175-94fadfaf1b55 | thindata    |     0 |    3245 | /dev/drbd3245 | 341.46 GiB | Unused | UpToDate |  <-- old resource
+-----------------------------------------------------------------------------------------------------------------------------------+

after drbdadm verify:

drbdsetup -s status pvc-b43abdf4-5de1-4a0c-a175-94fadfaf1b55
pvc-b43abdf4-5de1-4a0c-a175-94fadfaf1b55 role:Secondary
  disk:UpToDate
      read:734006072 written:0
  m7c7 role:Secondary
    peer-disk:UpToDate
        received:0 sent:0 out-of-sync:342406228
ghernadi commented 4 years ago

Hi, sorry for not responding here. If I am not mistaken this was a bug in DRBD, but should have been fixed with the recent release.

Can you please upgrade DRBD and see if this is fixed for you?