search

openebs-archive / node-disk-manager

Kubernetes Storage Device Management
https://openebs.io/docs
Apache License 2.0
182 stars 113 forks source link

BlockDevice name not unique, same ssd disk generate dulplicate BlockDevices #660

Open Icedroid opened 2 years ago

Icedroid commented 2 years ago

What steps did you take and what happened: [A clear and concise description of what the bug is, and the sequence of operations performed / commands you ran.]

When the BlockDevice Claimed, I run into container use the bd in LVM filesystem. Then after upgarde node-disk-manager to 0.7.2 from 0.7.1. I found dulplicate BlockDevices generated.

What did you expect to happen: Same nvme disk only one bd.

The output of the following commands will help us better understand what's going on: [Pasting long output into a GitHub gist or other pastebin is fine.]

*kubectl get bd blockdevice-9996981c2155727b2271f51ad74de685 -n openebs -o yaml

apiVersion: openebs.io/v1alpha1
kind: BlockDevice
metadata:
  creationTimestamp: "2021-11-29T06:37:22Z"
  generation: 7
  labels:
    kubernetes.io/hostname: 192.168.189.42
    ndm.io/blockdevice-type: blockdevice
    ndm.io/managed: "true"
  name: blockdevice-9996981c2155727b2271f51ad74de685
  namespace: openebs
  resourceVersion: "1154072554"
  selfLink: /apis/openebs.io/v1alpha1/namespaces/openebs/blockdevices/blockdevice-9996981c2155727b2271f51ad74de685
  uid: 6d8345c6-738e-4ac2-8db7-7ba29243c8c7
spec:
  capacity:
    logicalSectorSize: 512
    physicalSectorSize: 512
    storage: 2048408248320
  details:
    compliance: ""
    deviceType: disk
    driveType: SSD
    firmwareRevision: ""
    hardwareSectorSize: 512
    logicalBlockSize: 512
    model: HS-SSD-C2000Pro 2048G
    physicalBlockSize: 512
    serial: "30036236827"
    vendor: ""
  devlinks:
  - kind: by-id
    links:
    - /dev/disk/by-id/nvme-HS-SSD-C2000Pro_2048G_30036236827
    - /dev/disk/by-id/nvme-nvme.126f-3330303336323336383237-48532d5353442d433230303050726f203230343847-00000001
  - kind: by-path
    links:
    - /dev/disk/by-path/pci-0000:03:00.0-nvme-1
  filesystem:
    fsType: xfs
  nodeAttributes:
    nodeName: 192.168.189.42
  partitioned: "No"
  path: /dev/nvme0n1
status:
  claimState: Unclaimed
  state: Active

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

akhilerm commented 2 years ago

Can you share the logs from the the NDM pod and also the output of lslbk from the node. Also when was the disk formatted with xfs filesystem?

Icedroid commented 2 years ago

@akhilerm I found the bug in these code, when ID_TYPE is empty and the k8s node hostname changed, the bd name will change.

image

akhilerm commented 2 years ago

This is the old algorithm that we use to generate UUIDs. Currently there is a GPTBasedUUID feature flag which is enabled by default, which causes a different UUID algorithm to be used. This GPT based algorithm does not have the issue of the UUID getting changed when node name gets changed.

Icedroid commented 2 years ago

@akhilerm For many online customer reason, now we couldn't upgrade NDM Feauture: GPTBasedUUID. We plan to think about upgrade smoothly. The ndm new version 1.7.2 compatible with ndm 0..7.2 and openebs/provisioner-localpv:2.12.0 and openebs/m-apiserver:1.12.0?

Icedroid commented 2 years ago

@akhilerm Ask for help. we only use device LocalPV, If only upgrade ndm to new version v1.7.2. But openebs keep v1.7.2. will it work?

akhilerm commented 2 years ago

Do you mean keeping NDM version at 1.7.2 and localpv version at 2.12.0? Yes it will work. Also from your above comments, it was mentioned about using m-apiserver, which is not required to run localpv provisioner. You can just have localpv provisioner 2.12.0 and NDM 1.7.2.

Also a few more questions before you perform the upgrade:

To upgrade :

Icedroid commented 2 years ago

Today I upgrade our develop env to NDM 1.7.0 and localpv privisioner 3.0.0. I found all host disks dumplicate. Before I Upgrade, now we focus on 192.168.88.103 node. all-bds

We have a kubevirt virtual machine run in 192.168.88.103 node and mount the node device /dev/sda /dev/sdb for Block mode PVC. In the vm we parted sda sdb and mount to the filesystem.

Now I upgrade NDM and localpv.

After upgrade, all BDs be dumplicate and change to unknow as follow:



> * are localpv device  PVCs in Block mode or FileSystem mode. Because the upgrade is supported only for FileSystem mode PVCs.

We use BDs Of Block mode PVC for kubevirt virutal machine disk devices

I found upgrade not for our production schema. If you have idea . Thanksgiving to you.
Icedroid commented 2 years ago

@akhilerm I have a idea. If I make it PR for implement lke this, could be meet my needs:

akhilerm commented 2 years ago

We use BDs Of Block mode PVC for kubevirt virutal machine disk devices

Unfortunately we do not support upgrade for local PV using BlockMode, from old UUID to the new GPTScheme,

@Icedroid This is the design that we followed for generating the UUID.

Before BD UUID be generated, I get all bd list to checkout if already have same WWN in the node. If exists, I use old BD UUID. If not exists, generate the new BD UUID.

There are some issues with this approach:

Icedroid commented 2 years ago

@akhilerm For physical disk unique WWN and model assigned by the manufacturer. So attaching to a different machine or a different SCSI port of the same machine or different node all keep same. For us, we only use bd for pyhical disk.

  • What if there are more than 2 disks, which have this issue. The first disk gets UUID based on new algorithm, the 2nd and 3rd disk gets UUID based on old algorithm. This will have chances of collision again.'
  • Suppose the node / pod restarts, so how will I identify both the disks. since they used the old UUID. i.e diskA has UUID1 and diskB has UUID2, instead of the other way around.

I compare WWN for 2 disk , not use bd UUID1 and UUID2. Cloud change md5 hash to uuid algorithm?

  • Consider the case where one of the disk is unplugged from nodeA and plugged into nodeB, how will it behave in this case?

physical disk has unique WWN.

Icedroid commented 2 years ago

@akhilerm The new version of NDM not support NVME disk scan for BlockDevice ? If fail log :

smartprobe.go:101] map[errorCheckingConditions:the device type is not supported yet, device type: "NVMe"]
akhilerm commented 2 years ago

The new version of NDM not support NVME disk scan for BlockDevice ? If fail log :

It does support @Icedroid . its an issue with SMART not able to detect some NVMe details. It can be ignored.