No news disks are added to the LVMCluster/LVMvolumegroupnodestatuses

[jgato]

I had one SNO with an LVMCluster created to manage the VG vg1 (/dev/nvme0n1, /dev/nvme1n1, /dev/sda). I wanted to test the addition of new disks. So, I rebooted and created some more disks from the raid I have in the server.

After creating some more disks the SNO is rebooted and:

• The new disks (/dev/sdc, /dev/sdd) have been added to the vg1 in the node:

  
  [root@master-0 core]# vgs
  VG  #PV #LV #SN Attr   VSize VFree 
  vg1   5   4   0 wz--n- 6.00t <6.00t
  [root@master-0 core]# pvs
  PV           VG  Fmt  Attr PSize   PFree  
  /dev/nvme0n1 vg1 lvm2 a--  745.21g 745.21g
  /dev/nvme1n1 vg1 lvm2 a--  745.21g 745.21g
  /dev/sda     vg1 lvm2 a--   <2.73t   2.72t
  /dev/sdc     vg1 lvm2 a--  931.48g 931.48g
  /dev/sde     vg1 lvm2 a--  931.48g 931.48g

 * This new disks are not recognized by the LVMCluster/LVMvolumegroupnodestatuses:

$ oc get lvmvolumegroupnodestatuses -o yaml apiVersion: v1 items:

• apiVersion: lvm.topolvm.io/v1alpha1 kind: LVMVolumeGroupNodeStatus metadata: creationTimestamp: "2022-02-05T18:58:01Z" generation: 1 name: master-0.apollo2.hpecloud.org namespace: lvm-operator-system resourceVersion: "6276721" uid: b6cc0395-e17c-4116-941f-9b95b8c4ed82 spec: nodeStatus:
  • devices:
    • /dev/nvme0n1
    • /dev/nvme1n1
    • /dev/sda name: vg1 status: Ready kind: List metadata: resourceVersion: "" selfLink: ""

      $ oc get lvmcluster lvmcluster-sample -o yaml apiVersion: lvm.topolvm.io/v1alpha1 kind: LVMCluster metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"lvm.topolvm.io/v1alpha1","kind":"LVMCluster","metadata":{"annotations":{},"name":"lvmcluster-sample","namespace":"lvm-operator-system"},"spec":{"deviceClasses":[{"name":"vg1"}]}} creationTimestamp: "2022-02-05T18:56:54Z" finalizers:

  • lvmcluster.topolvm.io generation: 1 name: lvmcluster-sample namespace: lvm-operator-system resourceVersion: "6276722" uid: fb91728e-1300-4fe1-9041-78d5a8e166d6 spec: deviceClasses:
  • name: vg1 status: deviceClassStatuses:
  • name: vg1 nodeStatus:
  • devices:
    • /dev/nvme0n1
    • /dev/nvme1n1
    • /dev/sda node: master-0.apollo2.hpecloud.org status: Ready ready: true

I have created a second LVMCluster because I thought this would collect the new disks. Now I understand it is only one LVMCluster supported, would this have interfere?  

[nbalacha]

I have created a second LVMCluster because I thought this would collect the new disks. Now I understand it is only one LVMCluster supported, would this have interfere?

It should not. Do you see any errors in the controller-manager or vgmanager logs files?

[jgato]

From the controller-manager:

{
  "level": "error",
  "ts": 1644229480.7411144,
  "logger": "controller.lvmcluster.lvmcluster-controller",
  "msg": "failed to create or update vgManager daemonset",
  "reconciler group": "lvm.topolvm.io",
  "reconciler kind": "LVMCluster",
  "name": "vg-manager",
  "namespace": "lvm-operator-system",
  "error": "failed to update controller reference on vgManager daemonset \"vg-manager\". Object lvm-operator-system/vg-manager is already owned by another LVMCluster controller lvmcluster-sample",
  "stacktrace": "github.com/red-hat-storage/lvm-operator/controllers.(*LVMClusterReconciler).Reconcile\n\t/workspace/controllers/lvmcluster_controller.go:104\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:227"
}
{
  "level": "error",
  "ts": 1644229480.7504883,
  "logger": "controller.lvmcluster",
  "msg": "Reconciler error",
  "reconciler group": "lvm.topolvm.io",
  "reconciler kind": "LVMCluster",
  "name": "lvmcluster-sample-2",
  "namespace": "lvm-operator-system",
  "error": "failed reconciling: vg-manager failed to update controller reference on vgManager daemonset \"vg-manager\". Object lvm-operator-system/vg-manager is already owned by another LVMCluster controller lvmcluster-sample",
  "stacktrace": "sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:227"
}

So it could be the second LVMCCluster creation could be causing problems. Do you want me to delete it?

No errors in the vgmanager.

[nbalacha]

So it could be the second LVMCCluster creation could be causing problems. Do you want me to delete it?

Yes, please delete the second LVMCluster.

[jgato]

Ok, I have cleaned up everything. The second LVMCluster was causing problems and it is something not supported yet. I have repeated the process: Phase 1)

• Everything in a clean stage
• Operator deployed an LVMCluster created.
• it detects all the available disks and the VG/PV are created correctly

  Device Class Statuses:
  Name:  vg1
  Node Status:
    Devices:
      /dev/nvme0n1
      /dev/nvme1n1
      /dev/sda
      /dev/sdc
      /dev/sdd
      /dev/sdg
    Node:    master-0.apollo2.hpecloud.org

  [root@master-0 core]# vgs                                                                                                                                                                                                                                     
  VG  #PV #LV #SN Attr   VSize VFree                                                                                                                                                                                                                          
  vg1   6   6   0 wz--n- 6.91t 6.90t       
  [root@master-0 core]# pvs                                                                                                                                                                                                                                     
  PV           VG  Fmt  Attr PSize   PFree                                                                                                                                                                                                                    
  /dev/nvme0n1 vg1 lvm2 a--  745.21g 745.21g                                                                                                                                                                                                                  
  /dev/nvme1n1 vg1 lvm2 a--  745.21g 745.21g                                                                                                                                                                                                                  
  /dev/sda     vg1 lvm2 a--   <2.73t  <2.72t                                                                                                                                                                                                                  
  /dev/sdc     vg1 lvm2 a--  931.48g 931.48g                                                                                                                                                                                                                  
  /dev/sdd     vg1 lvm2 a--  931.48g 931.48g                                                                                                                                                                                                                  
  /dev/sdg     vg1 lvm2 a--  931.48g 931.48g    

  Phase 2)

• Notice /dev/sde and /dev/sdf were not included initialy. This is ok, because these were not empty.
• Cleaning both disks but these are still not included into the VG
• Reboot the sno
• Now the disks have been included in the VG

  
  # pvs
  PV           VG  Fmt  Attr PSize   PFree  
  /dev/nvme0n1 vg1 lvm2 a--  745.21g 745.21g
  /dev/nvme1n1 vg1 lvm2 a--  745.21g 745.21g
  /dev/sda     vg1 lvm2 a--   <2.73t  <2.72t
  /dev/sdc     vg1 lvm2 a--  931.48g 931.48g
  /dev/sdd     vg1 lvm2 a--  931.48g 931.48g
  /dev/sde     vg1 lvm2 a--  931.48g 931.48g
  /dev/sdf     vg1 lvm2 a--  931.48g 931.48g
  /dev/sdg     vg1 lvm2 a--  931.48g 931.48g

 * But still not recognizable by the LVMCluster

Device Class Statuses: Name: vg1 Node Status: Devices: /dev/nvme0n1 /dev/nvme1n1 /dev/sda /dev/sdc /dev/sdd /dev/sdg