search

apache / cloudstack

Apache CloudStack is an opensource Infrastructure as a Service (IaaS) cloud computing platform
https://cloudstack.apache.org/
Apache License 2.0
2.05k stars 1.1k forks source link

Migrate Volume disk offering change when Replace disk offering does not selected #4936

Closed leolleeooleo closed 3 years ago

leolleeooleo commented 3 years ago
ISSUE TYPE
COMPONENT NAME
API, UI
CLOUDSTACK VERSION
CloudStack 4.15.0 LTS
CONFIGURATION

http://docs.cloudstack.apache.org/en/4.14.0.0/adminguide/storage.html#id2

OS / ENVIRONMENT

CentOS 7.9 (2009)

SUMMARY

Error while migrating VM, (error: Unsafe migration: Migration may lead to data corruption if disks use cache != none) Because VM's root disk offering changed when Migrate Volume does not selecte "Replace disk offering"

STEPS TO REPRODUCE
  1. Select a root Volume in the Volumes list and click "Migrate Volume"
  2. Select a Primary Storage and does not select "Replace disk offering?"
  3. The "serviceofferingid" of Volume change from Volume's "Disk Offering" to VM's "Compute Offering"
EXPECTED RESULTS
Do not change "Disk Offering" when Migrate Volume while "Replace disk offering?" does not selected
ACTUAL RESULTS
"Disk Offering" (serviceofferingdisplaytext, serviceofferingid, serviceofferingname)
changed to VM's "Compute Offering"
DaanHoogland commented 3 years ago

@leolleeooleo I do not understand your issue. when creating a VM from the UI the disk offering for the root disk is that service offering, only the optional data disk will have a different offering. Can you specify how you created the VM, please?

leolleeooleo commented 3 years ago

I test this issue again. It does not happen with "Migrate Volume". It is happened by "Scale VM".

Steps:

  1. Create VM
  2. Migrate instance to another host (Works fine)
  3. Scale VM to any Compute Offering
  4. Start Instance
  5. Migrate instance to another host (Error) Migrating test-migrate (test-migrate) Exception during migrate: org.libvirt.LibvirtException: Unsafe migration: Migration may lead to data corruption if disks use cache != none or cache != directsync

Before Scale VM The root disk Compute Offering was 20GB before

After Scale VM The root disk Compute Offering change to 2C4G after

Migrate instance Error error

Before Scale VM

(localcloud) cloudstack > list virtualmachines id=644205e5-5a01-4a62-afac-6f03278e9886
{
  "count": 1,
  "virtualmachine": [
    {
      "account": "D10307009",
      "affinitygroup": [],
      "cpunumber": 2,
      "cpuspeed": 2000,
      "cpuused": "4.43%",
      "created": "2021-04-23T23:12:52+0800",
      "details": {
        "Message.ReservedCapacityFreed.Flag": "false",
        "cpuOvercommitRatio": "32",
        "memoryOvercommitRatio": "1.0"
      },
      "diskioread": 61817,
      "diskiowrite": 29704,
      "diskkbsread": 2835343,
      "diskkbswrite": 3974206,
      "diskofferingid": "ffe2070a-8c0b-4a66-9687-cbe70346fb6d",
      "diskofferingname": "20 GB",
      "displayname": "test-migrate",
      "displayvm": true,
      "domain": "ROOT",
      "domainid": "0805834e-d677-11ea-9082-ae75c86dd51b",
      "guestosid": "78c0272c-d677-11ea-9082-ae75c86dd51b",
      "haenable": false,
      "hypervisor": "KVM",
      "id": "644205e5-5a01-4a62-afac-6f03278e9886",
      "instancename": "i-4-174-VM",
      "isdynamicallyscalable": false,
      "memory": 2048,
      "memoryintfreekbs": 726916,
      "memorykbs": 2097152,
      "memorytargetkbs": 2097152,
      "name": "test-migrate",
      "networkkbsread": 1762540,
      "networkkbswrite": 13636,
      "nic": [
        {
          "extradhcpoption": [],
          "gateway": "10.1.1.1",
          "id": "786197c8-08f2-4833-8c83-0c8453e1208d",
          "ipaddress": "10.1.1.202",
          "isdefault": true,
          "macaddress": "02:00:04:1d:00:25",
          "netmask": "255.255.255.0",
          "networkid": "2adaf3cc-b972-4aa6-9cfe-3eba40af3dcc",
          "networkname": "D10307009-net",
          "secondaryip": [],
          "traffictype": "Guest",
          "type": "Isolated"
        }
      ],
      "osdisplayname": "CentOS 8",
      "ostypeid": "78c0272c-d677-11ea-9082-ae75c86dd51b",
      "passwordenabled": false,
      "rootdeviceid": 0,
      "rootdevicetype": "ROOT",
      "securitygroup": [],
      "serviceofferingid": "5bb789af-886a-4609-b57d-de65de822c49",
      "serviceofferingname": "2C2G",
      "state": "Stopped",
      "tags": [],
      "templatedisplaytext": "CentOS 8 cckuo",
      "templateid": "99239f20-2562-4bf6-83bb-ff443e223138",
      "templatename": "CentOS 8",
      "userid": "fb425cb8-c816-43b1-9727-9ef22ab60904",
      "username": "D10307009",
      "zoneid": "80d5cccc-fabb-425b-89ff-ebb023da3d5d",
      "zonename": "T2-421"
    }
  ]
}
(localcloud) cloudstack > list volumes id=2618ddb8-2c43-4c10-bc95-b9c5f9cd15e4
{
  "count": 1,
  "volume": [
    {
      "account": "D10307009",
      "clusterid": "a27d3806-48fa-4952-9545-f0a2a035d8a5",
      "clustername": "Dell PowerEdge R410",
      "created": "2021-04-24T00:38:30+0800",
      "destroyed": false,
      "deviceid": 0,
      "displayvolume": true,
      "domain": "ROOT",
      "domainid": "0805834e-d677-11ea-9082-ae75c86dd51b",
      "hypervisor": "KVM",
      "id": "2618ddb8-2c43-4c10-bc95-b9c5f9cd15e4",
      "isextractable": true,
      "name": "ROOT-174",
      "path": "f507992c-0cbe-4edf-ba9c-cc66aeea586c",
      "podid": "d3c4b2e6-2915-47be-a58d-424b55e56eee",
      "podname": "T2-421-1",
      "provisioningtype": "thin",
      "quiescevm": false,
      "serviceofferingdisplaytext": "20 GB",
      "serviceofferingid": "ffe2070a-8c0b-4a66-9687-cbe70346fb6d",
      "serviceofferingname": "20 GB",
      "size": 21474836480,
      "state": "Ready",
      "storage": "Volume01",
      "storageid": "7b3cb8ef-2fb7-3209-ace8-e9c7e7282489",
      "storagetype": "shared",
      "tags": [],
      "type": "ROOT",
      "virtualmachineid": "644205e5-5a01-4a62-afac-6f03278e9886",
      "vmdisplayname": "test-migrate",
      "vmname": "test-migrate",
      "vmstate": "Stopped",
      "zoneid": "80d5cccc-fabb-425b-89ff-ebb023da3d5d",
      "zonename": "T2-421"
    }
  ]
}

After Scale VM

(localcloud) cloudstack > list volumes id=2618ddb8-2c43-4c10-bc95-b9c5f9cd15e4
{
  "count": 1,
  "volume": [
    {
      "account": "D10307009",
      "clusterid": "a27d3806-48fa-4952-9545-f0a2a035d8a5",
      "clustername": "Dell PowerEdge R410",
      "created": "2021-04-24T00:38:30+0800",
      "destroyed": false,
      "deviceid": 0,
      "displayvolume": true,
      "domain": "ROOT",
      "domainid": "0805834e-d677-11ea-9082-ae75c86dd51b",
      "hypervisor": "KVM",
      "id": "2618ddb8-2c43-4c10-bc95-b9c5f9cd15e4",
      "isextractable": true,
      "name": "ROOT-174",
      "path": "f507992c-0cbe-4edf-ba9c-cc66aeea586c",
      "podid": "d3c4b2e6-2915-47be-a58d-424b55e56eee",
      "podname": "T2-421-1",
      "provisioningtype": "thin",
      "quiescevm": false,
      "serviceofferingdisplaytext": "2C4G",
      "serviceofferingid": "589766b7-ce4d-4baf-875c-9f6d1af7dba6",
      "serviceofferingname": "2C4G",
      "size": 21474836480,
      "state": "Ready",
      "storage": "Volume01",
      "storageid": "7b3cb8ef-2fb7-3209-ace8-e9c7e7282489",
      "storagetype": "shared",
      "tags": [],
      "type": "ROOT",
      "virtualmachineid": "644205e5-5a01-4a62-afac-6f03278e9886",
      "vmdisplayname": "test-migrate",
      "vmname": "test-migrate",
      "vmstate": "Stopped",
      "zoneid": "80d5cccc-fabb-425b-89ff-ebb023da3d5d",
      "zonename": "T2-421"
    }
  ]
}
(localcloud) cloudstack > list serviceofferings 
{
  "count": 13,
  "serviceoffering": [
    {
      "cpunumber": 1,
      "cpuspeed": 500,
      "created": "2020-08-05T01:25:44+0800",
      "defaultuse": false,
      "displaytext": "Small Instance",
      "id": "a50d423e-1ca3-4daf-a764-d61df4f9a2b5",
      "iscustomized": false,
      "issystem": false,
      "isvolatile": false,
      "limitcpuuse": false,
      "memory": 512,
      "name": "Small Instance",
      "offerha": false,
      "provisioningtype": "thin",
      "rootdisksize": 0,
      "storagetype": "shared"
    },
    {
      "cpunumber": 1,
      "cpuspeed": 1000,
      "created": "2020-08-05T01:25:44+0800",
      "defaultuse": false,
      "displaytext": "Medium Instance",
      "id": "48ddc666-3a67-4593-b435-9fb9f76d3991",
      "iscustomized": false,
      "issystem": false,
      "isvolatile": false,
      "limitcpuuse": false,
      "memory": 1024,
      "name": "Medium Instance",
      "offerha": false,
      "provisioningtype": "thin",
      "rootdisksize": 0,
      "storagetype": "shared"
    },
    {
      "cacheMode": "writeback",
      "cpunumber": 2,
      "cpuspeed": 2000,
      "created": "2020-08-06T22:11:21+0800",
      "defaultuse": false,
      "displaytext": "2C2G",
      "id": "5bb789af-886a-4609-b57d-de65de822c49",
      "iscustomized": false,
      "issystem": false,
      "isvolatile": false,
      "limitcpuuse": false,
      "memory": 2048,
      "name": "2C2G",
      "offerha": false,
      "provisioningtype": "thin",
      "rootdisksize": 0,
      "storagetype": "shared"
    },
    {
      "cacheMode": "writeback",
      "cpunumber": 2,
      "cpuspeed": 2000,
      "created": "2020-08-06T22:13:20+0800",
      "defaultuse": false,
      "displaytext": "2C4G",
      "id": "589766b7-ce4d-4baf-875c-9f6d1af7dba6",
      "iscustomized": false,
      "issystem": false,
      "isvolatile": false,
      "limitcpuuse": false,
      "memory": 4096,
      "name": "2C4G",
      "offerha": false,
      "provisioningtype": "thin",
      "rootdisksize": 0,
      "storagetype": "shared"
    },
(localcloud) cloudstack > list diskofferings 
{
  "count": 5,
  "diskoffering": [
    {
      "created": "2020-08-05T01:25:44+0800",
      "disksize": 5,
      "displayoffering": true,
      "displaytext": "5 GB",
      "id": "0c1afda9-a17e-4b1a-a7ea-ea87d85e7015",
      "iscustomized": false,
      "name": "5 GB",
      "provisioningtype": "thin",
      "storagetype": "shared"
    },
    {
      "created": "2020-08-05T01:25:44+0800",
      "disksize": 20,
      "displayoffering": true,
      "displaytext": "20 GB",
      "id": "ffe2070a-8c0b-4a66-9687-cbe70346fb6d",
      "iscustomized": false,
      "name": "20 GB",
      "provisioningtype": "thin",
      "storagetype": "shared"
    },
    {
      "created": "2020-08-05T01:25:44+0800",
      "disksize": 100,
      "displayoffering": true,
      "displaytext": "100 GB",
      "id": "1a9cbe1e-a824-405b-8f79-16796b781fc3",
      "iscustomized": false,
      "name": "100 GB",
      "provisioningtype": "thin",
      "storagetype": "shared"
    },
    {
      "cacheMode": "none",
      "created": "2021-01-14T22:21:58+0800",
      "disksize": 500,
      "displayoffering": true,
      "displaytext": "500 GB",
      "id": "fa9ace64-4d92-4a02-b124-a3239713d377",
      "iscustomized": false,
      "name": "500 GB",
      "provisioningtype": "thin",
      "storagetype": "shared"
    },
    {
      "created": "2020-08-05T01:25:44+0800",
      "disksize": 0,
      "displayoffering": true,
      "displaytext": "Custom Disk",
      "id": "0dc9045c-12ba-464d-be6c-fecc9feda9dc",
      "iscustomized": true,
      "name": "Custom",
      "provisioningtype": "thin",
      "storagetype": "shared"
    }
  ]
}
rohityadavcloud commented 3 years ago

@leolleeooleo is the issue that the offering of the volume was changed during migration even though it wasn't asked?

rohityadavcloud commented 3 years ago

cc @shwstppr @Spaceman1984 @Pearl1594 @davidjumani - anyone with bandwidth want to pick this?

Pearl1594 commented 3 years ago

@leolleeooleo cc @rhtyd It seems like the issue occurs because during scaling the new compute offering chosen has a cache mode set to writeback. Migrating a VM with this cache mode is restricted from libvirt / qemu end as mentioned in https://libvirt.org/manpages/virsh.html. Following is the excerpt from the aforementioned site:

In some cases libvirt may refuse to migrate the domain because doing so may lead to potential problems such as data corruption, and thus the migration is considered unsafe. For QEMU domain, this may happen if the domain uses disks without explicitly setting cache mode to "none". Migrating such domains is unsafe unless the disk images are stored on coherent clustered filesystem, such as GFS2 or GPFS. If you are sure the migration is safe or you just do not care, use --unsafe to force the migration.

That said, we may want to discuss whether we would want to force the migration anyway by passing the '--unsafe' flag / handle it in some other way.

leolleeooleo commented 3 years ago

The issue I think is that the root disk offering is not service offering when I create VM.

@leolleeooleo is the issue that the offering of the volume was changed during migration even though it wasn't asked?

Offering of the volume was changed when I migrated the volume from NFS primary storage to another NFS primary storage. But I destroy one of my NFS primary storage and create CEPH primary storage last week. It doesn't change when I migrate the volume from NFS primary storage to CEPH primary storage.

@Pearl1594 Yes, compute offering cache mode is set to writeback. I was confused why I can migrate VM before scale VM (or before migrate root disk).

I set compute offering cache mode to none now.

rohityadavcloud commented 3 years ago

Based on your remarks, it does not look like an issue. Can you confirm and close this @leolleeooleo