search

vexxhost / migratekit

Near-live migration toolkit for VMware to OpenStack
Apache License 2.0
70 stars 10 forks source link

Error: volume not found #39

Open doskelion opened 5 days ago

doskelion commented 5 days ago

For some reason I am getting this error right after removal of vmware snapshots:

INFO[0000] Data does not exist, full copy needed
INFO[0000] Creating new volume
INFO[0000] Creating new volume
INFO[0002] Volume created, setting to bootable           volume_id=43310398-a572-40d3-a585-4921c62856c8
INFO[0002] Attaching volume                              volume_id=43310398-a572-40d3-a585-4921c62856c8
Removing snapshot 100% [=======================================================================================================================================================] (100/100) [0s:0s]
Error: volume not found
Usage:
  migratekit migrate [flags]

Flags:
  -h, --help   help for migrate

Global Flags:
      --availability-zone string                Openstack availability zone for blockdevice & server
      --compression-method compression-method   Specifies the compression method to use for the disk (default skipz)
      --debug                                   Enable debug logging
      --disk-bus-type disk-bus-type             Specifies the type of disk controller to attach disk devices to. (default virtio)
      --vmware-endpoint string                  VMware endpoint (hostname or IP only)
      --vmware-password string                  VMware password
      --vmware-path string                      VMware VM path (e.g. '/Datacenter/vm/VM')
      --vmware-username string                  VMware username
      --volume-type string                      Openstack volume type

Tried both rocky8 and ubuntu 22.04 jammy with same error. am i missing something? Hints will be much appreciated. Thanks!

jmussmann commented 4 days ago

For some reason I am getting this error right after removal of vmware snapshots:

INFO[0000] Data does not exist, full copy needed
INFO[0000] Creating new volume
INFO[0000] Creating new volume
INFO[0002] Volume created, setting to bootable           volume_id=43310398-a572-40d3-a585-4921c62856c8
INFO[0002] Attaching volume                              volume_id=43310398-a572-40d3-a585-4921c62856c8
Removing snapshot 100% [=======================================================================================================================================================] (100/100) [0s:0s]
Error: volume not found
Usage:
  migratekit migrate [flags]

Flags:
  -h, --help   help for migrate

Global Flags:
      --availability-zone string                Openstack availability zone for blockdevice & server
      --compression-method compression-method   Specifies the compression method to use for the disk (default skipz)
      --debug                                   Enable debug logging
      --disk-bus-type disk-bus-type             Specifies the type of disk controller to attach disk devices to. (default virtio)
      --vmware-endpoint string                  VMware endpoint (hostname or IP only)
      --vmware-password string                  VMware password
      --vmware-path string                      VMware VM path (e.g. '/Datacenter/vm/VM')
      --vmware-username string                  VMware username
      --volume-type string                      Openstack volume type

Tried both rocky8 and ubuntu 22.04 jammy with same error. am i missing something? Hints will be much appreciated. Thanks!

Hi, can you please send the output with the "--debug" option set?

doskelion commented 4 days ago

I do have that --debug option set in place, refer screenshot below: image

Also, here is my openrc vars, not sure if i missed any variables migratekit is looking for: image

jmussmann commented 4 days ago

Has the volume been successfully created in openstack?

Does the openstack user have permission to attach volumes to the migratekit instance?

Maybe you can test if attaching the volume via the cli works.

doskelion commented 4 days ago

i tried default admin creds too, but sure will try it in cli as you suggested. will keep you posted. Thanks!

doskelion commented 4 days ago

To answer your question: Has the volume been successfully created in openstack?

Does the openstack user have permission to attach volumes to the migratekit instance?

Maybe you can test if attaching the volume via the cli works.

Are there any other way to watch logs? debug doesnt seem helpful. Any verbose option? Also I have tried to test development environment using fedora:40 as per readme and build migratekit, still the same issue. Not sure which volume is it referring to. I see migratekit was able to create a vmware snapshot and was able to create volume in openstack with 0 size. :(

Thanks.

jmussmann commented 4 days ago

I have added some more debugging prints to a test branch: https://github.com/jmussmann/migratekit/tree/issue/39

You can use my docker container: docker.io/dorfpinguin/migratekit:issue-39

Can you please also show the output of openstack volume show?

doskelion commented 4 days ago

will try the new branch, here is the volume show output as you requested:

#openstack --insecure volume show odoo-netcentral-com-ph-922-2000
+------------------------------+----------------------------------------------------+
| Field                        | Value                                              |
+------------------------------+----------------------------------------------------+
| attachments                  | []                                                 |
| availability_zone            | nova                                               |
| bootable                     | true                                               |
| consistencygroup_id          | None                                               |
| created_at                   | 2024-10-16T17:15:56.461337                         |
| description                  | None                                               |
| encrypted                    | False                                              |
| id                           | 247c5467-3e9d-4c18-988f-ea8f6faef3a9               |
| multiattach                  | False                                              |
| name                         | odoo-netcentral-com-ph-922-2000                    |
| os-vol-tenant-attr:tenant_id | 42e50f555f7e4d31a4dd2c527940beb4                   |
| properties                   | disk='922-2000', migrate_kit='true', vm='vm-37498' |
| replication_status           | None                                               |
| size                         | 40                                                 |
| snapshot_id                  | None                                               |
| source_volid                 | None                                               |
| status                       | available                                          |
| type                         | default                                            |
| updated_at                   | 2024-10-16T17:15:57.600261                         |
| user_id                      | d11d24bfe461457ca8f6cbea9a62fa68                   |
+------------------------------+----------------------------------------------------+
jmussmann commented 4 days ago

I forgot, you have to delete the volume before the test.

doskelion commented 4 days ago

Cleaned up the volume and executed the new image branch you provided: image

heres the volume show:

# openstack --insecure volume show odoo-netcentral-com-ph-922-2000
+------------------------------+----------------------------------------------------+
| Field                        | Value                                              |
+------------------------------+----------------------------------------------------+
| attachments                  | []                                                 |
| availability_zone            | nova                                               |
| bootable                     | true                                               |
| consistencygroup_id          | None                                               |
| created_at                   | 2024-10-16T17:46:10.435175                         |
| description                  | None                                               |
| encrypted                    | False                                              |
| id                           | 271d9b9d-f79b-4e2c-ae9d-ffd0fea0bbed               |
| multiattach                  | False                                              |
| name                         | odoo-netcentral-com-ph-922-2000                    |
| os-vol-tenant-attr:tenant_id | 42e50f555f7e4d31a4dd2c527940beb4                   |
| properties                   | disk='922-2000', migrate_kit='true', vm='vm-37498' |
| replication_status           | None                                               |
| size                         | 40                                                 |
| snapshot_id                  | None                                               |
| source_volid                 | None                                               |
| status                       | available                                          |
| type                         | default                                            |
| updated_at                   | 2024-10-16T17:46:11.562985                         |
| user_id                      | d11d24bfe461457ca8f6cbea9a62fa68                   |
+------------------------------+----------------------------------------------------+

I see it says "ERROR: Cannot get path" .. which path is that? Also might help, I noticed too when it creates vm snapshot, its pretty much quick and removes it. If I were to do a manual vm snapshot, it takes couple of seconds.

jmussmann commented 4 days ago

Thanks, now I know a little bit more. I have added more debugging information. Can you please pull the container and run it again?

What OS / distro are you using?

doskelion commented 4 days ago

i tried both Rocky 8 and Ubuntu 22.04. will try shortly.

doskelion commented 4 days ago

I am now testing only with Ubuntu 22.04 and removed old volumes to have a new one and this is what I've got on couple of last lines:

time="2024-10-16T19:08:43Z" level=info msg="Data does not exist, full copy needed"
time="2024-10-16T19:08:43Z" level=info msg="Creating new volume"
time="2024-10-16T19:08:43Z" level=info msg="Creating new volume"
time="2024-10-16T19:08:43Z" level=debug msg="DiskCapacityInBytes: 42949672960"
time="2024-10-16T19:08:43Z" level=debug msg="Creating new volume: {\"size\":40,\"metadata\":{\"disk\":\"922-2000\",\"migrate_kit\":\"true\",\"vm\":\"vm-37498\"},\"name\":\"odoo-netcentral-com-ph-922-2000\"}"
time="2024-10-16T19:08:44Z" level=info msg="Volume created, setting to bootable" volume_id=b1eb213d-5ee4-4dbe-aaa2-f3bbc27cbc3e
time="2024-10-16T19:08:44Z" level=info msg="Attaching volume" volume_id=b1eb213d-5ee4-4dbe-aaa2-f3bbc27cbc3e
time="2024-10-16T19:08:44Z" level=debug msg="Error, cannot find volume"
time="2024-10-16T19:08:44Z" level=debug msg="ERROR: Cannot get path"
Removing snapshot   0% [                                      ] (0/100) [0s:0s]
Removing snapshot 100% [====================================] (100/100)Error: volume not found
Usage:
  migratekit migrate [flags]

Flags:
  -h, --help   help for migrate

Global Flags:
      --availability-zone string                Openstack availability zone for blockdevice & server
      --compression-method compression-method   Specifies the compression method to use for the disk (default skipz)
      --debug                                   Enable debug logging
      --disk-bus-type disk-bus-type             Specifies the type of disk controller to attach disk devices to. (default virtio)
      --vmware-endpoint string                  VMware endpoint (hostname or IP only)
      --vmware-password string                  VMware password
      --vmware-path string                      VMware VM path (e.g. '/Datacenter/vm/VM')
      --vmware-username string                  VMware username
      --volume-type string                      Openstack volume type

Above those logs are just same. Hopefully that helps.

jmussmann commented 4 days ago

I have a new debug build. Can you please run it again?

doskelion commented 4 days ago

trying again shortly.

doskelion commented 4 days ago

as requested:

time="2024-10-16T19:43:49Z" level=debug msg="Info: Volumename: odoo-netcentral-com-ph-922-2000 VM: vm-37498, disk: 922-2000"
time="2024-10-16T19:43:49Z" level=debug msg="Error, volume list empty"
time="2024-10-16T19:43:49Z" level=info msg="Data does not exist, full copy needed"
time="2024-10-16T19:43:49Z" level=debug msg="Info: Volumename: odoo-netcentral-com-ph-922-2000 VM: vm-37498, disk: 922-2000"
time="2024-10-16T19:43:49Z" level=debug msg="Error, volume list empty"
time="2024-10-16T19:43:49Z" level=info msg="Creating new volume"
time="2024-10-16T19:43:49Z" level=info msg="Creating new volume"
time="2024-10-16T19:43:49Z" level=debug msg="DiskCapacityInBytes: 42949672960"
time="2024-10-16T19:43:49Z" level=debug msg="Creating new volume: {\"size\":40,\"metadata\":{\"disk\":\"922-2000\",\"migrate_kit\":\"true\",\"vm\":\"vm-37498\"},\"name\":\"odoo-netcentral-com-ph-922-2000\"}"
time="2024-10-16T19:43:51Z" level=info msg="Volume created, setting to bootable" volume_id=78ba1e37-e90c-4a9b-bc06-9f244ea28327
time="2024-10-16T19:43:51Z" level=info msg="Attaching volume" volume_id=78ba1e37-e90c-4a9b-bc06-9f244ea28327
time="2024-10-16T19:43:51Z" level=debug msg="Info: Volumename: odoo-netcentral-com-ph-922-2000 VM: vm-37498, disk: 922-2000"
time="2024-10-16T19:43:51Z" level=debug msg="Error, volume list empty"
time="2024-10-16T19:43:51Z" level=debug msg="Error, cannot find volume"
time="2024-10-16T19:43:51Z" level=debug msg="ERROR: Cannot get path"
Removing snapshot 100% [====================================] (100/100) [0s:0s]
Error: volume not found
Usage:
  migratekit migrate [flags]

Flags:
  -h, --help   help for migrate

Global Flags:
jmussmann commented 3 days ago

I have a new debug build. Can you please run it again?

Somehow the volume cannot be found by the name. Since we know the ID, we can also use the id to get it. It is only a test build, it might not work for incremental runs.

doskelion commented 3 days ago

roger that. let me try it again.

doskelion commented 3 days ago

well this is something duh, not sure whats your take on this:

time="2024-10-16T20:19:34Z" level=debug msg="Info: Volumename: odoo-netcentral-com-ph-922-2000 VM: vm-37498, disk: 922-2000"
time="2024-10-16T20:19:34Z" level=debug msg="Error, volume list empty"
time="2024-10-16T20:19:34Z" level=info msg="Data does not exist, full copy needed"
time="2024-10-16T20:19:34Z" level=debug msg="Info: Volumename: odoo-netcentral-com-ph-922-2000 VM: vm-37498, disk: 922-2000"
time="2024-10-16T20:19:34Z" level=debug msg="Error, volume list empty"
time="2024-10-16T20:19:34Z" level=info msg="Creating new volume"
time="2024-10-16T20:19:34Z" level=info msg="Creating new volume"
time="2024-10-16T20:19:34Z" level=debug msg="DiskCapacityInBytes: 42949672960"
time="2024-10-16T20:19:34Z" level=debug msg="Creating new volume: {\"size\":40,\"metadata\":{\"disk\":\"922-2000\",\"migrate_kit\":\"true\",\"vm\":\"vm-37498\"},\"name\":\"odoo-netcentral-com-ph-922-2000\"}"
time="2024-10-16T20:19:35Z" level=info msg="Volume created, setting to bootable" volume_id=66e2e30e-eb38-4e55-b5a1-8a5fb6d51422
time="2024-10-16T20:19:35Z" level=info msg="Attaching volume" volume_id=66e2e30e-eb38-4e55-b5a1-8a5fb6d51422
time="2024-10-16T20:19:35Z" level=debug msg="Error, cannot readdir /dev/disk/by-id/"
time="2024-10-16T20:19:35Z" level=debug msg="Error, cannot find device"
time="2024-10-16T20:19:35Z" level=debug msg="ERROR: Cannot get path"
Removing snapshot   0% [                                      ] (0/100) [0s:0s]
Removing snapshot 100% [====================================] (100/100)Error: open /dev/disk/by-id/: no such file or directory
Usage:
  migratekit migrate [flags]

Flags:
  -h, --help   help for migrate
doskelion commented 3 days ago

so temporarily created /dev/disk/by-id folder and this is the new log:

time="2024-10-16T20:23:28Z" level=debug msg="Info: Volumename: odoo-netcentral-com-ph-922-2000 VM: vm-37498, disk: 922-2000"
time="2024-10-16T20:23:28Z" level=debug msg="Error, volume list empty"
time="2024-10-16T20:23:28Z" level=info msg="Data does not exist, full copy needed"
time="2024-10-16T20:23:28Z" level=debug msg="Info: Volumename: odoo-netcentral-com-ph-922-2000 VM: vm-37498, disk: 922-2000"
time="2024-10-16T20:23:28Z" level=debug msg="Error, volume list empty"
time="2024-10-16T20:23:28Z" level=info msg="Creating new volume"
time="2024-10-16T20:23:28Z" level=info msg="Creating new volume"
time="2024-10-16T20:23:28Z" level=debug msg="DiskCapacityInBytes: 42949672960"
time="2024-10-16T20:23:28Z" level=debug msg="Creating new volume: {\"size\":40,\"metadata\":{\"disk\":\"922-2000\",\"migrate_kit\":\"true\",\"vm\":\"vm-37498\"},\"name\":\"odoo-netcentral-com-ph-922-2000\"}"
time="2024-10-16T20:23:29Z" level=info msg="Volume created, setting to bootable" volume_id=9b7ba9a2-5c98-4b34-a606-2072af3aeacc
time="2024-10-16T20:23:29Z" level=info msg="Attaching volume" volume_id=9b7ba9a2-5c98-4b34-a606-2072af3aeacc
Removing snapshot 100% [====================================] (100/100) [0s:0s]
Error: Get "http://169.254.169.254/openstack/latest/meta_data.json": dial tcp 169.254.169.254:80: i/o timeout
Usage:
  migratekit migrate [flags]

Flags:
  -h, --help   help for migrate

Global Flags:
jmussmann commented 3 days ago

The VM you running migrate kit on is on the same openstack as you are migration to?

That the "/dev/disk/by-id" directory does not exists is strange. This sould be there for standard ubuntu and also rocky installations.

Can you reach the metadata service from vm migratekit is running on (http://169.254.169.254/openstack/latest/meta_data.json)?

Are you using regions (like here https://github.com/vexxhost/migratekit/pull/37)?

doskelion commented 3 days ago

the VM i am running to is in vmware environment. Does it have to be in openstack?

jmussmann commented 3 days ago

Yes, the migrate kit vm has to be located in openstack, since it attaches directly to the volume. This explains why I was confused by the error messages.

doskelion commented 3 days ago

i may have missed any instructions in readme but couldn't find that tells me so. Let me try spinning one and keep you posted.

doskelion commented 3 days ago

so did spin up a new one and got stuck in "http://169.254.169.254/openstack/latest/meta_data.json"

time="2024-10-16T20:45:36Z" level=debug msg="Info: Volumename: odoo-netcentral-com-ph-922-2000 VM: vm-37498, disk: 922-2000"
time="2024-10-16T20:45:36Z" level=debug msg="Error, volume list empty"
time="2024-10-16T20:45:36Z" level=info msg="Data does not exist, full copy needed"
time="2024-10-16T20:45:36Z" level=debug msg="Info: Volumename: odoo-netcentral-com-ph-922-2000 VM: vm-37498, disk: 922-2000"
time="2024-10-16T20:45:36Z" level=debug msg="Error, volume list empty"
time="2024-10-16T20:45:36Z" level=info msg="Creating new volume"
time="2024-10-16T20:45:36Z" level=info msg="Creating new volume"
time="2024-10-16T20:45:36Z" level=debug msg="DiskCapacityInBytes: 42949672960"
time="2024-10-16T20:45:36Z" level=debug msg="Creating new volume: {\"size\":40,\"metadata\":{\"disk\":\"922-2000\",\"migrate_kit\":\"true\",\"vm\":\"vm-37498\"},\"name\":\"odoo-netcentral-com-ph-922-2000\"}"
time="2024-10-16T20:45:37Z" level=info msg="Volume created, setting to bootable" volume_id=d84d8a6a-8d12-4e31-a88a-f2aee2c88bf3
time="2024-10-16T20:45:37Z" level=info msg="Attaching volume" volume_id=d84d8a6a-8d12-4e31-a88a-f2aee2c88bf3
Removing snapshot 100% [====================================] (100/100) [0s:0s]
Error: Get "http://169.254.169.254/openstack/latest/meta_data.json": dial tcp 169.254.169.254:80: i/o timeout
Usage:
  migratekit migrate [flags]

looks like my vm couldnt not reach that url, i may have missed somewhere again.

To answer also your other question, no i dont use regions like you mentioned in issue #37

Let me explore more and see what i find. Will keep you posted.

doskelion commented 3 days ago

Figured out api is in different url and port. Our environment is an openstack variant flavor called VHI (Virtuozzo Hybrid Infra) and upon checking vhi compute api in https://docs.virtuozzo.com/virtuozzo_hybrid_infrastructure_6_1_compute_api_reference/index.html#showing-virtual-machine-details.html , so pulling hardcoded 169.x.x.x api can't work. Hopefully in future would support it. Thank you Jan for prompt response and do let me know if there are future plans to support it.