Permissions error when trying to Clone VM

[michaelrdixey]

Hello,

I am attempting to clone a VM and running into a "Permission to perform this operation was denied" error. I am following the script located in https://github.com/vmware/pyvmomi-community-samples/blob/master/samples/clone_vm.py. I am able to clone the VM successfully through the GUI. I have also followed the other samples to create, delete, and revert snapshots, all of which have been working successfully.
One area of concern I have is with "vm_folder" var. If I do not provide one, it automatically gets a folder called "vm" from the datacenter. I do not see this folder in the GUI anywhere, so I'm not sure this is even correct. How do I go about identifying the folder for the existing vm I'm trying to clone? (I would like the new VM to go in the same location, on the same host).
I have added the code below, but it is the same as the original clone_vm.py.

def clone_vm(
        content, template, vm_name, si,
        datacenter_name, vm_folder, datastore_name,
        cluster_name, resource_pool, power_on, datastorecluster_name):
    """
    Clone a VM from a template/VM, datacenter_name, vm_folder, datastore_name
    cluster_name, resource_pool, and power_on are all optional.
    """

    # if none git the first one
    datacenter = get_obj(content, [vim.Datacenter], datacenter_name)

    if vm_folder:
        destfolder = get_obj(content, [vim.Folder], vm_folder)
    else:
        destfolder = datacenter.vmFolder

    if datastore_name:
        datastore = get_obj(content, [vim.Datastore], datastore_name)
    else:
        datastore = get_obj(
            content, [vim.Datastore], template.datastore[0].info.name)

    # if None, get the first one
    cluster = get_obj(content, [vim.ClusterComputeResource], cluster_name)

    if resource_pool:
        resource_pool = get_obj(content, [vim.ResourcePool], resource_pool)
    else:
        resource_pool = cluster.resourcePool

    vmconf = vim.vm.ConfigSpec()

    if datastorecluster_name:
        podsel = vim.storageDrs.PodSelectionSpec()
        pod = get_obj(content, [vim.StoragePod], datastorecluster_name)
        podsel.storagePod = pod

        storagespec = vim.storageDrs.StoragePlacementSpec()
        storagespec.podSelectionSpec = podsel
        storagespec.type = 'create'
        storagespec.folder = destfolder
        storagespec.resourcePool = resource_pool
        storagespec.configSpec = vmconf

        try:
            rec = content.storageResourceManager.RecommendDatastores(
                storageSpec=storagespec)
            rec_action = rec.recommendations[0].action[0]
            real_datastore_name = rec_action.destination.name
        except:
            real_datastore_name = template.datastore[0].info.name

        datastore = get_obj(content, [vim.Datastore], real_datastore_name)

    # set relospec
    relospec = vim.vm.RelocateSpec()
    relospec.datastore = datastore
    relospec.pool = resource_pool

    clonespec = vim.vm.CloneSpec()
    clonespec.location = relospec
    clonespec.powerOn = power_on

    print("cloning VM...")
    task = template.Clone(folder=destfolder, name=vm_name, spec=clonespec)
    WaitForTask(task)

I am calling this function from my main() via:

si = SmartConnectNoSSL(host=args["host"],
                           user=args["user"],
                           pwd=args["pwd"],
                           port=args["port"])
atexit.register(Disconnect, si)
content = si.RetrieveContent()
template = None
template = get_obj(content, [vim.VirtualMachine], args["template_name"])
clone_vm(content, template, "QA_Cloned_VM_Test1", si, datacenter_name="Development", vm_folder=None, datastore_name="datastore1 (2)",
             cluster_name=None, resource_pool=None, power_on=True, datastorecluster_name=None)

This is the full error output:

cloning VM...
Traceback (most recent call last):
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\Upgrade_Script_Base.py", line 379, in <module>
    main()
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\Upgrade_Script_Base.py", line 340, in main
    clone_vm(content, template, "QA_Cloned_VM_Test1", si, datacenter_name="Development", vm_folder="vm", datastore_name="datastore1 (2)",
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\Upgrade_Script_Base.py", line 220, in clone_vm
    task = template.Clone(folder=destfolder, name=vm_name, spec=clonespec)
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\venv\lib\site-packages\pyVmomi\VmomiSupport.py", line 706, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\venv\lib\site-packages\pyVmomi\VmomiSupport.py", line 512, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\venv\lib\site-packages\pyVmomi\SoapAdapter.py", line 1397, in InvokeMethod
    raise obj # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.NoPermission: (vim.fault.NoPermission) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'Permission to perform this operation was denied.',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   object = 'vim.Datacenter:datacenter-2',
   privilegeId = 'VirtualMachine.Inventory.CreateFromExisting'
}

Process finished with exit code 1

Any help would be much appreciated. Thanks!

[prziborowski]

I think the folder of the existing VM/template would be its parent. So based on your code template.parent would be the folder object. The clone_vm function is doing a lookup by the name, so you could try changing that to allow passing in the object and using that as-is. But if you can clone from the UI and you select the datacenter and not a folder under it, then it would be the equivalent of destfolder = datacenter.vmFolder in the code.

[michaelrdixey]

@prziborowski Thanks for the prompt response. You are correct in that I just select the datacenter in the UI and not a folder under it. In this case it makes sense it should be destfolder = datacenter.vmFolder. The debugger shows this folder name as "vm". Any thoughts regarding the permission error? I have max permissions for this datacenter, but not for other datacenters. I assumed this should not affect the process as I am trying to stay on the same datacenter and host.

[prziborowski]

At the moment the only thing I can think to check is going to the UI and clicking on your "Development" datacenter that you should have full access to. In the URL it should mention a moId. Can you confirm that is "datacenter-2"?

The backtrace of your error also has me a bit worried:

    clone_vm(content, template, "QA_Cloned_VM_Test1", si, datacenter_name="Development", vm_folder="vm", datastore_name="datastore1 (2)",

as it mentions vm_folder being specified, which in the clone_vm function would take priority over the datacenter_name. But I've never done a get_obj on "vm", so not sure if that is picking up the default vmFolder of a datacenter.

[michaelrdixey]

When clicking on the "Development" datacenter, the mold that appears in the URL is NOT "datacenter-2", but rather "datacenter-1653". Interestingly, one of the other datacenters does in fact have the URL "datacenter-2" (and it is one of the datacenters that this user doesn't have access to). I'm not sure why it's getting the object with "datacenter-2" in the error output, as the source VM is not on that datacenter.

Regarding vm_folder being specified, I have removed the declaration and am now calling the clone_vm function with vm_folder=None instead.

I have created another vsphere user with full access across all datacenters, and it did indeed cause the permissions error to go away. I am now running into a new error with the output message: "Datastore not found in destination datacenter". This error occurs regardless of whether or not I specify the datastore_name parameter or simply leave it as "None". The datastore "datastore1 (2)" is definitely within the "Development" datacenter, as seen in the image below:

I have pasted the current error output below:

cloning VM...
Traceback (most recent call last):
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\Upgrade_Script_Base.py", line 379, in <module>
    main()
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\Upgrade_Script_Base.py", line 340, in main
    clone_vm(content, template, "QA_Cloned_VM_Test1", si, datacenter_name="Development", vm_folder=None, datastore_name=None,
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\Upgrade_Script_Base.py", line 221, in clone_vm
    WaitForTask(task)
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\venv\lib\site-packages\pyVim\task.py", line 139, in WaitForTask
    raise task.info.error
pyVmomi.VmomiSupport.InvalidArgument: (vmodl.fault.InvalidArgument) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'A specified parameter was not correct: spec.datastore',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [
      (vmodl.LocalizableMessage) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         key = 'com.vmware.vim.vpxd.vpx.vmprov.error.destinationDatastoreNotFound',
         arg = (vmodl.KeyAnyValue) [],
         message = 'Datastore not found in destination datacenter'
      }
   ],
   invalidProperty = 'spec.datastore'
}

Process finished with exit code 1

[prziborowski]

I think something that might be confusing is that get_obj is running over the whole VC inventory, so it is possibly matching a different datacenter that has the datastore with that name.

I think because you have

    datacenter = get_obj(content, [vim.Datacenter], datacenter_name)

you can do:

    if datastore_name:
        datastore = get_obj(datacenter, [vim.Datastore], datastore_name)
    else:
        datastore = get_obj(
            datacenter, [vim.Datastore], template.datastore[0].info.name)

So that it will only fetch datastores that are within your datacenter.

[michaelrdixey]

With your proposed change, it fails on the get_obj call in both if/else conditions with "'vim.Datacenter' object has no attribute 'viewManager'":

Traceback (most recent call last):
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\Upgrade_Script_Base.py", line 385, in <module>
    main()
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\Upgrade_Script_Base.py", line 346, in main
    clone_vm(content, template, "QA_Cloned_VM_Test1", si, datacenter_name="Development", vm_folder=None, datastore_name="datastore1 (2)",
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\Upgrade_Script_Base.py", line 179, in clone_vm
    datastore = get_obj(datacenter, [vim.Datastore], datastore_name)
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\Upgrade_Script_Base.py", line 236, in get_obj
    container = content.viewManager.CreateContainerView(
AttributeError: 'vim.Datacenter' object has no attribute 'viewManager'

Process finished with exit code 1

[prziborowski]

Oh, bummer. I didn't realize that get_obj was implemented like that.

I prefer one that is more flexible like:

def get_obj(si, root, vim_type, name):
    container = si.content.viewManager.CreateContainerView(root, vim_type,
                                                           True)
    for c in container.view:
        if name:
            if c.name == name:
                obj = c
                break
        else:
            obj = c
            break
    container.Destroy()
    return obj

That would make you rework the inputs though like:

    datacenter = get_obj(si, si.content.rootFolder, [vim.Datacenter], datacenter_name)
...
    if datastore_name:
        datastore = get_obj(si, datacenter, [vim.Datastore], datastore_name)
    else:
        datastore = get_obj(si, 
            datacenter, [vim.Datastore], template.datastore[0].info.name)

[edited to fix an issue: si.content -> si.content.rootFolder]

[michaelrdixey]

Cool, I will give your method a try and report back. Thanks!

[michaelrdixey]

Unfortunately, this brings me back to the same error of "Datastore not found in destination datacenter". This occurs regardless of whether or not I specify the datastore_name parameter.

cloning VM...
Traceback (most recent call last):
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\Upgrade_Script_Base.py", line 408, in <module>
    main()
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\Upgrade_Script_Base.py", line 369, in main
    clone_vm(content, template, "QA_Cloned_VM_Test1", si, datacenter_name="Development", vm_folder=None, datastore_name="datastore1 (2)",
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\Upgrade_Script_Base.py", line 235, in clone_vm
    WaitForTask(task)
  File "C:\Users\Michael.Dixey\PycharmProjects\Scrut_Upgrade_Tester\venv\lib\site-packages\pyVim\task.py", line 139, in WaitForTask
    raise task.info.error
pyVmomi.VmomiSupport.InvalidArgument: (vmodl.fault.InvalidArgument) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'A specified parameter was not correct: spec.datastore',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [
      (vmodl.LocalizableMessage) {
         dynamicType = <unset>,
         dynamicProperty = (vmodl.DynamicProperty) [],
         key = 'com.vmware.vim.vpxd.vpx.vmprov.error.destinationDatastoreNotFound',
         arg = (vmodl.KeyAnyValue) [],
         message = 'Datastore not found in destination datacenter'
      }
   ],
   invalidProperty = 'spec.datastore'
}

Process finished with exit code 1

[prziborowski]

I'm fairly stumped at the moment. I only have a couple ideas of things to check on. If the current VM (or template) is in the same datacenter as your target, then you might try not passing in a datastore (in which case the cloned VM will be on the same datastore). Meaning remove or comment out:

    relospec.datastore = datastore

Also I just noticed a round-about thing with:

        datastore = get_obj(si, 
            datacenter, [vim.Datastore], template.datastore[0].info.name)

it should be the same as saying:

        datastore = template.datastore[0]

But without doing a lookup by name. You could try that to basically explicitly say you want the VM on the same datastore.

Obviously neither of these solve the underlying question of why it is throwing the error. But it might help determine scenarios the error is not hit.

[michaelrdixey]

Hmm, no changes after your recent suggestions above. Still getting the same datastore error. I have stepped through the debugger and verified the datastore it is getting is "datastore-1934", which matches the URL that appears when clicking on "datastore1 (2)", so it is the correct datastore. The datacenter it's getting is "datacenter-1653", which is also correct. I'd be happy to print out any output throughout the process, just let me know if there's anything you'd like to see.

[michaelrdixey]

@prziborowski I have some more info after debugging further.
I reset back to using the original default clone_vm function provided in the samples. I tried cloning a template instead of a VM, no difference. (Still receiving the 'Datastore not found in destination datacenter' msg). I then tried cloning another VM on a different datastore and datacenter, and got a different error message stating my resource_pool was incorrect. This was interesting considering I didn't provide this value, and my vsphere instance does not have any configured resource_pools to begin with. After commenting out relospec.pool = resource_pool, the clone process completed successfully for this new VM on the different datastore/datacenter. I then tried again with the original VM, but specifying the datastore as the one associated with the VM I cloned successfully. This provided a new error message stating 'The input arguments had entities that did not belong to the same datacenter.' This message makes sense, as the original VM is not on the datastore I provided. I then tried cloning the 2nd VM, but while providing the datastore from the 1st VM, expecting to receive the same error message. Instead I got the original error message stating 'Datastore not found in destination datacenter'. This leads me to the following conclusion: There is a bug in the Clone function revolving around the use of datastores that contain spaces in the name. My datastore in question is "datastore1 (2)". This block

if datastore_name:
        datastore = get_obj(content, [vim.Datastore], datastore_name)
 else:
        datastore = get_obj(
            content, [vim.Datastore], template.datastore[0].info.name)

is working correctly, and is getting the correct datastore object. But when this is passed into the Clone function, it results in the 'Datastore not found in destination datacenter' message. I have tried with a datastore containing parentheses but no spaces, and that works correctly.

[prziborowski]

That is very interesting. Getting an error of "The input arguments had entities that did not belong to the same datacenter" sounds like a mismatch of Ids between datacenters. A lot of the logic for the datastores is based on the URL (which should be a more unique identifier) and not the name, so I'd be surprised if a space in the name was the cause of an issue.

I'd love to be able to reproduce this on my own, and from the latest update it doesn't seem that it has to do with user visibility permissions, unless this is still a scenario where the user only has access to this datacenter and other datacenters show those (I'm guessing local) datastores. That sounds unlikely as well. What version (and patch/build) of VC and ESX are you using?

[michaelrdixey]

Yes this issue is no longer related to permissions, as I am using an account with full elevated permissions now. As I said, the only common factor in every clone attempt i've tried that hasn't worked, has been the fact that the template VM is on a datastore with a space in the name, regardless of whether or not I specify the datastore_name argument.
My specs: vSphere Client version 6.7.0.20000 All of the attached hosts are using ESXi 6.0 or 6.7, but in particular the host that contains the problematic source VM/datastore is: VMware ESXi, 6.0.0, 2494585

[michaelrdixey]

Nevermind, I just changed the datastore name from "datastore1 (2)" to "datastore1_(2)" and it still doesn't work and gets the same 'Datastore not found in destination datacenter' message.

[michaelrdixey]

After further troubleshooting, it has become apparent that I am able to clone any VM from the "Quality Assurance" datacenter, while I cannot clone any VM from the "Development" datacenter. My permissions are set to the same for each.

[prziborowski]

Was there any "datastore1 (2)" on the "Quality Assurance" datacenter? I'm wondering where the mismatch could be happening, and if the UI is working fine, then I'd think it is a test script (because of inputs).

I was looking to see if there was a way to enable better tracing of what gets sent from the UI versus the script, but I couldn't find a good enough article that won't cause you to shoot yourself in the foot in the process of configuring it.

[michaelrdixey]

Nope. That datastore is only attached to 1 host, which is only part of the "Development" datacenter.

[prziborowski]

While I'm going over what clone_vm.py does (and confused over how it seems to require either resource pool or cluster name), one thing to double check...

    print("cloning VM...")
    print("Folder %s" % destfolder)
    print("Spec %s" % clonespec)
    task = template.Clone(folder=destfolder, name=vm_name, spec=clonespec)
    wait_for_task(task)

or looking at the vpxd.log and backward searching CloneSpec.

And maybe doing a comparison of the IDs to make sure the VM folder id is part of the datacenter you expect, and the host/resource pool/datastore also are there. If MOB access is enabled you can reach it at https://$VCIP/mob and specifically punching in those IDs like https://$VCIP/mob/?moid=$ID You can start with the datacenter id, https://$VCIP/mob/?moid=datacenter-1653 which will show the datastores it knows about, and vmFolder should be on that page as well. The hosts and resource pools would be under hostFolder, and then the childEntity of that (which is either cluster or standalone compute).

[michaelrdixey]

Here is the output of the 2 added print statements you requested:

cloning VM...
Folder 'vim.Folder:group-v328469'
Spec (vim.vm.CloneSpec) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   location = (vim.vm.RelocateSpec) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      service = <unset>,
      folder = <unset>,
      datastore = 'vim.Datastore:datastore-1934',
      diskMoveType = <unset>,
      pool = <unset>,
      host = <unset>,
      disk = (vim.vm.RelocateSpec.DiskLocator) [],
      transform = <unset>,
      deviceChange = (vim.vm.device.VirtualDeviceSpec) [],
      profile = (vim.vm.ProfileSpec) [],
      cryptoSpec = <unset>
   },
   template = false,
   config = <unset>,
   customization = <unset>,
   powerOn = true,
   snapshot = <unset>,
   memory = <unset>
}

I also am able to view the MOB pages as you suggested. The datacenter-1653 page does list datastore-1934 ("datastore1_(2)" now that i've changed the space). The vmFolder on that page is listed as group-v1654 ("vm"). Viewing the MOB page for group-v1654 lists all the various VMs including the one I'm trying to clone from. From the GUI, these VMs are directly under the datacenter and not actually within a folder. Interestingly, the destfolder printout you had me add above outputs "group-v328469", which is the folder containing the VMs on the "Quality Assurance" datacenter. This is obviously not what it should have gotten. I'll try and poke around to see why exactly the destfolder isn't correct here.

[michaelrdixey]

Got it. It's automatically grabbing the wrong datacenter and not the one of the template VM because of:

# if none git the first one
    datacenter = get_obj(content, [vim.Datacenter], datacenter_name)

The destfolder is set using:

destfolder = datacenter.vmFolder

Which isn't going to be correct.
It seems to be working on both datacenters now. Overall, I still had to comment out relospec.pool = resource_pool from the original script as it doesn't like the fact that my vcenter doesn't have resource pools. I will keep that change and continue to specify the datacenter parameter.