search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.19k stars 5.48k forks source link

salt-cloud vsphere vsan virtual machine creation failure #52802

Open wattwood opened 5 years ago

wattwood commented 5 years ago

Description of Issue/Question

When attempting to create a VM on vCenter 6.5 using Salt Cloud on a VSAN, the creation fails. When applying the same configuration to local ESXI storage, the VM is created.

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)

centos_secure_salt:
    provider: vxrail_vcenter
    clone_from: 'template-centos7-secure-salt'
    devices:
        disk:
            Hard disk 1:
                size: 30
    cluster: '...-Virtual-SAN-Cluster-...'
    num_cpus: 2
    memory: 2GB
    network:
        Network adapter 1:
            name: es190
            switch_type: standard
            ip: ...
            gateway: ...
            subnet_mask: 255.255.0.0
            domain: ...
    domain: ...
    dns_servers: 
        - ...
        - ...
    datastore: '...-Virtual-SAN-Datastore-...'
    datacenter: '...-Datacenter'
    template: False
    power_on: False
    extra_config:
        mem.hotadd: 'yes'
    annotation: Test
    minion:
        master: saltmaster.domain.ext
    deploy: True

Steps to Reproduce Issue

(Include debug logs if possible and relevant.)

[DEBUG   ] Closing IPCMessageClient instance
[INFO    ] Creating new_vm
[INFO    ] [ new_vm ] Waiting for create task to finish [0 s]
[ERROR   ] (vim.fault.CannotCreateFile) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'Cannot complete file creation operation.',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   file = '/vmfs/volumes/vsan:5292a6b5bdfbb488-d9b171feda0214ed/new_vm/new_vm-1eb02083.hlog'
}
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/utils/vmware.py", line 3153, in wait_for_task
    raise task_info.error
vim.fault.CannotCreateFile: (vim.fault.CannotCreateFile) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'Cannot complete file creation operation.',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) [],
   file = '/vmfs/volumes/vsan:5292a6b5bdfbb488-d9b171feda0214ed/new_vm/new_vm-1eb02083.hlog'
}
[ERROR   ] Error creating new_vm: Cannot complete file creation operation.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/vmware.py", line 2901, in create
    salt.utils.vmware.wait_for_task(task, vm_name, "create", 15, 'info')
  File "/usr/lib/python2.7/site-packages/salt/utils/vmware.py", line 3164, in wait_for_task
    raise salt.exceptions.VMwareApiError(exc.msg)
VMwareApiError: Cannot complete file creation operation.
[DEBUG   ] LazyLoaded nested.output
new_vm:
    ----------
    Error:
        Error creating new_vm: Cannot complete file creation operation.

2019-05-02T19:53:58.407Z cpu19:69226 opID=3d35b10d)osfs: OSFS_CreateFile:168: mkdir not enabled, failing mkdir request (pid: [    vsan], cid: 5292a6b5bdfbb488-d9b171feda0214ed, childName: new_vm)

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)


Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: 1.12.2
       cherrypy: unknown
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Apr  9 2019, 14:30:50)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.5.1.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core
waynew commented 5 years ago

@saltstack/team-cloud Any ideas here?

gtmanfred commented 5 years ago

It says that the building of the vsan timed out, did it actually build or was there an error in vsphere?

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/vmware.py", line 2901, in create
    salt.utils.vmware.wait_for_task(task, vm_name, "create", 15, 'info')
  File "/usr/lib/python2.7/site-packages/salt/utils/vmware.py", line 3164, in wait_for_task
    raise salt.exceptions.VMwareApiError(exc.msg)
VMwareApiError: Cannot complete file creation operation.
wattwood commented 5 years ago

It says that the building of the vsan timed out, did it actually build or was there an error in vsphere?

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/vmware.py", line 2901, in create
    salt.utils.vmware.wait_for_task(task, vm_name, "create", 15, 'info')
  File "/usr/lib/python2.7/site-packages/salt/utils/vmware.py", line 3164, in wait_for_task
    raise salt.exceptions.VMwareApiError(exc.msg)
VMwareApiError: Cannot complete file creation operation.

This is from my most recent attempt

May  7 17:07:09 node02 Hostd:[info] Task Created : haTask--vim.host.OperationCleanupManager.createEntry-207395421
May  7 17:07:09 node02 Hostd:[verbose] RecordOp ADD: recentTask["haTask--vim.host.OperationCleanupManager.createEntry-207395421"], ha-taskmgr. Applied change to temp map.
May  7 17:07:09 node02 Hostd:[verbose] RecordOp ASSIGN: info, haTask--vim.host.OperationCleanupManager.createEntry-207395421. Applied change to temp map.
May  7 17:07:09 node02 Hostd:[error] Failed to create directory for host log file /vmfs/volumes/vsan:5292a6b5bdfbb488-d9b171feda0214ed/new_vm2/new_vm2-0b4bcfaf.hlog
May  7 17:07:09 node02 Hostd:[info] AdapterServer caught exception: vim.fault.CannotCreateFile
May  7 17:07:09 node02 Hostd:[info] Task Completed : haTask--vim.host.OperationCleanupManager.createEntry-207395421 Status error
May  7 17:07:09 node02 Hostd:[verbose] RecordOp ASSIGN: info, haTask--vim.host.OperationCleanupManager.createEntry-207395421. Applied change to temp map.
May  7 17:07:09 node02 Hostd:[info] Activation [N5Vmomi10ActivationE:0x54e5fca0] : Invoke done [createEntry] on [vim.host.OperationCleanupManager:ha-op-cleanup-mgr]
May  7 17:07:09 node02 Hostd:[verbose] Arg entry:
May  7 17:07:09 node02 Hostd:[-->] {
May  7 17:07:09 node02 Hostd:[-->]  hlogFile = "/vmfs/volumes/vsan:5292a6b5bdfbb488-d9b171feda0214ed/new_vm2/new_vm2-0b4bcfaf.hlog",
May  7 17:07:09 node02 Hostd:[-->]  opId = 2221475849602357233,
May  7 17:07:09 node02 Hostd:[-->]  opState = "running",
May  7 17:07:09 node02 Hostd:[-->]  opActivity = "create",
May  7 17:07:09 node02 Hostd:[-->]  curHostUuid = "6253a760-7c66-1000-95a0-54ab3a89ef4a",
May  7 17:07:09 node02 Hostd:[-->]  itemList = (vim.host.OperationCleanupManager.CleanupItemEntry) [
May  7 17:07:09 node02 Hostd:[-->]     (vim.host.OperationCleanupManager.CleanupItemEntry) {
May  7 17:07:09 node02 Hostd:[-->]        path = "/vmfs/volumes/vsan:5292a6b5bdfbb488-d9b171feda0214ed/new_vm2/",
May  7 17:07:09 node02 Hostd:[-->]        type = "dir",
May  7 17:07:09 node02 Hostd:[-->]        removeUpon = "failure"
May  7 17:07:09 node02 Hostd:[-->]     },
May  7 17:07:09 node02 Hostd:[-->]     (vim.host.OperationCleanupManager.CleanupItemEntry) {
May  7 17:07:09 node02 Hostd:[-->]        path = "/vmfs/volumes/vsan:5292a6b5bdfbb488-d9b171feda0214ed/new_vm2/new_vm2.vmx",
May  7 17:07:09 node02 Hostd:[-->]        type = "vm",
May  7 17:07:09 node02 Hostd:[-->]        removeUpon = "failure"
May  7 17:07:09 node02 Hostd:[-->]     }
May  7 17:07:09 node02 Hostd:[-->]  ]
May  7 17:07:09 node02 Hostd:[-->] 
May  7 17:07:09 node02 Hostd:[info] Throw vim.fault.CannotCreateFile
May  7 17:07:09 node02 Hostd:[info] Result:
May  7 17:07:09 node02 Hostd:[-->] {
May  7 17:07:09 node02 Hostd:[-->]  faultCause = (vmodl.MethodFault) null,
May  7 17:07:09 node02 Hostd:[-->]  faultMessage = <unset>,
May  7 17:07:09 node02 Hostd:[-->]  file = "/vmfs/volumes/vsan:5292a6b5bdfbb488-d9b171feda0214ed/new_vm2/new_vm2-0b4bcfaf.hlog"
May  7 17:07:09 node02 Hostd:[-->]  msg = ""
May  7 17:07:09 node02 Hostd:[-->] 
May  7 17:07:09 node02 Hostd:[info] Task Created : haTask--vim.DatastoreNamespaceManager.DeleteDirectory-207395422
May  7 17:07:09 node02 Hostd:[verbose] RecordOp ADD: recentTask["haTask--vim.DatastoreNamespaceManager.DeleteDirectory-207395422"], ha-taskmgr. Applied change to temp map.
May  7 17:07:09 node02 Hostd:[verbose] RecordOp ASSIGN: info, haTask--vim.DatastoreNamespaceManager.DeleteDirectory-207395422. Applied change to temp map.
May  7 17:07:09 vxrail-node01.vsphere.local.vsphere.local Hostd:[verbose] RecordOp ASSIGN: guest.disk, 104. Sent notification immediately.
May  7 17:07:09 node02 Hostd:[info] AdapterServer caught exception: vim.fault.FileNotFound
May  7 17:07:09 node02 Hostd:[info] Task Completed : haTask--vim.DatastoreNamespaceManager.DeleteDirectory-207395422 Status error
May  7 17:07:09 node02 Hostd:[verbose] RecordOp ASSIGN: info, haTask--vim.DatastoreNamespaceManager.DeleteDirectory-207395422. Applied change to temp map.
May  7 17:07:09 node02 Hostd:[info] Activation [N5Vmomi10ActivationE:0x54e4ab20] : Invoke done [DeleteDirectory] on [vim.DatastoreNamespaceManager:ha-datastore-namespace-manager]
May  7 17:07:09 node02 Hostd:[verbose] Arg datacenter:
May  7 17:07:09 node02 Hostd:[-->] 
May  7 17:07:09 node02 Hostd:[verbose] Arg datastorePath:
May  7 17:07:09 node02 Hostd:[-->] 
May  7 17:07:09 node02 Hostd:[info] Throw vim.fault.FileNotFound
May  7 17:07:09 node02 Hostd:[info] Result:
May  7 17:07:09 node02 Hostd:[-->] {
May  7 17:07:09 node02 Hostd:[-->]  faultCause = (vmodl.MethodFault) null,
May  7 17:07:09 node02izableMessage) [
May  7 17:07:09 node02izableMessage) {
May  7 17:07:09 node02 Hostd:[-->]        key = "com.vmware.esx.hostctl.default",
May  7 17:07:09 node02 Hostd:[-->]        arg = (vmodl.KeyAnyValue) [
May  7 17:07:09 node02 Hostd:[-->]           (vmodl.KeyAnyValue) {
May  7 17:07:09 node02 Hostd:[-->]              key = "reason",
May  7 17:07:09 node02 Hostd:[-->]              value = "Could not get ObjectStoreFileSystem for  (File Not Found)"
May  7 17:07:09 node02 Hostd:[-->]           }
May  7 17:07:09 node02 Hostd:[-->]        ],
May  7 17:07:09 node02 Hostd:[-->]        message = <unset>
May  7 17:07:09 node02 Hostd:[-->]     }
May  7 17:07:09 node02 Hostd:[-->]  ],
May  7 17:07:09 node02 Hostd:[-->]  file = ""
May  7 17:07:09 node02 Hostd:[-->]  msg = ""
May  7 17:07:09 node02 Hostd:[-->] 
==> vmkernel.log <==
May  7 17:07:09 vxrail-node02.vsphere.local.vsphere.local vmkernel:[cpu5:68973] 12235: VC opID 2e5e7ee8-01-45-aba8 maps to vmkernel opID badcf7bd
May  7 17:07:09 vxrail-node02.vsphere.local.vsphere.local vmkernel:[cpu5:68973] OSFS_CreateFile:168: mkdir not enabled, failing mkdir request (pid: [    vsan], cid: 5292a6b5bdfbb488-d9b171feda0214ed, childName: new_vm2)
wattwood commented 5 years ago

@saltstack/team-cloud -- I read that a vSAN datastore is a namespace object, which explains why mkdir fails. This lines up with the command working on local storage.

Does salt-cloud's vCenter integration functionality need to be enhanced to use objtool or osfs-mkdir for vSAN storage?

wattwood commented 5 years ago

Will this get any additional attention?

gtmanfred commented 5 years ago

@nmadhok can you take a look at this?

xeacott commented 4 years ago

Hey @wattwood I'm going to start taking this over today. After I get my environment setup to address this I'll swing back - there also may be another python package to make this work. Recently I've been working with vmware's vcloud, which they re-named to their automation-sdk but there may be some tools in there that can also address this.

xeacott commented 4 years ago

I think I have my environment setup correctly now, can you provide the command you ran to reproduce this? So far I have salt-cloud -f test_vcenter_connection vmware returning fine.

wattwood commented 4 years ago

Sorry, I didn't see this. The connection is fine, it's creating a VM that fails. The VM config is in the original post (above).

wattwood commented 4 years ago

Technically I'm cloning from an existing VM template that's on the VSAN

waynew commented 4 years ago

@xeacott were you able to repro this?