search

vmware / pyvmomi

VMware vSphere API Python Bindings
Apache License 2.0
2.2k stars 766 forks source link

ParseError on CreateCollectorForTasks.ReadNextTasks(10) #182

Open grzn opened 9 years ago

grzn commented 9 years ago

We had an incident at a customer side today with the following piece of code:

    from pyVmomi import vim
    filter_spec = vim.TaskFilterSpec(alarm=None, scheduledTask=None, state=["running", "queued"])
    collector = client.service_content.taskManager.CreateCollectorForTasks(filter_spec)
    while True:
        tasks = collector.ReadNextTasks(10)
        if len(tasks) == 0:
            break
        for task in tasks:
            yield task
    collector.DestroyCollector()

The traceback is:

  File "src/infi/vendata/vmware_powertools/vcenter/stale_tasks.py", line 9, in _get_all_running_tasks
    tasks = collector.ReadNextTasks(10)
  File "eggs/pyvmomi-5.5.0.1-py2.7.egg/pyVmomi/VmomiSupport.py", line 569, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "eggs/pyvmomi-5.5.0.1-py2.7.egg/pyVmomi/VmomiSupport.py", line 378, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "eggs/pyvmomi-5.5.0.1-py2.7.egg/pyVmomi/SoapAdapter.py", line 1292, in InvokeMethod
    raise exc
ParserError: 'xml document: <?xml version="1.0" encoding="UTF-8"?>\n<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"\n xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"\n xmlns:xsd="http://www.w3.org/2001/XMLSchema"\n xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">\n<soapenv:Body>\n<ReadNextTasksResponse xmlns="urn:vim25"><returnval><key>task-3996443</key><task type="Task">task-3996443</task><description><key>com.vmware.vdp2.task.description</key><arg><key>workOrderId</key><value xsi:type="xsd:string">cp.20141020150650</value></arg></description><descriptionId>com.vmware.vdp2.integritycheck</descriptionId><entity type="VirtualMachine">vm-133364</entity><entityName>VDP3</entityName><state>running</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>vmware\\sa-vdp3</userName></reason><queueTime>2014-10-20T15:09:08.789999Z</queueTime><startTime>2014-10-20T15:09:09.073Z</startTime><eventChainId>21691195</eventChainId></returnval><returnval><key>task-1178117</key><task type="Task">task-1178117</task><descriptionId>com.vmware.vcIntegrity.CheckNotificationTask</descriptionId><entity type="Folder">group-d1</entity><entityName>Datacenters</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonSchedule"><name>VMware vSphere Update Manager Check Notification</name><scheduledTask type="ScheduledTask">schedule-2</scheduledTask></reason><queueTime>2013-01-08T14:12:01.556999Z</queueTime><eventChainId>3410272</eventChainId></returnval><returnval><key>task-746590</key><task type="Task">task-746590</task><name>LeaseMapDiskRegion</name><descriptionId>host.DiskManager.Lease.MapDiskRegion</descriptionId><entity type="VirtualMachine">vm-982</entity><entityName>Applicure</entityName><state>queued</state><cancelled>false</cancelled><cancelable>false</cancelable><reason xsi:type="TaskReasonUser"><userName>VMWARE\\sa-veeam1</userName></reason><queueTime>2012-09-27T12:03:03.412999Z</queueTime><eventChainId>2325872</eventChainId></returnval><returnval><key>task-56327</key><task type="Task">task-56327</task><name>LeaseMapDiskRegion</name><descriptionId>host.DiskManager.Lease.MapDiskRegion</descriptionId><entity type="VirtualMachine">vm-1010</entity><entityName>vCD-Oracle</entityName><state>queued</state><cancelled>false</cancelled><cancelable>false</cancelable><reason xsi:type="TaskReasonUser"><userName>VMWARE\\sa-veeam1</userName></reason><queueTime>2012-02-27T10:56:50.656999Z</queueTime><eventChainId>345916</eventChainId></returnval></ReadNextTasksResponse>\n</soapenv:Body>\n</soapenv:Envelope> parse error at: line:7, col:1377'

The error is in line7 column 1337, which is the '/' in the ending name tag ... task-746590</task><name>LeaseMapDiskRegion</name>

with proper indentation, the response looks legit:

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <soapenv:Body>
        <ReadNextTasksResponse xmlns="urn:vim25">
            <returnval>
                <key>task-3996443</key>
                <task type="Task">task-3996443</task>
                <description>
                    <key>com.vmware.vdp2.task.description</key>
                    <arg>
                        <key>workOrderId</key>
                        <value xsi:type="xsd:string">cp.20141020150650</value>
                    </arg>
                </description>
                <descriptionId>com.vmware.vdp2.integritycheck</descriptionId>
                <entity type="VirtualMachine">vm-133364</entity>
                <entityName>VDP3</entityName>
                <state>running</state>
                <cancelled>false</cancelled>
                <cancelable>true</cancelable>
                <reason xsi:type="TaskReasonUser">
                    <userName>vmware\\sa-vdp3</userName>
                </reason>
                <queueTime>2014-10-20T15:09:08.789999Z</queueTime>
                <startTime>2014-10-20T15:09:09.073Z</startTime>
                <eventChainId>21691195</eventChainId>
            </returnval>
            <returnval>
                <key>task-1178117</key>
                <task type="Task">task-1178117</task>
                <descriptionId>com.vmware.vcIntegrity.CheckNotificationTask</descriptionId>
                <entity type="Folder">group-d1</entity>
                <entityName>Datacenters</entityName>
                <state>queued</state>
                <cancelled>false</cancelled>
                <cancelable>true</cancelable>
                <reason xsi:type="TaskReasonSchedule">
                    <name>VMware vSphere Update Manager Check Notification</name>
                    <scheduledTask type="ScheduledTask">schedule-2</scheduledTask>
                </reason>
                <queueTime>2013-01-08T14:12:01.556999Z</queueTime>
                <eventChainId>3410272</eventChainId>
            </returnval>
            <returnval>
                <key>task-746590</key>
                <task type="Task">task-746590</task>
                <name>LeaseMapDiskRegion</name>
                <descriptionId>host.DiskManager.Lease.MapDiskRegion</descriptionId>
                <entity type="VirtualMachine">vm-982</entity>
                <entityName>Applicure</entityName>
                <state>queued</state>
                <cancelled>false</cancelled>
                <cancelable>false</cancelable>
                <reason xsi:type="TaskReasonUser">
                    <userName>VMWARE\\sa-veeam1</userName>
                </reason>
                <queueTime>2012-09-27T12:03:03.412999Z</queueTime>
                <eventChainId>2325872</eventChainId>
            </returnval>
            <returnval>
                <key>task-56327</key>
                <task type="Task">task-56327</task>
                <name>LeaseMapDiskRegion</name>
                <descriptionId>host.DiskManager.Lease.MapDiskRegion</descriptionId>
                <entity type="VirtualMachine">vm-1010</entity>
                <entityName>vCD-Oracle</entityName>
                <state>queued</state>
                <cancelled>false</cancelled>
                <cancelable>false</cancelable>
                <reason xsi:type="TaskReasonUser">
                    <userName>VMWARE\\sa-veeam1</userName>
                </reason>
                <queueTime>2012-02-27T10:56:50.656999Z</queueTime>
                <eventChainId>345916</eventChainId>
            </returnval>
        </ReadNextTasksResponse>
    </soapenv:Body>
</soapenv:Envelope>

This was against vCenter 5.1, don't know which update, but I can find out. I'll try tomorrow to re-create this tomorrow by creating a task with a name attribute

Does anyone has more insight about this?

Thanks, Guy

grzn commented 9 years ago

From what I understand the name property I can't control the name attribute from the outside, so I can't recreate the scenario; we can just pass the response to the parser again and see the exception.

wiggin15 commented 9 years ago

You can reproduce the exception by copying the xml above (with the indentation) and parsing it manually:

from pyVmomi import vim
from pyVmomi.SoapAdapter import SoapStubAdapter, SoapResponseDeserializer
data = open("xml.xml", "rb").read()
stub = SoapStubAdapter(version="vim.version.version6")
deserializer = SoapResponseDeserializer(stub)
result = vim.TaskHistoryCollector._GetMethodInfo("ReadNext").result
obj = deserializer.Deserialize(data, result)

The resulting exception is replaced by ParserError so it's harder to debug, but looking further using this code, I found that this is the real exception:

  File "pyVmomi/SoapAdapter.py", line 458, in ReadDocument
    parser.Parse(data)
  File "pyVmomi/SoapAdapter.py", line 691, in EndElementHandler
    obj = GuessWsdlMethod(name)
  File "pyVmomi/VmomiSupport.py", line 1231, in GuessWsdlMethod
    raise KeyError(name)
KeyError(u'LeaseMapDiskRegion',)

The problem is that the XML contains the task LeaseMapDiskRegion which is a method of the ManagedObject HostDiskManagerLease - which is not exposed in pyvmomi's ServerObjects.py... This is (again) a problem with ServerObjects being redacted and not containing all the objects and methods, only this time it interferes with a legitimate call...

@hartsock is there a solution? maybe pyvmomi should expose all the "hidden" objects?

hartsock commented 9 years ago

Unfortunately this sounds like a fairly straight forward XML parsing issue. To fix this I would write a test that mocked up in a fixture the XML @grzn has posted and then iterate a few times to see if I couldn't make everything behave properly.

Notes:

Hacky solutions: @wiggin15 has provided enough info that you could hack ServerObjects.py to include the redacted method, this would solve your immediate problem. The bigger question is what is the least surprising behavior permitted? I've mentioned we could make the library metaphorically shrug and add the new method/datatype for itself... or we could throw an exception saying something like: "uh oh! you just pulled an object I don't know about!"

Possible related discussions:

Those are possible paths forward.

mytran commented 8 years ago

Any progress or recommended fixes for this issue? I'm running into a similar issue reading events. Looks like its failing with on a FileManager.delete event. You can reproduce the problem by creating a datastore file then deleting it.

JavaScriptDude commented 8 years ago

I am having this same error when running thc.CreateCollectorForTasks.ReadNextTasks(maxCount=10).

I believe this is 100% reproducible on my system. Let me know if I can grab info that will help.

adamkbmc commented 8 years ago

I am running into this problem using QueryEvents() - ParserError caused by a KeyError, in my case due to "Delete_Task"

Is there any info on when this will be fixed?

tianhao64 commented 8 years ago

@adamkbmc I was able to reproduce this issue. I will look into this.

tianhao64 commented 8 years ago

This is a bug on server side. In this case server is returning internal tasks by mistake. I have filed a bug against vc. I will try to make a patch to workaround this issue for now.

adamkbmc commented 8 years ago

@tianhao64 Thanks for the swift response. Let me know if I can help with testing a fix

tianhao64 commented 8 years ago

@adamkbmc I posted a pull request with the fix https://github.com/vmware/pyvmomi/pull/398. It will be great if you try it out and let me know if it works.

adamkbmc commented 8 years ago

@tianhao64 I pulled the code and it solves the problem I was having. I can now get events out of vCenter

andrew-regan commented 8 years ago

Seeing more version of this with VSAN events:

ParserError: 'xml document: <?xml version="1.0" encoding="UTF-8"?>\n<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"\n xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"\n xmlns:xsd="http://www.w3.org/2001/XMLSchema"\n xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">\n<soapenv:Body>\n<ReadNextEventsResponse xmlns="urn:vim25"><returnval xsi:type="TaskEvent"><key>43780</key><chainId>43780</chainId><createdTime>2016-05-25T23:55:59.075Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.62.186</name><host type="HostSystem">host-2274</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31947</key><task type="Task">task-31947</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-2274</entity><entityName>172.20.62.186</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:55:59.074932Z</queueTime><eventChainId>43780</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43781</key><chainId>43781</chainId><createdTime>2016-05-25T23:55:59.970999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.46.182</name><host type="HostSystem">host-2308</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31948</key><task type="Task">task-31948</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-2308</entity><entityName>172.20.46.182</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:55:59.970162Z</queueTime><eventChainId>43781</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43782</key><chainId>43782</chainId><createdTime>2016-05-25T23:55:59.984999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.118.30</name><host type="HostSystem">host-234</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31949</key><task type="Task">task-31949</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-234</entity><entityName>172.20.118.30</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:55:59.984591Z</queueTime><eventChainId>43782</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43783</key><chainId>43783</chainId><createdTime>2016-05-25T23:56:00.477Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.107.123</name><host type="HostSystem">host-2342</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31950</key><task type="Task">task-31950</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-2342</entity><entityName>172.20.107.123</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:56:00.476516Z</queueTime><eventChainId>43783</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43784</key><chainId>43784</chainId><createdTime>2016-05-25T23:56:01.141999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.187.44</name><host type="HostSystem">host-2376</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31951</key><task type="Task">task-31951</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-2376</entity><entityName>172.20.187.44</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:56:01.14166Z</queueTime><eventChainId>43784</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43785</key><chainId>43785</chainId><createdTime>2016-05-25T23:56:01.75Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.177.223</name><host type="HostSystem">host-2444</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31952</key><task type="Task">task-31952</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-2444</entity><entityName>172.20.177.223</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:56:01.749157Z</queueTime><eventChainId>43785</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43786</key><chainId>43786</chainId><createdTime>2016-05-25T23:56:01.829Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.149.51</name><host type="HostSystem">host-2410</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31953</key><task type="Task">task-31953</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-2410</entity><entityName>172.20.149.51</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:56:01.829214Z</queueTime><eventChainId>43786</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43787</key><chainId>43787</chainId><createdTime>2016-05-25T23:56:02.343999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.90.174</name><host type="HostSystem">host-2478</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31954</key><task type="Task">task-31954</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-2478</entity><entityName>172.20.90.174</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:56:02.343975Z</queueTime><eventChainId>43787</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43788</key><chainId>43788</chainId><createdTime>2016-05-25T23:56:02.355Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.79.27</name><host type="HostSystem">host-2512</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31955</key><task type="Task">task-31955</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-2512</entity><entityName>172.20.79.27</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:56:02.354626Z</queueTime><eventChainId>43788</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43789</key><chainId>43789</chainId><createdTime>2016-05-25T23:56:02.847999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.204.172</name><host type="HostSystem">host-2580</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31956</key><task type="Task">task-31956</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-2580</entity><entityName>172.20.204.172</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:56:02.847469Z</queueTime><eventChainId>43789</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43790</key><chainId>43790</chainId><createdTime>2016-05-25T23:57:02.321Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.52.29</name><host type="HostSystem">host-2546</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31957</key><task type="Task">task-31957</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-2546</entity><entityName>172.20.52.29</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:57:02.320221Z</queueTime><eventChainId>43790</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43791</key><chainId>43791</chainId><createdTime>2016-05-25T23:57:03.156Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.239.72</name><host type="HostSystem">host-2614</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31958</key><task type="Task">task-31958</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-2614</entity><entityName>172.20.239.72</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:57:03.155124Z</queueTime><eventChainId>43791</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43792</key><chainId>43792</chainId><createdTime>2016-05-25T23:57:03.218999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.218.110</name><host type="HostSystem">host-268</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31959</key><task type="Task">task-31959</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-268</entity><entityName>172.20.218.110</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:57:03.218568Z</queueTime><eventChainId>43792</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43793</key><chainId>43793</chainId><createdTime>2016-05-25T23:57:03.676Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.151.45</name><host type="HostSystem">host-2648</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31960</key><task type="Task">task-31960</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-2648</entity><entityName>172.20.151.45</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:57:03.676004Z</queueTime><eventChainId>43793</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43794</key><chainId>43794</chainId><createdTime>2016-05-25T23:57:04.351Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.232.10</name><host type="HostSystem">host-2682</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31961</key><task type="Task">task-31961</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-2682</entity><entityName>172.20.232.10</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:57:04.350825Z</queueTime><eventChainId>43794</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43795</key><chainId>43795</chainId><createdTime>2016-05-25T23:57:04.963Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.222.40</name><host type="HostSystem">host-29</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31962</key><task type="Task">task-31962</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-29</entity><entityName>172.20.222.40</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:57:04.962622Z</queueTime><eventChainId>43795</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43796</key><chainId>43796</chainId><createdTime>2016-05-25T23:57:05.035Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.15.104</name><host type="HostSystem">host-336</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31963</key><task type="Task">task-31963</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-336</entity><entityName>172.20.15.104</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:57:05.03521Z</queueTime><eventChainId>43796</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43797</key><chainId>43797</chainId><createdTime>2016-05-25T23:57:05.569Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.167.26</name><host type="HostSystem">host-2716</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31964</key><task type="Task">task-31964</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-2716</entity><entityName>172.20.167.26</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:57:05.569191Z</queueTime><eventChainId>43797</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43798</key><chainId>43798</chainId><createdTime>2016-05-25T23:57:05.578999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.44.174</name><host type="HostSystem">host-302</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31965</key><task type="Task">task-31965</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-302</entity><entityName>172.20.44.174</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:57:05.578343Z</queueTime><eventChainId>43798</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43799</key><chainId>43799</chainId><createdTime>2016-05-25T23:57:06.057999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.93.162</name><host type="HostSystem">host-404</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31966</key><task type="Task">task-31966</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-404</entity><entityName>172.20.93.162</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:57:06.057627Z</queueTime><eventChainId>43799</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43800</key><chainId>43800</chainId><createdTime>2016-05-25T23:58:05.561Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.34.245</name><host type="HostSystem">host-370</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31967</key><task type="Task">task-31967</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-370</entity><entityName>172.20.34.245</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:58:05.560556Z</queueTime><eventChainId>43800</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43801</key><chainId>43801</chainId><createdTime>2016-05-25T23:58:06.357999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.232.11</name><host type="HostSystem">host-438</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31968</key><task type="Task">task-31968</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-438</entity><entityName>172.20.232.11</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:58:06.357014Z</queueTime><eventChainId>43801</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43802</key><chainId>43802</chainId><createdTime>2016-05-25T23:58:06.418Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.113.70</name><host type="HostSystem">host-472</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31969</key><task type="Task">task-31969</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-472</entity><entityName>172.20.113.70</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:58:06.417817Z</queueTime><eventChainId>43802</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43803</key><chainId>43803</chainId><createdTime>2016-05-25T23:58:06.876999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.244.126</name><host type="HostSystem">host-506</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31970</key><task type="Task">task-31970</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-506</entity><entityName>172.20.244.126</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:58:06.876498Z</queueTime><eventChainId>43803</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43804</key><chainId>43804</chainId><createdTime>2016-05-25T23:58:07.562999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.178.161</name><host type="HostSystem">host-574</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31971</key><task type="Task">task-31971</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-574</entity><entityName>172.20.178.161</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:58:07.563124Z</queueTime><eventChainId>43804</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43805</key><chainId>43805</chainId><createdTime>2016-05-25T23:58:08.164999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.22.4</name><host type="HostSystem">host-540</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31972</key><task type="Task">task-31972</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-540</entity><entityName>172.20.22.4</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:58:08.164329Z</queueTime><eventChainId>43805</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43806</key><chainId>43806</chainId><createdTime>2016-05-25T23:58:08.256999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.185.100</name><host type="HostSystem">host-64</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31973</key><task type="Task">task-31973</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-64</entity><entityName>172.20.185.100</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:58:08.257133Z</queueTime><eventChainId>43806</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43807</key><chainId>43807</chainId><createdTime>2016-05-25T23:58:08.759999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.87.90</name><host type="HostSystem">host-642</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31974</key><task type="Task">task-31974</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-642</entity><entityName>172.20.87.90</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:58:08.759997Z</queueTime><eventChainId>43807</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43808</key><chainId>43808</chainId><createdTime>2016-05-25T23:58:08.781Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.96.231</name><host type="HostSystem">host-608</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31975</key><task type="Task">task-31975</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-608</entity><entityName>172.20.96.231</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:58:08.781051Z</queueTime><eventChainId>43808</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43809</key><chainId>43809</chainId><createdTime>2016-05-25T23:58:09.249Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.65.20</name><host type="HostSystem">host-676</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31976</key><task type="Task">task-31976</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-676</entity><entityName>172.20.65.20</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:58:09.249189Z</queueTime><eventChainId>43809</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43810</key><chainId>43810</chainId><createdTime>2016-05-25T23:59:08.756999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.124.60</name><host type="HostSystem">host-710</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31977</key><task type="Task">task-31977</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-710</entity><entityName>172.20.124.60</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:59:08.756261Z</queueTime><eventChainId>43810</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43811</key><chainId>43811</chainId><createdTime>2016-05-25T23:59:09.535999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.240.193</name><host type="HostSystem">host-744</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31978</key><task type="Task">task-31978</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-744</entity><entityName>172.20.240.193</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:59:09.535793Z</queueTime><eventChainId>43811</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43812</key><chainId>43812</chainId><createdTime>2016-05-25T23:59:09.617Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.190.194</name><host type="HostSystem">host-812</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31979</key><task type="Task">task-31979</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-812</entity><entityName>172.20.190.194</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:59:09.617112Z</queueTime><eventChainId>43812</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43813</key><chainId>43813</chainId><createdTime>2016-05-25T23:59:10.074999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.5.145</name><host type="HostSystem">host-778</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31980</key><task type="Task">task-31980</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-778</entity><entityName>172.20.5.145</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:59:10.075161Z</queueTime><eventChainId>43813</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43814</key><chainId>43814</chainId><createdTime>2016-05-25T23:59:10.772999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.67.26</name><host type="HostSystem">host-846</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31981</key><task type="Task">task-31981</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-846</entity><entityName>172.20.67.26</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:59:10.772527Z</queueTime><eventChainId>43814</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43815</key><chainId>43815</chainId><createdTime>2016-05-25T23:59:11.367Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.18.121</name><host type="HostSystem">host-880</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31982</key><task type="Task">task-31982</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-880</entity><entityName>172.20.18.121</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:59:11.366249Z</queueTime><eventChainId>43815</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43816</key><chainId>43816</chainId><createdTime>2016-05-25T23:59:11.442Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.64.125</name><host type="HostSystem">host-98</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31983</key><task type="Task">task-31983</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-98</entity><entityName>172.20.64.125</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:59:11.441596Z</queueTime><eventChainId>43816</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43817</key><chainId>43817</chainId><createdTime>2016-05-25T23:59:11.946999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.99.171</name><host type="HostSystem">host-948</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31984</key><task type="Task">task-31984</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-948</entity><entityName>172.20.99.171</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:59:11.946343Z</queueTime><eventChainId>43817</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43818</key><chainId>43818</chainId><createdTime>2016-05-25T23:59:11.971999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.47.198</name><host type="HostSystem">host-982</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31985</key><task type="Task">task-31985</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-982</entity><entityName>172.20.47.198</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:59:11.972039Z</queueTime><eventChainId>43818</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43819</key><chainId>43819</chainId><createdTime>2016-05-25T23:59:12.454Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.95.52</name><host type="HostSystem">host-914</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31986</key><task type="Task">task-31986</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-914</entity><entityName>172.20.95.52</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-25T23:59:12.453488Z</queueTime><eventChainId>43819</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43820</key><chainId>43820</chainId><createdTime>2016-05-26T00:00:11.971Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.135.56</name><host type="HostSystem">host-1016</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31987</key><task type="Task">task-31987</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1016</entity><entityName>172.20.135.56</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:00:11.970313Z</queueTime><eventChainId>43820</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43821</key><chainId>43821</chainId><createdTime>2016-05-26T00:00:12.765Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.65.34</name><host type="HostSystem">host-1050</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31988</key><task type="Task">task-31988</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1050</entity><entityName>172.20.65.34</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:00:12.764401Z</queueTime><eventChainId>0</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43822</key><chainId>43822</chainId><createdTime>2016-05-26T00:00:12.829Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.156.44</name><host type="HostSystem">host-1084</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31989</key><task type="Task">task-31989</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1084</entity><entityName>172.20.156.44</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:00:12.828677Z</queueTime><eventChainId>43822</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43823</key><chainId>43823</chainId><createdTime>2016-05-26T00:00:13.275Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.124.96</name><host type="HostSystem">host-1152</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31990</key><task type="Task">task-31990</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1152</entity><entityName>172.20.124.96</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:00:13.274748Z</queueTime><eventChainId>43823</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43824</key><chainId>43824</chainId><createdTime>2016-05-26T00:00:13.967Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.175.88</name><host type="HostSystem">host-1118</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31991</key><task type="Task">task-31991</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1118</entity><entityName>172.20.175.88</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:00:13.966671Z</queueTime><eventChainId>43824</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43825</key><chainId>43825</chainId><createdTime>2016-05-26T00:00:14.563Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.200.238</name><host type="HostSystem">host-1186</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31992</key><task type="Task">task-31992</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1186</entity><entityName>172.20.200.238</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:00:14.562761Z</queueTime><eventChainId>43825</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43826</key><chainId>43826</chainId><createdTime>2016-05-26T00:00:14.647Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.110.62</name><host type="HostSystem">host-1220</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31993</key><task type="Task">task-31993</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1220</entity><entityName>172.20.110.62</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:00:14.646956Z</queueTime><eventChainId>43826</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43827</key><chainId>43827</chainId><createdTime>2016-05-26T00:00:15.141999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.193.97</name><host type="HostSystem">host-1254</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31994</key><task type="Task">task-31994</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1254</entity><entityName>172.20.193.97</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:00:15.141764Z</queueTime><eventChainId>43827</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43828</key><chainId>43828</chainId><createdTime>2016-05-26T00:00:15.166Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.21.17</name><host type="HostSystem">host-1288</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31995</key><task type="Task">task-31995</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1288</entity><entityName>172.20.21.17</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:00:15.165599Z</queueTime><eventChainId>43828</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43829</key><chainId>43829</chainId><createdTime>2016-05-26T00:00:15.653Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.40.43</name><host type="HostSystem">host-132</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31996</key><task type="Task">task-31996</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-132</entity><entityName>172.20.40.43</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:00:15.652746Z</queueTime><eventChainId>43829</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43830</key><chainId>43830</chainId><createdTime>2016-05-26T00:01:15.185999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.114.55</name><host type="HostSystem">host-1322</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31997</key><task type="Task">task-31997</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1322</entity><entityName>172.20.114.55</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:01:15.185842Z</queueTime><eventChainId>43830</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43831</key><chainId>43831</chainId><createdTime>2016-05-26T00:01:15.971Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.113.105</name><host type="HostSystem">host-1356</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31998</key><task type="Task">task-31998</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1356</entity><entityName>172.20.113.105</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:01:15.970681Z</queueTime><eventChainId>43831</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43832</key><chainId>43832</chainId><createdTime>2016-05-26T00:01:16.033999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.223.6</name><host type="HostSystem">host-1390</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-31999</key><task type="Task">task-31999</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1390</entity><entityName>172.20.223.6</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:01:16.033297Z</queueTime><eventChainId>43832</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43833</key><chainId>43833</chainId><createdTime>2016-05-26T00:01:16.469999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.117.119</name><host type="HostSystem">host-1424</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-32000</key><task type="Task">task-32000</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1424</entity><entityName>172.20.117.119</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:01:16.470146Z</queueTime><eventChainId>43833</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43834</key><chainId>43834</chainId><createdTime>2016-05-26T00:01:17.147999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.220.81</name><host type="HostSystem">host-1458</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-32001</key><task type="Task">task-32001</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1458</entity><entityName>172.20.220.81</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:01:17.147984Z</queueTime><eventChainId>43834</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43835</key><chainId>43835</chainId><createdTime>2016-05-26T00:01:17.76Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.132.200</name><host type="HostSystem">host-1492</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-32002</key><task type="Task">task-32002</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1492</entity><entityName>172.20.132.200</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:01:17.759495Z</queueTime><eventChainId>43835</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43836</key><chainId>43836</chainId><createdTime>2016-05-26T00:01:17.836999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.106.122</name><host type="HostSystem">host-1526</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-32003</key><task type="Task">task-32003</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1526</entity><entityName>172.20.106.122</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:01:17.836851Z</queueTime><eventChainId>43836</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43837</key><chainId>43837</chainId><createdTime>2016-05-26T00:01:18.346Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.251.39</name><host type="HostSystem">host-1628</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-32004</key><task type="Task">task-32004</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1628</entity><entityName>172.20.251.39</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:01:18.345492Z</queueTime><eventChainId>43837</eventChainId></info></returnval><returnval xsi:type="TaskEvent"><key>43838</key><chainId>43838</chainId><createdTime>2016-05-26T00:01:18.356999Z</createdTime><userName>com.vmware.vsan.health</userName><datacenter><name>Datacenter</name><datacenter type="Datacenter">datacenter-21</datacenter></datacenter><computeResource><name>Test</name><computeResource type="ClusterComputeResource">domain-c26</computeResource></computeResource><host><name>172.20.48.101</name><host type="HostSystem">host-1594</host></host><fullFormattedMessage>Task: Retrieve a ticket to register the Virtual SAN VASA Provider</fullFormattedMessage><info><key>task-32005</key><task type="Task">task-32005</task><name>FetchVsanSharedSecret</name><descriptionId>host.VsanSystem.fetchVsanSharedSecret</descriptionId><entity type="HostSystem">host-1594</entity><entityName>172.20.48.101</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>com.vmware.vsan.health</userName></reason><queueTime>2016-05-26T00:01:18.356583Z</queueTime><eventChainId>43838</eventChainId></info></returnval></ReadNextEventsResponse>\n</soapenv:Body>\n</soapenv:Envelope> parse error at: line:7, col:724'
tianhao64 commented 8 years ago

@andrew-regan thanks for the update. I will file another defect. Can you also try the fix see if it works for you?

tianhao64 commented 8 years ago

@andrew-regan the vsan event issue is already resolved by vsan team. The fix will be available in next release.

andrew-regan commented 8 years ago

Thanks!

andrew-regan commented 8 years ago

We had thought this was fixed by picking up the v6.0.0 tag + commit 848edbe, but we've hit this issue again. We haven't picked up commit f3245a6 yet. Do you think that's why we're still hitting this?

We're using vSphere/ESX v6. To reproduce in vSphere go to vSphere Inventory List --> Content Libraries. Select a Library and choose Actions --> Import Item. When an item is imported the TaskEvent below is generated which triggers the issue:

Traceback (most recent call last):
  File "/plexxiconnect/packs/vsphere/sensors/../lib/vsphere_event_connector.py", line 110, in get_events
    next_events = collector.ReadNext(250)
  File "/opt/stackstorm/virtualenvs/vsphere/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 570, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/opt/stackstorm/virtualenvs/vsphere/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 376, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/opt/stackstorm/virtualenvs/vsphere/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1326, in InvokeMethod
    raise exc
ParserError: 'xml document: <?xml version="1.0" encoding="UTF-8"?>\n<soapenv:Envelope xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"\n xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"\n xmlns:xsd="http://www.w3.org/2001/XMLSchema"\n xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">\n<soapenv:Body>\n<ReadNextEventsResponse xmlns="urn:vim25"><returnval xsi:type="TaskEvent"><key>10394972</key><chainId>10394972</chainId><createdTime>2016-08-19T12:12:39.246999Z</createdTime><userName>vsphere.local\\Administrator</userName><fullFormattedMessage>Task: Create Library Item</fullFormattedMessage><info><key>task-3564</key><task type="Task">task-3564</task><descriptionId>com.vmware.content.LibraryItem.Create</descriptionId><entity type="ContentLibrary">clib-3553</entity><entityName>Test</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>vsphere.local\\Administrator</userName></reason><queueTime>2016-08-19T12:12:39.246924Z</queueTime><eventChainId>10394972</eventChainId><activationId>0c113bc6-65e7-4bd5-b1dc-9cfc6b176a6d</activationId></info></returnval><returnval xsi:type="TaskEvent"><key>10394974</key><chainId>10394974</chainId><createdTime>2016-08-19T12:12:41.091Z</createdTime><userName>VSPHERE.LOCAL\\Administrator</userName><fullFormattedMessage>Task: Upload Files to a Library Item</fullFormattedMessage><info><key>task-3565</key><task type="Task">task-3565</task><descriptionId>com.vmware.content.LibraryItem.UploadContent</descriptionId><entity type="ContentLibraryItem">clibitem-1424</entity><entityName>rpms_2.0.1-420.txt</entityName><state>queued</state><cancelled>false</cancelled><cancelable>true</cancelable><reason xsi:type="TaskReasonUser"><userName>VSPHERE.LOCAL\\Administrator</userName></reason><queueTime>2016-08-19T12:12:41.091183Z</queueTime><eventChainId>10394974</eventChainId><activationId>955b3353-5729-42dc-9a54-5aa64da24a62</activationId></info></returnval></ReadNextEventsResponse>\n</soapenv:Body>\n</soapenv:Envelope> parse error at: line:7, col:450'
andrew-regan commented 8 years ago

I was just looking to update pyVmomi on my end to test with the latest official release. It looks like the latest tag is v6.0.0.2016.4, but latest in pypi is 6.0.0.2016.6. Was the 2016.6 release not tagged? Thanks!

tianhao64 commented 8 years ago

@andrew-regan Thanks for reporting this. Yes, looks like the last release was not tagged properly. I will fix that.

Can you pick up the latest release from pypi and see if it fixes your issue?

andrew-regan commented 8 years ago

I picked up the latest pyvmomi (6.0.0.2016.6) and the issue still reproduces. It looks to not print out the bad XML now. I followed the same steps mentioned above of uploading a file via the vSphere UI to a Content Library.

Traceback (most recent call last):
  File "/plexxiconnect/packs/vsphere/sensors/../lib/vsphere_event_connector.py", line 110, in get_events
    next_events = collector.ReadNext(250)
  File "/opt/stackstorm/virtualenvs/vsphere/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 580, in <lambda>
    self.f(*(self.args + (obj,) + args), **kwargs)
  File "/opt/stackstorm/virtualenvs/vsphere/lib/python2.7/site-packages/pyVmomi/VmomiSupport.py", line 386, in _InvokeMethod
    return self._stub.InvokeMethod(self, info, args)
  File "/opt/stackstorm/virtualenvs/vsphere/lib/python2.7/site-packages/pyVmomi/SoapAdapter.py", line 1364, in InvokeMethod
    raise exc
ParserError: 'xml document: <httplib.HTTPResponse instance at 0x7fef7d366fc8> parse error at: line:7, col:450'

This is my method that is hitting the issue:

    def get_events(self, event_types):
        """Retrieve events from vSphere that occured within the filter.

        Arguments:
            event_types (str[]): List of vSphere Event types that are supported

        Returns:
            Event[]: List of vSphere Events
        """
        events = []
        collector = None

        # noinspection PyBroadException
        try:
            # create a filter to retrieve Events based on a slice of time
            timespec = vim.event.EventFilterSpec.ByTime()
            timespec.beginTime = self._last_vsphere_event_time
            timespec.endTime = self.get_current_vsphere_time()

            self._logger.debug('Retrieving vSphere events from %s to %s',
                               timespec.beginTime,
                               timespec.endTime)

            filterspec = vim.event.EventFilterSpec(
                time=timespec,
                eventTypeId=event_types)
            svc_content = self.vsphere_connection.si.RetrieveServiceContent()
            collector = svc_content.eventManager.CreateCollectorForEvents(filter=filterspec)

            # retrieve the events from vSphere via the collector and filter
            while True:
                next_events = collector.ReadNext(250)

                if next_events:
                    self._logger.debug('Latest vSphere events: %s', next_events)
                    events += next_events
                else:
                    break

            self._last_vsphere_event_time = timespec.endTime
        except Exception:
            self._logger.warning('Could not get events from vSphere.')
            self._logger.warning(traceback.format_exc())
        finally:
            if collector:
                collector.DestroyCollector()

        return events
andrew-regan commented 8 years ago

@tianhao64, are you able to reproduce this on your side?

tianhao64 commented 8 years ago

@andrew-regan Yes, I am able to reproduce the issue by parsing the same xml in your error message. However, the stack trace is a bit difference:

obj = deserializer.Deserialize(data, result) Traceback (most recent call last): File "", line 1, in File "pyVmomi/SoapAdapter.py", line 831, in Deserialize ParseData(self.parser, response) File "pyVmomi/SoapAdapter.py", line 511, in ParseData reraise(ParserError, pe, tb) File "pyVmomi/SoapAdapter.py", line 498, in ParseData parser.Parse(data) File "pyVmomi/SoapAdapter.py", line 698, in StartElementHandler self.stack.append(GuessWsdlType(name)) File "pyVmomi/VmomiSupport.py", line 1066, in GuessWsdlType raise UnknownWsdlTypeError(name) pyVmomi.SoapAdapter.ParserError: 'xml document: <?xml version="1.0" encoding="UTF-8"?>\n\n soapenv:Body\n \n \n 10394972\n 10394972\n 2016-08-19T12:12:39.246999Z\n vsphere.local\Administrator\n Task: Create Library Item\n \n task-3564\n task-3564\n com.vmware.content.LibraryItem.Create\n clib-3553\n Test\n queued\n false\n true\n \n vsphere.local\Administrator\n \n 2016-08-19T12:12:39.246924Z\n 10394972\n 0c113bc6-65e7-4bd5-b1dc-9cfc6b176a6d\n \n \n \n 10394974\n 10394974\n 2016-08-19T12:12:41.091Z\n VSPHERE.LOCAL\Administrator\n Task: Upload Files to a Library Item\n \n task-3565\n task-3565\n com.vmware.content.LibraryItem.UploadContent\n clibitem-1424\n rpms_2.0.1-420.txt\n queued\n false\n true\n \n VSPHERE.LOCAL\Administrator\n \n 2016-08-19T12:12:41.091183Z\n 10394974\n 955b3353-5729-42dc-9a54-5aa64da24a62\n \n \n \n /soapenv:Body\n/soapenv:Envelope parse error at: line:15, col:45'

I am afraid I can't work on this this week. Will try to work on it next week. Sorry for the delay.

andrew-regan commented 8 years ago

Any update on this issue? Thanks!