[Metricbeat] - vsphere module request

[EcHoOfInSaNiTy]

Hello,

I love the vsphere module that was added to metricbeat, however, there aren't enough different kinds of metrics being pulled for me to move away from my current metric collection methods. Would it be possible to include metrics that for example Zabbix can pull by default? This would be huge for me in terms of moving everything over to metricbeat for data collection.

I.E: _Where hv is hyper-visor metrics and vm is virtual-machine metrics__ vmware.hv.cluster.name vmware.hv.cpu.usage vmware.hv.datacenter.name vmware.hv.fullname[ vmware.hv.hw.cpu.freq vmware.hv.hw.cpu.model vmware.hv.hw.cpu.num vmware.hv.hw.cpu.threads vmware.hv.hw.memory vmware.hv.hw.model vmware.hv.hw.uuid vmware.hv.hw.vendor vmware.hv.memory.size.ballooned vmware.hv.memory.used vmware.hv.network.in vmware.hv.network.out vmware.hv.sensor.health.state vmware.hv.status vmware.hv.uptime vmware.hv.version vmware.hv.vm.num vmware.hv.datastore.read vmware.hv.datastore.write vmware.hv.datastore.size vmware.hv.datastore.size vmware.hv.cluster.name vmware.hv.cpu.usage vmware.hv.datacenter.name vmware.hv.fullname vmware.hv.hw.cpu.freq vmware.hv.hw.cpu.model vmware.hv.hw.cpu.num vmware.hv.hw.cpu.threads vmware.hv.hw.memory vmware.hv.hw.model vmware.hv.hw.uuid vmware.hv.hw.vendor vmware.hv.memory.size.ballooned vmware.hv.memory.used vmware.hv.network.in vmware.hv.network.out vmware.hv.sensor.health.state vmware.hv.status vmware.hv.uptime vmware.hv.version vmware.hv.vm.num vmware.vm.cluster.name vmware.vm.cpu.num vmware.vm.cpu.ready vmware.vm.cpu.usage vmware.vm.datacenter.name vmware.vm.hv.name vmware.vm.memory.size.ballooned vmware.vm.memory.size.compressed vmware.vm.memory.size.private vmware.vm.memory.size.shared vmware.vm.memory.size.swapped vmware.vm.memory.size.usage.guest vmware.vm.memory.size.usage.host vmware.vm.memory.size vmware.vm.powerstate vmware.vm.storage.committed vmware.vm.storage.uncommitted vmware.vm.storage.unshared vmware.vm.uptime vmware.cluster.status

[ruflin]

It seem for the virtual machine metrics we already have a subset on what you suggest above. This means here the number of metrics which are collected would have to be expanded.

We don't have a hyper-visor metricset yet. This would have to be created. The key here is if this API is exposed by the github.com/vmware/govmomi library we use.

Want to try to contribute the above metrics? You seem to have all the environment already available which is needed for manual testing.

[EcHoOfInSaNiTy]

Sure!

[PauliniMrtns]

This a cool topic, I need this as well. I would appreciate some help from the community with this because I too need this implementation for VMWare monitoring!

[MnrGreg]

@cblomart has done a great deal of work collecting vsphere metrics using GOVMOMI and the vSphere Metric Groups / perfManager method: vsphere.go

With this implementation one is able to define a list metrics deemed relevant to the administrator. One could apply this same logic to the MetricBeats vsphere module by adding a parameter similar to get_custom_fields.

For example:

metricbeat.modules:
- module: vsphere
   get_custom_metrics:
     - cpu.ready.summation
     - virtualDisk.totalWriteLatency.average
     - virtualDisk.totalReadLatency.average

I expect there would be a significant increase in adoption of Elastic/MetricBeats if this could be implemented. Large organizations are always looking to rationalize and standardize their portfolio of tools. In terms of on-premise hypervisor deployments VMware must be in the top 3.

This would presumably be more efficient then collecting OS level metrics from the MetricBeats System agent installed within the OS. In a few scenarios this eliminates the need for the MetricBeats OS agent altogether.

To get a true performance monitoring picture, such as evaluating CPU oversubscription, one will always need the hypervisor level stats.

I would love to see the community jump onto this. Also hopeful that Elasticsearch BV can dedicate some people!?

Addresses #5180

[MnrGreg]

For those interested, we've added an Elastic backend to vsphere-graphite. With this binary one is able to specify which metrics should be collected from vSphere's realtime performance counters. This includes useful metrics like guest disk capacity usage and guest CPU Ready.

Grafana dashboards also available here

[botelastic[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[gabrieletosca]

Hi everyone! I see that this issues has been reopened great! So i have only one question for you guys, I would be interested in collecting the cpu ready time metric from the vCenter to put the data in enough to elastic. I'm looking around how to do it (I'm not very experienced) do you have any updates or advice?

[jsoriano]

Hi everyone! I see that this issues has been reopened great! So i have only one question for you guys, I would be interested in collecting the cpu ready time metric from the vCenter to put the data in enough to elastic. I'm looking around how to do it (I'm not very experienced) do you have any updates or advice?

Hey @gabrieletosca, current vsphere module is able to collect used CPU per virtualmachine (see example document here). If you need more performance metrics, https://github.com/elastic/beats/issues/20313 would need to be implemented.