search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.15k stars 4.91k forks source link

Add support for vsphere performance metrics in metricbeat #20313

Open jsoriano opened 4 years ago

jsoriano commented 4 years ago

Current metricbeat module for vsphere uses the "quick stats" metrics available in the API objects for virtual machines, host information and so on. vsphere SDKs also expose a set of performance metrics that expose many more metrics. We could complete vsphere support with these metrics.

We could review if this would replace the current metricsets, and/or if this could be implemented as new metricsets.

Go library we are using includes an example to collect these metrics: https://github.com/vmware/govmomi/blob/master/performance/example_test.go, with this code metrics like these ones can be collected.

This would help on the following issues:

elasticmachine commented 4 years ago

Pinging @elastic/integrations (Team:Integrations)

IshanOhri commented 3 years ago

Any update on adding the above fields to the Metricbeat vSphere module?

IshanOhri commented 3 years ago

@jsoriano - Do you know if work being done on adding the performance metrics?

blambo10 commented 3 years ago

@jsoriano @IshanOhri

I have completely rewritten the current module in my local instance to leverage the performance metric api instead of the quickstats.

I raised a question within the elastic forum attempting to gauge the community desire for such code before contributing back.

The module I have written, allows for specifying any of the performance metric counter names from the config yaml for the module, then it will iterate the manage object and request all specified metrics.

Additionally it allows the user to specify a retrospective start time and interval ID to allow someone to configure the beat to poll the vsphere api at a given interval in an attempt to mitigate VC load in large environments, it defaults to realtime.

Judging by this Open issue id assume the desire is there, though the fact i have written it to leverage the metrics api completely and no longer any quick stats, im thinking maybe another module all together "vsphere-perf-metrics" perhaps

Thoughts?

jsoriano commented 3 years ago

Hey @blambo10,

Yes, there is some interest in the community for improvements on this module, as you can see in the comments and issues linked in this issue. So it'd be great if you would like to contribute a more complete module.

i have written it to leverage the metrics api completely and no longer any quick stats, im thinking maybe another module all together "vsphere-perf-metrics" perhaps

There would be some options for that:

Probably the best option is to open a PR with your module, and we can discuss there the specifics of the implementation and how it could be released.

Thanks!

blambo10 commented 3 years ago

Hi,

I havent forgot this, ive just been busy, im aiming to get back around to raising the PR with the refactored code soon.

Thanks, Bryce

bmbrit commented 3 years ago

Hi @jsoriano,

I would like to see IOPS and Total Latency for the Datastore metricset, Throughput and Network like Current Send/Receive KBps, Interface Name of Disk's of the Virtual Host and the same for the Host. I tried the 7.15(latest now) metric beat, which claimed to have these details which looks missing. Also, there was a rumour that there is a metricset called "Performance" which could provide a lot of performance based on VSphere exposed metrics.

Please clarify, if there is anyway we can pull the above mentioned details or the claimed Performance metricset.

Thanks, Britto

jsoriano commented 3 years ago

Hey @bmbrit, I am not aware of any recent change in the datastore metricset adding the information you mention. The only fields that have been recently added are the ones for total and free CPU (#26167). There is no "performance" metricset for vsphere.

svanschalkwyk commented 2 years ago

@jsoriano @IshanOhri

I have completely rewritten the current module in my local instance to leverage the performance metric api instead of the quickstats.

I raised a question within the elastic forum attempting to gauge the community desire for such code before contributing back.

The module I have written, allows for specifying any of the performance metric counter names from the config yaml for the module, then it will iterate the manage object and request all specified metrics.

Additionally it allows the user to specify a retrospective start time and interval ID to allow someone to configure the beat to poll the vsphere api at a given interval in an attempt to mitigate VC load in large environments, it defaults to realtime.

Judging by this Open issue id assume the desire is there, though the fact i have written it to leverage the metrics api completely and no longer any quick stats, im thinking maybe another module all together "vsphere-perf-metrics" perhaps

Thoughts?

Hi Jaime Is there a gist with your code which we can use to add to the codebase? Beats having to reinvent the wheel. Steph

jsoriano commented 2 years ago

Hey @svanschalkwyk,

Is there a gist with your code which we can use to add to the codebase? Beats having to reinvent the wheel.

I don't have additional code apart of what is already in the repo :slightly_smiling_face:

@blambo10 mentioned to have something, but needed some refactors.

As far as I know there is no open PR about supporting the performance metric API. Would you be interested on working on this?

svanschalkwyk commented 2 years ago

Hi Jaime How did you manage to pull the metrics in https://gist.github.com/jsoriano/c759597aebd0ebba6ecb9ad0ec995336 from govmomi? I can only get the following: Kind regards, Steph



datastores:

Name: Type: Capacity: Free:
LocalDS_0 OTHER 961.2GB 417.9GB

events:
11 [Tue Dec 14 11:58:36 2021] [DatacenterCreatedEvent] Created datacenter
DC0 in folder Datacenters 22 [Tue Dec 14 11:58:36 2021]
[VmBeingCreatedEvent] Creating DC0_H0_VM0 on host DC0_H0 in DC0 33 [Tue Dec
14 11:58:36 2021] [VmInstanceUuidAssignedEvent] Assign a new instance UUID
(b4689bed-97f0-5bcd-8a4c-07477cc8f06f) to DC0_H0_VM0 44 [Tue Dec 14
11:58:36 2021] [VmUuidAssignedEvent] Assigned new BIOS UUID
(265104de-1472-547c-b873-6dc7883fb6cb) to DC0_H0_VM0 on DC0_H0 in DC0 55
[Tue Dec 14 11:58:36 2021] [VmCreatedEvent] Created virtual machine
DC0_H0_VM0 on DC0_H0 in DC0 66 [Tue Dec 14 11:58:36 2021] [VmStartingEvent]
DC0_H0_VM0 on host DC0_H0 in DC0 is starting 77 [Tue Dec 14 11:58:36 2021]
[VmPoweredOnEvent] DC0_H0_VM0 on DC0_H0 in DC0 is powered on 88 [Tue Dec 14
11:58:36 2021] [VmBeingCreatedEvent] Creating DC0_H0_VM1 on host DC0_H0 in
DC0 99 [Tue Dec 14 11:58:36 2021] [VmInstanceUuidAssignedEvent] Assign a
new instance UUID (12f8928d-f144-5c57-89db-dd2d0902c9fa) to DC0_H0_VM1 1010
[Tue Dec 14 11:58:36 2021] [VmUuidAssignedEvent] Assigned new BIOS UUID
(39365506-5a0a-5fd0-be10-9586ad53aaad) to DC0_H0_VM1 on DC0_H0 in DC0 1111
[Tue Dec 14 11:58:36 2021] [VmCreatedEvent] Created virtual machine
DC0_H0_VM1 on DC0_H0 in DC0 1212 [Tue Dec 14 11:58:36 2021]
[VmStartingEvent] DC0_H0_VM1 on host DC0_H0 in DC0 is starting 1313 [Tue
Dec 14 11:58:36 2021] [VmPoweredOnEvent] DC0_H0_VM1 on DC0_H0 in DC0 is
powered on 1414 [Tue Dec 14 11:58:36 2021] [VmBeingCreatedEvent] Creating
DC0_C0_RP0_VM0 on host DC0_C0_H1 in DC0 1515 [Tue Dec 14 11:58:36 2021]
[VmInstanceUuidAssignedEvent] Assign a new instance UUID
(bfff331f-7f07-572d-951e-edd3701dc061) to DC0_C0_RP0_VM0 1616 [Tue Dec 14
11:58:36 2021] [VmUuidAssignedEvent] Assigned new BIOS UUID
(cd0681bf-2f18-5c00-9b9b-8197c0095348) to DC0_C0_RP0_VM0 on DC0_C0_H1 in
DC0 1717 [Tue Dec 14 11:58:36 2021] [VmCreatedEvent] Created virtual
machine DC0_C0_RP0_VM0 on DC0_C0_H1 in DC0 1818 [Tue Dec 14 11:58:36 2021]
[VmStartingEvent] DC0_C0_RP0_VM0 on host DC0_C0_H1 in DC0 is starting 1919
[Tue Dec 14 11:58:36 2021] [VmPoweredOnEvent] DC0_C0_RP0_VM0 on DC0_C0_H1
in DC0 is powered on 2020 [Tue Dec 14 11:58:36 2021] [VmBeingCreatedEvent]
Creating DC0_C0_RP0_VM1 on host DC0_C0_H0 in DC0 2121 [Tue Dec 14 11:58:36
2021] [VmInstanceUuidAssignedEvent] Assign a new instance UUID
(6132d223-1566-5921-bc3b-df91ece09a4d) to DC0_C0_RP0_VM1 2222 [Tue Dec 14
11:58:36 2021] [VmUuidAssignedEvent] Assigned new BIOS UUID
(f7c371d6-2003-5a48-9859-3bc9a8b08908) to DC0_C0_RP0_VM1 on DC0_C0_H0 in
DC0 2323 [Tue Dec 14 11:58:36 2021] [VmCreatedEvent] Created virtual
machine DC0_C0_RP0_VM1 on DC0_C0_H0 in DC0 2424 [Tue Dec 14 11:58:36 2021]
[VmStartingEvent] DC0_C0_RP0_VM1 on host DC0_C0_H0 in DC0 is starting 2525
[Tue Dec 14 11:58:36 2021] [VmPoweredOnEvent] DC0_C0_RP0_VM1 on DC0_C0_H0
in DC0 is powered on 2626 [Tue Dec 14 11:58:36 2021]
[UserLoginSessionEvent] User ***@***.*** logged in as Go-http-client/1.1
27

hosts:
1Name: Used CPU: Total CPU: Free CPU: Used Memory: Total Memory: Free
Memory: 2DC0_H0 67 4588 4521 1.4GB 4.0GB 2822230016 3DC0_C0_H0 67 4588 4521
1.4GB 4.0GB 2822230016 4DC0_C0_H1 67 4588 4521 1.4GB 4.0GB 2822230016
5DC0_C0_H2
67 4588 4521 1.4GB 4.0GB 2822230016 6

networks:
1VM Network: Network:network-7 2DVS0-DVUplinks-9:
DistributedVirtualPortgroup:dvportgroup-11 3DC0_DVPG0:
DistributedVirtualPortgroup:dvportgroup-13 4

perfmanager:
1VirtualMachine:vm-57 * cpu.demand.average 292 MHz 2VirtualMachine:vm-57 *
cpu.demandEntitlementRatio.latest 627.91 % 3VirtualMachine:vm-57 *
cpu.entitlement.latest 50 MHz 4VirtualMachine:vm-57 * cpu.idle.summation
44539 ms 5VirtualMachine:vm-57 * cpu.latency.average 0.7 %
6VirtualMachine:vm-57
* cpu.overlap.summation 13 ms 7VirtualMachine:vm-57 * cpu.readiness.average
0.37 % 8VirtualMachine:vm-57 * cpu.ready.summation 175 ms 9VirtualMachine:vm-57
* cpu.run.summation 2662 ms 10VirtualMachine:vm-57 * cpu.usage.average 6.86
% 11VirtualMachine:vm-57 * cpu.usagemhz.average 321 MHz 12VirtualMachine:vm-57
* cpu.used.summation 2770 ms 13VirtualMachine:vm-57 * cpu.wait.summation
45742 ms 14VirtualMachine:vm-57 * disk.read.average 127 KBps
15VirtualMachine:vm-57
* disk.usage.average 295 KBps 16VirtualMachine:vm-57 * disk.write.average
155 KBps 17VirtualMachine:vm-57 * mem.active.average 2472829 KB
18VirtualMachine:vm-57
* mem.activewrite.average 3038212 KB 19VirtualMachine:vm-57 *
mem.consumed.average 8130044 KB 20VirtualMachine:vm-57 *
mem.entitlement.average 7445004 KB 21VirtualMachine:vm-57 *
mem.granted.average 12430445 KB 22VirtualMachine:vm-57 *
mem.overhead.average 46485 KB 23VirtualMachine:vm-57 *
mem.overheadMax.average 71269 KB 24VirtualMachine:vm-57 *
mem.overheadTouched.average 50864 KB 25VirtualMachine:vm-57 *
mem.usage.average 31.5 % 26VirtualMachine:vm-57 * net.broadcastRx.summation
44 num 27VirtualMachine:vm-57 * net.bytesRx.average 428 KBps
28VirtualMachine:vm-57
* net.bytesTx.average 647 KBps 29VirtualMachine:vm-57 *
net.multicastRx.summation 24 num 30VirtualMachine:vm-57 *
net.packetsRx.summation 3196 num 31VirtualMachine:vm-57 *
net.packetsTx.summation 1608 num 32VirtualMachine:vm-57 *
net.pnicBytesRx.average 452 KBps 33VirtualMachine:vm-57 *
net.pnicBytesTx.average 612 KBps 34VirtualMachine:vm-57 *
net.received.average 296 KBps 35VirtualMachine:vm-57 *
net.transmitted.average 388 KBps 36VirtualMachine:vm-57 * net.usage.average
1022 KBps 37VirtualMachine:vm-57 * rescpu.actav1.latest 8.54 %
38VirtualMachine:vm-57
* rescpu.actav5.latest 15.96 % 39VirtualMachine:vm-57 *
rescpu.actpk1.latest 17.45 % 40VirtualMachine:vm-57 * rescpu.actpk15.latest
26.34 % 41VirtualMachine:vm-57 * rescpu.actpk5.latest 15.3 %
42VirtualMachine:vm-57
* rescpu.runav1.latest 9.06 % 43VirtualMachine:vm-57 *
rescpu.runav15.latest 10.82 % 44VirtualMachine:vm-57 * rescpu.runav5.latest
13.35 % 45VirtualMachine:vm-57 * rescpu.runpk1.latest 21.67 %
46VirtualMachine:vm-57
* rescpu.runpk15.latest 16.06 % 47VirtualMachine:vm-57 *
rescpu.runpk5.latest 22.73 % 48VirtualMachine:vm-57 *
rescpu.sampleCount.latest 215 num 49VirtualMachine:vm-57 *
rescpu.samplePeriod.latest 6908 ms 50VirtualMachine:vm-57 *
sys.heartbeat.latest 32 num 51VirtualMachine:vm-57 * sys.osUptime.latest
1920375 s 52VirtualMachine:vm-57 * sys.uptime.latest 3153129 s
53VirtualMachine:vm-57
* virtualDisk.read.average 1 KBps 54VirtualMachine:vm-57 *
virtualDisk.write.average 122 KBps 55

virtualmachines:
1DC0_H0_VM0: otherGuest 2DC0_H0_VM1: otherGuest 3DC0_C0_RP0_VM0: otherGuest
4DC0_C0_RP0_VM1: otherGuest 5

On Thu, Dec 9, 2021 at 9:51 AM Jaime Soriano Pastor <
***@***.***> wrote:

> Hey @svanschalkwyk <https://github.com/svanschalkwyk>,
>
> Is there a gist with your code which we can use to add to the codebase?
> Beats having to reinvent the wheel.
>
> I don't have additional code apart of what is already in the repo 🙂
>
> @blambo10 <https://github.com/blambo10> mentioned to have something, but
> needed some refactors.
>
> As far as I know there is no open PR about supporting the performance
> metric API. Would you be interested on working on this?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/elastic/beats/issues/20313#issuecomment-989978205>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AABTCTGGAESQOQDCHOZ6RNDUQDGAVANCNFSM4PLSWOYQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
svanschalkwyk commented 2 years ago

@blambo10 would you be interested in sharing what you have? I'm adding perfCounters.

jsoriano commented 2 years ago

How did you manage to pull the metrics in https://gist.github.com/jsoriano/c759597aebd0ebba6ecb9ad0ec995336 from govmomi?

Hey @svanschalkwyk, sorry for my late reply, I have been a bit off the grid during the last month :slightly_smiling_face:

If I remember correctly I adapted this part of the test to print everything: https://github.com/vmware/govmomi/blob/18f21c33e5e783d9c8e0d431b6b3e37a3c58b7a3/performance/example_test.go#L97-L100. And processed a bit the output to sort it and so on.