search

dainok / netdoc

Automatic Network Documentation plugin for NetBox
GNU General Public License v3.0
89 stars 13 forks source link

Auto-discovery of VMs? #3

Closed dainok closed 1 year ago

dainok commented 1 year ago

Initially requested by @amdei

What I have:

  1. VMWare ESXi Cluster
  2. More than dozen of hosts in that cluster for VMs
  3. Each server host more that hundred VMs
  4. Each server has up to 20 D-switches
  5. Each server has up to 10 V-switches

What I need:

  1. Discover all Clusters/switches/hosts/VMs/Interfaces/IP-Addresses/connections between VMs

How I would like to do it:

  1. Specify credentials for each cluster in netdoc
  2. Specify addresses for each cluster in netdoc
  3. Press magic button
  4. ....
  5. Profit!

What I wan as a result:

  1. Populated lists of VMs
  2. Populated lists of VM's network interfaces
  3. Populated lists of V-switches
  4. Populated lists of D-switches
  5. Populated virtual cables, connecting components to each other.
  6. Possibly, populate some auxiliary information about VM: IPs, OS type, if VMWare tools running, etc.

Why:

  1. I desperately need L2 and L3 topology of all this mess. Evolving over time and completely undocumented.

What I can:

  1. I can develop (Python, in particular)
  2. I had some experience with VMWare ESX API
  3. I tried to improve netdoc once, but was overtaken by @dainok :-) what was anyway an excellent experience!

What I'm not sure about:

  1. Idea: How my ideas fit to nebox and netdoc philosophy. Is it a right place to introduce such kind of features?
  2. Tech: Where to put all discovered entities in netbox database. I've had a hard time trying to understand what all these required fields means and how to use them properly for creating single VM...

Would it be possible for wiser guys to give a piece of advice here on these uncertainties?

The hardest part is to resolve ideological issues. Implementation should not take long, and promised to be relatively straightforward.

dainok commented 1 year ago

I do not have NSX in my possession, only ESXi clusters. So, I guess L3 details could be obtained via results of previous effort - from IP address(es) on the VM. Mapping of VM NIC to OS network adapter name could be established via matching MAC addresses on both "ends".

Sure, I would like to collaborate!

But I need some guidance on both projects (netbox and netdoc) philosophy and terminology.

What I have done:

  1. Deploy latest versions of netbox and netdoc
  2. Import sample data from https://github.com/netbox-community/netbox-demo-data what allowed me to get familiar with places where to put assets details
  3. Import bunch of Device Types from https://github.com/netbox-community/devicetype-library via https://github.com/minitriga/Netbox-Device-Type-Library-Import what allowed me to make further progress of where to put other data
  4. Draw a mock-up plan of what need to be done (see below)

So far my plan looks like following: NetBox+NetDoc VMWare ESXi VMs autodiscovery

Thus, right now I have following open questions (to be updated)

  1. What netbox object should be created to represent connection from VM NIC to V-Switch? "Cable"?
  2. What would be best candidate for ESXi farm "discoverable" in netdoc? ESXi Host? Cluster? Whole Datacenter?
  3. Do we need to unconditionally create netdoc "Discoverable" for any populated VM?
  4. Do we need to unconditionally create netbox objects for V-Switches and D-Switches?
  5. Do we need to unconditionally create netbox objects for ESXi Hosts? Clusters? Datacenters?
  6. Where to put VLAN information form ESXi portgroups? NetBox/IPAM/VLANs?
  7. What about NetBox/IPAM/VLAN Groups? What are they for? Doe we need to put anything there?
dainok commented 1 year ago

Let's start from the beginning:

  1. I suggest to experiment with pyvmomi, once you have an idea of what you can do, it will be easier how to proceed.
  2. I guess cables, but I promise I'll give a look on that.
  3. Discoverable should be vCenters
  4. No, only vCenters and they should be manually created. There is no way to auto discover vCenters.
  5. Yes, but mind that dvSwitches are vSwitches centrally managed. From a L2 perspective dvSwitches and vSwitches are equals.
  6. Yes for ESXi Hosts, because they contains vSwitches and VMs. Clusters and datacenters are group of ESXi.
  7. Yes, VLANs.
  8. VLANs group are used by net engineers to logically group VLANs together: From a network perspective VLAN groups do not exists.

From vCenters you should retrieve:

Am I missing something?

dainok commented 1 year ago

@amdei i merged a first set of integration.

  1. Populated lists of VMs -> DONE
  2. Populated lists of VM's network interfaces -> DONE (L2 only, L3 requires running guest tools)
  3. Populated lists of V-switches
  4. Populated lists of D-switches
  5. Populated virtual cables, connecting components to each other.
  6. Possibly, populate some auxiliary information about VM: IPs, OS type, if VMWare tools running, etc.

Can you check? Do you have suggestion about how to get info using pyvmomi?

amdei commented 1 year ago

Great!

Will check what have been done ASAP.

dainok commented 1 year ago

To map ESXI networking into NetBox I would proceed as following:

image

I need a set of virtual switches to map:

I would group them using a NetBox virtual chassis. Let me know what you think about.