Pull important fields out of the device report and into the database

[sungo]

We currently keep the device report JSON blobs around "just in case" we need some data out of them. There are an immense amount of problems with this approach but we can't avoid the current reality that there is data in that JSON that is not available in the API, data that our users are currently accessing.

So, we need to figure out what report data is missing in the database, add tables etc, and make it available in the API.

[sungo]

From @jemershaw

Server reports

• interfaces.$interface_name.{ipaddr,mac,peer_port,peer_switch} -- I use this during multiple steps of the lifecycle integration, onsite, production
• interfaces.$interface_name.{state} -- just to tell if the physical layer is down or not to help create tickets for remote hands(integration, onsite, production?)
• dimms.{}.{memory-locator,memory-size,memory-bank-locator} - This sort of information is helpful when we are missing a dimm we can use this to tell what slot it is missing for better troubleshooting
• dimms.{}.{memory-manufacturer, memory-part-number, memory-serial-number} - This information will need to be used if we want to keep track of serial numbers for dimms so if we mark one as bad we can follow if it was moved and notify to not use it.
• disks.$disk_serial.{hba,model,size,drive_type,enclosure,slot} - This information is helpful when troubleshooting what disk is missing(more the physical topo of the system) -- We probably want to add this to the json spec so we know that the drives are always in the correct slot so then the ssd isn't in random slots on the same product.
• cpu.[].{microcode,model_name} -- I don't use this today, but I could see us using this in the specs if we want to make sure the cpu is patched to the correct microcode verison
• bios_version -- I'm not sure if we use this today, but I think this is what the firmware-agent uses iirc
• hba.$device_number.{slot,type} -- Physical topo of the hbas

Switch report

• lldp_neighbors.* -- This could probably be standardized into the interface record, but we just started to collect this information -- It is helpful to validate if the drd is plugged in correctly, but we will also gather this information from produciton switches at somepoint -- to help create a topology map
• mac_address_table.* -- This is the same as lldp_neighbors, but this will also give us a physically connecting report, but this infomration could be large since there could be multiple vms connected to one physical server.

[karenetheridge]

As part of this task I will write a script that will (attempt to) stomp through all existing device reports on file and attempt to backfill additional data into the database that it can figure out. Results may be limited, depending on how much the report format has changed.

[karenetheridge]

Since this involves formalization of a lot of data that has been so far been fairly loose in device reports, we should spec this out and ensure that the device report and database schemas are consistent and sensible.

[karenetheridge]

For v3 we need to figure out which information we want to send in device reports. All information passed in reports should be used for creating the device and running validations. Some data may also be saved in the database (but only if there is an endpoint to return it later upon request). Some of the data above falls into these categories; some do not; those that are already present in v2 reports that we do not want to use should be removed.

[karenetheridge]

There have been no requests for additional APIs to access data buried in the reports. We can add APIs, and copy data from reports into the database, on a case by case basis as such needs are identified. For now all I can assume is that the API spec and report schemas are now fixed for 3.0.0.