search

idaholab / Malcolm

Malcolm is a powerful, easily deployable network traffic analysis tool suite for full packet capture artifacts (PCAP files), Zeek logs and Suricata alerts.
https://idaholab.github.io/Malcolm/
Other
340 stars 58 forks source link

identify reason for differences in NetBox device types #397

Open mmguero opened 7 months ago

mmguero commented 7 months ago

On startup this script is used to preload the netbox device type library.

I've noticed that the number of device types seems to be slightly different on a few fresh installations of Malcolm (like, in the 2700's range but not exactly the same). Another box I have has about 2900.

We should determine the reason for the discrepancy and see if it needs to be addressed.

mmguero commented 6 months ago

The total number that seems to exist in the library (we remove the WatchGuard folder manually on the image build as it causes an error on import):

devicetype-library/device-types $ grep -Pr "^model:" | grep -v WatchGuard | wc -l
2936
mmguero commented 6 months ago

During a normal startup without any live capture, no running enrichment, etc., I have 2930 device types. We do consistently have a few errors:

netbox-1  | ... "POST /netbox/api/dcim/device-types/ HTTP/1.1" 400 47 ...
netbox-1  | ... "POST /netbox/api/dcim/device-types/ HTTP/1.1" 400 47 ...
netbox-1  | ... "POST /netbox/api/dcim/device-types/ HTTP/1.1" 400 47 ...
netbox-1  | ... "POST /netbox/api/dcim/device-types/ HTTP/1.1" 400 47 ...
netbox-1  | ... "POST /netbox/api/dcim/device-types/ HTTP/1.1" 400 47 ...
netbox-1  | ... "POST /netbox/api/dcim/device-types/ HTTP/1.1" 400 47 ...

However I don't think this is due to anything we're doing, and it is consistently reproducible. I think these are errors in the source data itself.

So seeing that there are 6 device types that got error 400s on them, and I ended up with exactly 6 less than what I counted as a "full" complement, this seems reasonable.

We also get some errors for some other records:

netbox-1  | 127.0.0.1 - - [13/Feb/2024:22:05:20 +0000] "POST /netbox/api/dcim/power-outlet-templates/ HTTP/1.1" 400 1089 "-" "python-requests/2.31.0"
netbox-1  | 127.0.0.1 - - [13/Feb/2024:22:18:34 +0000] "POST /netbox/api/dcim/module-types/ HTTP/1.1" 400 76 "-" "python-requests/2.31.0"
netbox-1  | 127.0.0.1 - - [13/Feb/2024:22:18:34 +0000] "POST /netbox/api/dcim/module-types/ HTTP/1.1" 400 76 "-" "python-requests/2.31.0"
netbox-1  | 127.0.0.1 - - [13/Feb/2024:22:18:37 +0000] "POST /netbox/api/dcim/module-types/ HTTP/1.1" 400 76 "-" "python-requests/2.31.0"

Again, I don't have an indication that this is really a problem, but is more likely an issue with the data.

I'm going to try this a few more times but if I get the same results (2930) I don't know what else we'd do about this.

mmguero commented 6 months ago

Because I don't think this is high priority, I'm going to bump it and get v24.02.0 out.