Duplicate changes of items that didn't change via source redfish

[koznov]

Hello! I'm using script to collect information every 2 hours from redfish source (HPe servers) to JSON files and then run a netbox-sync that using multiple configs (2 vCenters + redfish) to add info to NetBox: 15 /2 /opt/check_redfish/redfish-sync.sh >/dev/null 2>&1 23 /2 /opt/netbox-sync/.venv/bin/python3 /opt/netbox-sync/netbox-sync.py -c /opt/netbox-sync/redfish-config.ini /opt/netbox-sync/vcenter-new-config.ini /opt/netbox-sync/vcenter-old-config.ini >/dev/null 2>&1

In change log I see a lot of changes for hardware, but inside those changes only one thing that changes is added a duplicate tag to object:

Config of redfish source (redfish-config.ini):

[common]
log_level = DEBUG
log_to_file = True
log_file = /opt/netbox-sync/logs/netbox_sync.log

[netbox]
api_token = XXXX
host_fqdn = netbox.localdomain.local
port = 443
validate_tls_certs = False
prune_enabled = True
prune_delay_in_days = 7
timeout = 30
max_retry_attempts = 4
use_caching = True
cache_directory_location = cache

[source/redfish-sclabs]
enabled = True
type = check_redfish
inventory_file_path = /opt/check_redfish/inventory/
permitted_subnets = 10.137.0.0/24
overwrite_power_supply_name = True
overwrite_power_supply_attributes = True

Any ideas why we have duplicates of tags and how to prevent that? I'm trying to clean over 6 million records from change log :D

[bb-Ricardo]

Hey @koznov,

thank you for the bug report. Good question. Just wondering:

• any reason why you have split your config into multiple files?
• why it shows Source: redfish-SCLABS in the screenshot and redfish-sclabs in the config example?
• is a the same tag added on each run?
• can you run the script with -l DEBUG2 option and send me an example where NetBox gets updated with the same tag twice in on request. That might clarify things.

Thank you

[koznov]

Hey Ricardo,

Thanks for answering.

• any reason why you have split your config into multiple files? Only for having different configuration and easier to make changes in every file. I can join them together for easier debugging if you want :-)
• why it shows Source: redfish-SCLABS in the screenshot and redfish-sclabs in the config example? That's a good question and I cannot answer on it. Config that is used for redfish sync has string [source/redfish-sclabs]:

• is a the same tag added on each run? Yes, all changes that are made related to redfish has the same pattern with adding the same tag to objects with Type Inventory Item (RAM, PSU, NIC, etc) - all objects that are taken from json files of servers.

• can you run the script with -l DEBUG2 option and send me an example where NetBox gets updated with the same tag twice in on request. That might clarify things. Sure, attaching full log of command: /opt/netbox-sync/.venv/bin/python3 /opt/netbox-sync/netbox-sync.py -c /opt/netbox-sync/redfish-config.ini /opt/netbox-sync/vcenter-sclabs-config.ini /opt/netbox-sync/vcenter-norislab-config.ini -l DEBUG2 > /home/sa-netbox@sclabs.cloud/issue-363.log

[koznov]

Maybe should I try to clear cache folder and then try again to sync, and check after that again?

[bb-Ricardo]

Hi,

I checked your log and it's rather strange and should not happen. Also curios why the source tag character case in name changes. Maybe you would like to try out the latest development version.

[koznov]

No luck,

Cloned development branch, copied configs, and run once and then again - same result. Logs are attached.

[koznov]

I suppose that duplicating also related not only to Redfish, but to vCenter source as well: Adding also configs for analyzing:

akoznov@netbox:~$ egrep -v '^;|^$' /opt/netbox-sync/redfish-config.ini
[common]
log_level = DEBUG
log_to_file = True
log_file = /opt/netbox-sync/logs/netbox_sync.log
[netbox]
api_token = XXX
host_fqdn = netbox.sclabs.cloud
port = 443
validate_tls_certs = False
prune_enabled = True
prune_delay_in_days = 7
timeout = 30
max_retry_attempts = 4
use_caching = True
cache_directory_location = cache
[source/redfish-sclabs]
enabled = True
type = check_redfish
inventory_file_path = /opt/check_redfish/inventory/
permitted_subnets = 10.137.0.0/24
overwrite_power_supply_name = True
overwrite_power_supply_attributes = True
akoznov@netbox:~$ egrep -v '^;|^$' /opt/netbox-sync/vcenter-sclabs-config.ini
[common]
log_level = DEBUG
log_to_file = True
log_file = /opt/netbox-sync/logs/netbox_sync.log
[netbox]
api_token = XXX
host_fqdn = netbox.sclabs.cloud
port = 443
validate_tls_certs = False
prune_enabled = True
prune_delay_in_days = 7
timeout = 30
max_retry_attempts = 4
use_caching = True
cache_directory_location = cache
[source/vcenter-sclabs]
enabled = True
type = vmware
host_fqdn = vc01.sclabs.cloud
port = 443
username = sa-vcenter@sclabs.cloud
password = XXX
validate_tls_certs = False
permitted_subnets = 10.137.0.0/20, 10.137.32.0/19, 10.138.0.0/22, 10.137.64.0/24, 10.137.65.0/24, 10.138.0.0/22, 10.138.4.0/22, 10.139.0.0/16, 10.140.0.0/20, 172.22.64.0/22
cluster_site_relation = Noris/.* = NBG6ITA
cluster_tenant_relation = .* = Noris Lab
host_role_relation = .* = Hypervisor
vm_role_relation = .* = VM
match_host_by_serial = True
collect_hardware_asset_tag = True
dns_name_lookup = True
custom_dns_servers = 10.137.1.50, 10.137.1.51
set_primary_ip = when-undefined
skip_vm_comments = False
skip_vm_templates = True
skip_offline_vms = False
skip_srm_placeholder_vms = False
strip_host_domain_name = False
strip_vm_domain_name = False
cluster_tag_source = object
host_tag_source = object, cluster
vm_tag_source = object, cluster
sync_custom_attributes = True
host_custom_object_attributes = summary.runtime.bootTime
vm_custom_object_attributes = config.uuid
sync_vm_dummy_interfaces = True
disable_vlan_sync = False
track_vm_host = False
overwrite_device_interface_name = True
host_management_interface_match = management, mgmt
ip_tenant_inheritance_order = device, prefix
sync_vm_interface_mtu = True
akoznov@netbox:~$ egrep -v '^;|^$' /opt/netbox-sync/vcenter-norislab-config.ini
[common]
log_level = DEBUG
log_to_file = True
log_file = /opt/netbox-sync/logs/netbox_sync.log
[netbox]
api_token = XXX
host_fqdn = netbox.sclabs.cloud
port = 443
validate_tls_certs = False
prune_enabled = True
prune_delay_in_days = 7
timeout = 30
max_retry_attempts = 4
use_caching = True
cache_directory_location = cache
[source/vcenter-norislab]
enabled = True
type = vmware
host_fqdn = vc-1.vdi.sclabs.net
port = 443
username = sa-vcenter@vdi.sclabs.net
password = XXX
validate_tls_certs = False
permitted_subnets = 172.16.0.0/12, 10.0.0.0/8, 192.168.0.0/16, fd00::/8, 192.168.0.0/16, fd00:bade:fee:1::/64
cluster_site_relation = Noris/.* = NBG6ITA
cluster_tenant_relation = .* = Noris Lab
host_role_relation = .* = Hypervisor
vm_role_relation = .* = VM
match_host_by_serial = True
collect_hardware_asset_tag = True
dns_name_lookup = True
custom_dns_servers = 10.0.30.20, 10.0.20.61
set_primary_ip = when-undefined
skip_vm_comments = False
skip_vm_templates = True
skip_offline_vms = False
skip_srm_placeholder_vms = False
strip_host_domain_name = False
strip_vm_domain_name = False
cluster_tag_source = object
host_tag_source = object, cluster
vm_tag_source = object, cluster
sync_custom_attributes = True
host_custom_object_attributes = summary.runtime.bootTime
vm_custom_object_attributes = config.uuid
sync_vm_dummy_interfaces = True
disable_vlan_sync = False
track_vm_host = False
overwrite_device_interface_name = True
host_management_interface_match = management, mgmt
ip_tenant_inheritance_order = device, prefix
sync_vm_interface_mtu = True

[koznov]

Now I'm manually deleting all Tags that created by netbox-sync, clearing the cache and try to resync couple of the times. Come back with some feedback.

[koznov]

After removing tags, clearing the cache and rerunning sync a couple of the times, only the changed items are changing inside NetBox. Thanks!

[bb-Ricardo]

awesome