Sync failed

[jannikk09]

Hello, I followed the instructions and installed the plugin.
The connection to Netbox and vCenter works. Now I have this error:`

2023-10-26 12:06:07,891 - INFO: Starting NetBox Sync v1.5.1 (2023-09-25)
2023-10-26 12:06:07,995 - INFO: Successfully connected to NetBox 'srv-app03.domain.de'
2023-10-26 12:06:07,995 - INFO: Initializing sources
2023-10-26 12:06:08,080 - INFO: Successfully connected to vCenter SDK 'srv-vc.domain.de'
2023-10-26 12:06:08,080 - INFO: Querying necessary objects from NetBox. This might take a while.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/netbox-sync/netbox-sync.py", line 143, in <module>
    main()
  File "/opt/netbox-sync/netbox-sync.py", line 98, in main
    nb_handler.query_current_data(source.dependent_netbox_objects)
  File "/opt/netbox-sync/module/netbox/connection.py", line 461, in query_current_data
    full_nb_data = self.request(nb_object_class)
  File "/opt/netbox-sync/module/netbox/connection.py", line 282, in request
    result["results"].extend(response.json().get("results"))
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Here is my settings.ini:

;;; Welcome to the NetBox Sync configuration file.
;;; Version: 1.5.1 (2023-09-25)
;;; Project URL: https://github.com/bb-ricardo/netbox-sync

; The values in this file override the default values used by the system if a config
; option is not specified. The commented out lines are the configuration field and the
; default value used. Uncommenting a line and changing the value will change the value
; used at runtime when the process is restarted.

;;;
;;; [common]
;;;
;;; Controls the parameters for logging
;;;

[common]

; Logs will always be printed to stdout/stderr.
; Logging can be set to following log levels:
;   ERROR:      Fatal Errors which stops regular a run
;   WARNING:    Warning messages won't stop the syncing process but mostly worth
;               to have a look at.
;   INFO:       Information about objects that will be create/updated/deleted in NetBox
;   DEBUG:      Will log information about retrieved information, changes in internal
;               data structure and parsed config
;   DEBUG2:     Will also log information about how/why data is parsed or skipped.
;   DEBUG3:     Logs all source and NetBox queries/results to stdout. Very useful for
;               troubleshooting, but will log any sensitive data contained within a query.
;log_level = INFO

; Enabling this options will write all logs to a log file defined in 'log_file'
;log_to_file = False

; Destination of the log file if "log_to_file" is enabled. Log file will be rotated
; maximum 5 times once the log file reaches size of 10 MB
;log_file = log/netbox_sync.log

;;;
;;; [netbox]
;;;
;;; Controls the connection parameters to your netBox instance
;;;

[netbox]

; Requires an NetBox API token with full permissions on all objects except 'auth',
; 'secrets' and 'users'
api_token = secret_api_key

; Requires a hostname or IP which points to your NetBox instance
host_fqdn = srv-app03.domain.de

; Define the port your NetBox instance is listening on. If 'disable_tls' is set to "true"
; this option might be set to 80
port = 443

; Whether TLS encryption is enabled or disabled
disable_tls = False

; Enforces TLS certificate validation. If this system doesn't trust the NetBox web server
; certificate then this option needs to be changed
validate_tls_certs = True

; Defines a proxy which will be used to connect to NetBox. Proxy setting needs to include
; the schema. Proxy basic auth example: http://user:pass@10.10.1.10:312
;proxy = http://example.com:3128

; Specify a client certificate which can be used to authenticate to NetBox
;client_cert = client.pem

; Specify the client certificate private key belonging to the client cert
;client_cert_key = client.key

; Whether items which were created by this program but can't be found in any source
; anymore will be deleted or not
prune_enabled = true

; Orphaned objects will first be tagged before they get deleted. Once the amount of days
; passed the object will actually be deleted
;prune_delay_in_days = 30

; This will tell netbox-sync to ignore objects in NetBox with tag 'NetBox-synced' from
; pruning if the source is not defined in this config file (https://github.com/bb-
; Ricardo/netbox-sync/issues/176)
;ignore_unknown_source_object_pruning = False

; The maximum number of objects returned in a single request. If a NetBox instance is very
; quick responding the value should be raised
;default_netbox_result_limit = 200

; The maximum time a query is allowed to execute before being killed and considered failed
;timeout = 30

; The amount of times a failed request will be reissued. Once the maximum is reached the
; syncing process will be stopped completely.
;max_retry_attempts = 4

; Defines if caching of NetBox objects is used or not. If problems with unresolved
; dependencies occur, switching off caching might help.
;use_caching = True

; The location of the directory where the cache files should be stored
;cache_directory_location = cache

;;;
;;; [source/*]
;;;
;;; Controls the parameters of a defined source. The string past the slash will be used as
;;; a sources name. Sources can be defined multiple times to represent different sources.
;;;

[source/my-vcenter-example]

; Defines if this source is enabled or not
enabled = True

; type of source. This defines which source handler to use
type = vmware

; host name / IP address of the vCenter
host_fqdn = srv-vc.domain.de

; TCP port to connect to
;port = 443

; username to use to log into vCenter
username = readonly@vsphere.local

; password to use to log into vCenter
password = secret_password

; Enforces TLS certificate validation. If vCenter uses a valid TLS certificate then this
; option should be set to 'true' to ensure a secure connection.
;validate_tls_certs = False

; EXPERIMENTAL: Connect to a vCenter using a proxy server (socks proxies are not
; supported). define a host name or an IP address
;proxy_host = 10.10.1.10

; EXPERIMENTAL: Connect to a vCenter using a proxy server (socks proxies are not
; supported). define proxy server port number
;proxy_port = 3128

; IP networks eligible to be synced to NetBox. If an IP address is not part of this
; networks then it WON'T be synced to NetBox. To excluded small blocks from bigger IP
; blocks a leading '!' has to be added
permitted_subnets = 192.168.16.0/24

; filter options

; filters can be used to include/exclude certain objects from importing into NetBox.
; Include filters are checked first and exclude filters after. An object name has to pass
; both filters to be synced to NetBox. If a filter is unset it will be ignored. Filters
; are all treated as regex expressions! If more then one expression should match, a '|'
; needs to be used
;
; Example: (exclude all VMs with "replica" in their name and all VMs starting with
; "backup"): vm_exclude_filter = .*replica.*|^backup.*

; If a cluster is excluded from sync then ALL VMs and HOSTS inside the cluster will be
; ignored! a cluster can be specified as "Cluster-name" or "Datacenter-name/Cluster-name"
; if multiple clusters have the same name
;cluster_exclude_filter =
;cluster_include_filter =

; This will only include/exclude the host, not the VM if Host is part of a multi host
; cluster
;host_exclude_filter =
;host_include_filter =

; simply include/exclude VMs
;vm_exclude_filter =
;vm_include_filter =

; relations options

; This option defines which vCenter cluster is part of a NetBox site.
; This is done with a comma separated key = value list.
;   key: defines the cluster name as regex
;   value: defines the NetBox site name (use quotes if name contains commas)
; This is a quite important config setting as IP addresses, prefixes, VLANs
; and VRFs are site dependent. In order to assign the correct prefix to an IP
; address it is important to pick the correct site.
; A VM always depends on the cluster site relation
; a cluster can be specified as "Cluster-name" or
; "Datacenter-name/Cluster-name" if multiple clusters have the same name
;cluster_site_relation = Cluster_NYC = New York, Cluster_FFM.* = Frankfurt, Datacenter_TOKIO/.* = Tokio

; Same as cluster site but on host level. If unset it will fall back to
; cluster_site_relation
;host_site_relation = nyc02.* = New York, ffm01.* = Frankfurt

; This option defines which cluster/host/VM belongs to which tenant.
; This is done with a comma separated key = value list.
;   key: defines a hosts/VM name as regex
;   value: defines the NetBox tenant name (use quotes if name contains commas)
; a cluster can be specified as "Cluster-name" or
; "Datacenter-name/Cluster-name" if multiple clusters have the same name
;cluster_tenant_relation = Cluster_NYC.* = Customer A
;host_tenant_relation = esxi300.* = Infrastructure
;vm_tenant_relation = grafana.* = Infrastructure

; This option defines custom platforms if the VMWare created platforms are not suitable.
; Pretty much a mapping of VMWare platform name to your own platform name.
; This is done with a comma separated key = value list.
;   key: defines a VMWare returned platform name
;   value: defines the desired NetBox platform name
;vm_platform_relation = centos-7.* = centos7, microsoft-windows-server-2016.* = Windows2016

; Define the NetBox device role used for hosts. The default is
; set to "Server". This is done with a comma separated key = value list.
;   key: defines host(s) name as regex
;   value: defines the NetBox role name (use quotes if name contains commas)
;host_role_relation = .* = Server

; Define the NetBox device role used for VMs. This is done with a
; comma separated key = value list, same as 'host_role_relation'.
;   key: defines VM(s) name as regex
;   value: defines the NetBox role name (use quotes if name contains commas)
;vm_role_relation = .* = Server

; Define NetBox tags which are assigned to a cluster, host or VM. This is
; done with a comma separated key = value list.
;   key: defines a hosts/VM name as regex
;   value: defines the NetBox tag (use quotes if name contains commas)
; a cluster can be specified as "Cluster-name" or
; "Datacenter-name/Cluster-name" if multiple clusters have the same name
;cluster_tag_relation = Cluster_NYC.* = Infrastructure
;host_tag_relation = esxi300.* = Infrastructure
;vm_tag_relation = grafana.* = Infrastructure

; Try to find existing host based on serial number. This can cause issues with blade
; centers if VMWare does not report the blades serial number properly.
;match_host_by_serial = True

; Attempt to collect asset tags from vCenter hosts
;collect_hardware_asset_tag = True

; Perform a reverse lookup for all collected IP addresses. If a dns name was found it will
; be added to the IP address object in NetBox
dns_name_lookup = True

; use custom DNS server to do the reverse lookups
custom_dns_servers = 192.168.16.1, 192.168.16.2

; define how the primary IPs should be set
; possible values:
;
;   always:     will remove primary IP from the object where this address is
;               currently set as primary and moves it to new object
;
;   when-undefined:
;               only sets primary IP if undefined, will cause ERRORs if same IP is
;               assigned more then once to different hosts and IP is set as the
;               objects primary IP
;
;   never:      don't set any primary IPs, will cause the same ERRORs
;               as "when-undefined"
;set_primary_ip = when-undefined

; Do not sync notes from a VM in vCenter to the comments field on a VM in netbox
;skip_vm_comments = False

; Do not sync template VMs
skip_vm_templates = True

; Skip virtual machines which are reported as offline.
; ATTENTION: this option will keep purging stopped VMs if activated!
;skip_offline_vms = False

; If the VMware Site Recovery Manager is used to can skip syncing placeholder/replicated
; VMs from fail-over site to NetBox.
;skip_srm_placeholder_vms = False

; strip domain part from host name before syncing device to NetBox
;strip_host_domain_name = False

; strip domain part from VM name before syncing VM to NetBox
;strip_vm_domain_name = False

; tag source options

; sync tags assigned to clusters, hosts and VMs in vCenter to NetBox
; INFO: this requires the installation of the 'vsphere-automation-sdk',
; see docs about installation possible values:
;   * object : the host or VM itself
;   * parent_folder_1 : the direct folder this object is organized in (1 level up)
;   * parent_folder_2 : the indirect folder this object is organized in (2 levels up)
;   * cluster : the cluster this object is organized in
;   * datacenter : the datacenter this object is organized in
; this is a comma separated list of options. example: vm_tag_source = object, cluster
;
; Example: vm_tag_source = object, cluster
;cluster_tag_source =
;host_tag_source =
;vm_tag_source =

; sync custom attributes defined for hosts and VMs in vCenter to NetBox as custom fields
;sync_custom_attributes = False

; custom object attributes options

; add arbitrary host/vm object attributes as custom fields to NetBox.
; multiple attributes can be defined comma separated.
; to get a list of available attributes use '-l DEBUG3' as cli param (CAREFUL: output might be long)
; and here 'https://gist.github.com/bb-Ricardo/538768487bdac4efafabe56e005cb4ef' can be seen how to
; access these attributes
;host_custom_object_attributes = summary.runtime.bootTime
;vm_custom_object_attributes = config.uuid

; this will set the sources name as cluster group name instead of the datacenter. This
; works if the vCenter has ONLY ONE datacenter configured. Otherwise it will rename all
; datacenters to the source name!
;set_source_name_as_cluster_group = False

; activating this option will also include "dummy/virtual" interfaces which are only
; visible inside the VM and are exposed through VM guest tools. Dummy interfaces without
; an IP address will be skipped.
sync_vm_dummy_interfaces = true

; disables syncing of any VLANs visible in vCenter to NetBox
disable_vlan_sync = true

; enabling this option will add the ESXi host this VM is running on to the VM details
track_vm_host = true

; define if the name of the device interface discovered overwrites the interface name in
; NetBox. The interface will only be matched by identical MAC address
;overwrite_device_interface_name = True

; define if the name of the VM interface discovered overwrites the interface name in
; NetBox. The interface will only be matched by identical MAC address
;overwrite_vm_interface_name = True

; set a matching value for ESXi host management interface description (case insensitive,
; comma separated). Used to figure out the ESXi primary IP address
;host_management_interface_match = management, mgmt

; define in which order the IP address tenant will be assigned if tenant is undefined.
; possible values:
;   * device : host or VM tenant will be assigned to the IP address
;   * prefix : if the IP address belongs to an existing prefix and this prefix has a tenant assigned, then this one is used
;   * disabled : no tenant assignment to the IP address will be performed
; the order of the definition is important, the default is "device, prefix" which means:
; If the device has a tenant then this one will be used. If not, the prefix tenant will be used if defined
;ip_tenant_inheritance_order = device, prefix

; Usually netbox-sync grabs the MTU size for the VM interface from the ESXi hosts vSwitch.
; If this is not fitting or incorrect it is possible to disable the synchronisation by
; setting this option to 'False'
;sync_vm_interface_mtu = True

; defines a comma separated list of MAC addresses which should be excluded from sync. Any
; host NIC with a matching MAC address will be excluded from sync.
;host_nic_exclude_by_mac_list = AA:BB:CC:11:22:33, 66:77:88:AA:BB:CC

; defines a comma separated list of custom attribute which should be excluded from sync.
; Any custom attribute with a matching attribute key will be excluded from sync.
;custom_attribute_exclude = VB_LAST_BACKUP, VB_LAST_BACKUP2

[bb-Ricardo]

Hi, can you run the sync with -l DEBUG3 and post the output here?

[jannikk09]

2023-10-26 12:35:31,139 - DEBUG2: NetBox results are paginated. Getting next page
{'_body_position': None,
 '_cookies': <RequestsCookieJar[]>,
 'body': None,
 'headers': {'User-Agent': 'netbox-sync/1.5.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'keep-alive', 'Authorization': 'Token b7557275f404b35ac4ff964d3b45e37d75c4ac1e', 'Content-Type': 'application/json'},
 'hooks': {'response': []},
 'method': 'GET',
 'url': 'http://srv-app03.domain.de/api/ipam/ip-addresses/?exclude=config_context&limit=200&offset=200'}
2023-10-26 12:35:31,140 - DEBUG: Starting new HTTP connection (1): srv-app03.domain.de:80
send: b'GET /api/ipam/ip-addresses/?exclude=config_context&limit=200&offset=200 HTTP/1.1\r\nHost: srv-app03.domain.de\r\nUser-Agent: netbox-sync/1.5.1\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nAuthorization: Token b7557275f404b35ac4ff964d3b45e37d75c4ac1e\r\nContent-Type: application/json\r\n\r\n'
reply: 'HTTP/1.1 404 Not Found\r\n'
header: Date: Thu, 26 Oct 2023 12:35:31 GMT
header: Server: Apache/2.4.52 (Ubuntu)
header: Content-Length: 278
header: Keep-Alive: timeout=5, max=100
header: Connection: Keep-Alive
header: Content-Type: text/html; charset=iso-8859-1
2023-10-26 12:35:31,141 - DEBUG: http://srv-app03.domain.de:80 "GET /api/ipam/ip-addresses/?exclude=config_context&limit=200&offset=200 HTTP/1.1" 404 278
2023-10-26 12:35:31,142 - DEBUG2: Received HTTP Status 404.
2023-10-26 12:35:31,142 - DEBUG: Response Body:
2023-10-26 12:35:31,142 - ERROR: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/netbox-sync/netbox-sync.py", line 143, in <module>
    main()
  File "/opt/netbox-sync/netbox-sync.py", line 98, in main
    nb_handler.query_current_data(source.dependent_netbox_objects)
  File "/opt/netbox-sync/module/netbox/connection.py", line 461, in query_current_data
    full_nb_data = self.request(nb_object_class)
  File "/opt/netbox-sync/module/netbox/connection.py", line 282, in request
    result["results"].extend(response.json().get("results"))
  File "/usr/local/lib/python3.10/dist-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[bb-Ricardo]

The stack trace should not appear, I guess this is an issue within the library versioning. Have to test this.

But your webserver says:

reply: 'HTTP/1.1 404 Not Found\r\n'
header: Date: Thu, 26 Oct 2023 12:35:31 GMT
header: Server: Apache/2.4.52 (Ubuntu)
header: Content-Length: 278

Which seems to indicate a Webserver configuration issue.

[jannikk09]

Could it be because port 80 is not used by the Apache web server?
NetBox is also only on port 443 and not on 8000

[bb-Ricardo]

Best is to troubleshoot using curl and see if the endpoints are actually reachable as designed. Then you can adjust your configuration accordingly and try another sync.

[jannikk09]

I just reconfigured my Apache config. The Sync is over HTTP - just added this the config. Now it works