Open qduk opened 5 months ago
Traceback:
[2024-05-09 23:53:26,378: WARNING/ForkPoolWorker-30] Exception ignored in:
[2024-05-09 23:53:26,382: WARNING/ForkPoolWorker-30] <function _Rendezvous.__del__ at 0x7fdf5d99e480>
[2024-05-09 23:53:26,386: WARNING/ForkPoolWorker-30] Traceback (most recent call last):
[2024-05-09 23:53:26,388: WARNING/ForkPoolWorker-30] File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 561, in __del__
[2024-05-09 23:53:26,394: WARNING/ForkPoolWorker-30] with self._state.condition:
[2[2024-05-09 23:53:26,445: WARNING/ForkPoolWorker-30] AttributeError
[2024-05-09 23:53:26,447: WARNING/ForkPoolWorker-30] :
[2024-05-09 23:53:26,450: WARNING/ForkPoolWorker-30] '_MultiThreadedRendezvous' object has no attribute '_state'
[2024-05-09 23:53:26,452: WARNING/ForkPoolWorker-30] Exception ignored in:
[2024-05-09 23:53:26,454: WARNING/ForkPoolWorker-30] <function _Rendezvous.__del__ at 0x7fdf5d99e480>
[2024-05-09 23:53:26,457: WARNING/ForkPoolWorker-30] Traceback (most recent call last):
[2024-05-09 23:53:26,459: WARNING/ForkPoolWorker-30] File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 561, in __del__
[2024-05-09 23:53:26,463: WARNING/ForkPoolWorker-30]
[2024-05-09 23:53:26,465: WARNING/ForkPoolWorker-30] with self._state.condition:
[2024-05-09 23:53:26,516: WARNING/ForkPoolWorker-30] AttributeError
[2024-05-09 23:53:26,518: WARNING/ForkPoolWorker-30] :
[2024-05-09 23:53:26,521: WARNING/ForkPoolWorker-30] '_Rendezvous' object has no attribute '_state'
[2024-05-09 23:53:26,544: ERROR/ForkPoolWorker-30] Task nautobot.extras.jobs.run_job[a2a6ea55-fc2f-440e-89c1-8c82761c563b] raised unexpected: RpcError()
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
R = retval = fun(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
return self.run(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/nautobot/extras/jobs.py", line 1136, in run_job
result = job(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/nautobot/extras/jobs.py", line 149, in __call__
return self.run(*args, **deserialized_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/source/nautobot_ssot/integrations/aristacv/jobs.py", line 140, in run
super().run(dryrun=self.dryrun, memory_profiling=self.memory_profiling, *args, **kwargs)
File "/source/nautobot_ssot/jobs/base.py", line 317, in run
self.sync_data(memory_profiling)
File "/source/nautobot_ssot/jobs/base.py", line 136, in sync_data
self.load_source_adapter()
File "/source/nautobot_ssot/integrations/aristacv/jobs.py", line 119, in load_source_adapter
self.source_adapter.load()
File "/source/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 267, in load
self.load_devices()
File "/source/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 72, in load_devices
for index, dev in enumerate(cloudvision.get_devices(client=self.conn.comm_channel, import_active=True), start=1):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/source/nautobot_ssot/integrations/aristacv/utils/cloudvision.py", line 270, in get_devices
for resp in responses:
File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 543, in __next__
return self._next()
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 969, in _next
raise self
grpc.RpcError
This was an error brought to me by @chambersh1129 while at Cisco Live. I need dig into it more but I wanted to get the issue open. @chambersh1129 can you provide the python version you are using?
Python Version: 3.11.9
PLUGINS_CONFIG Arista SSoT Snippet:
PLUGINS_CONFIG = {
"nautobot_ssot": {
"enable_aristacv": True,
"aristacv_cvp_token": os.getenv("NAUTOBOT_ARISTACV_TOKEN", ""),
"aristacv_cvaas_url": "www.arista.io:443",
"aristacv_verify": is_truthy(os.getenv("NAUTOBOT_ARISTACV_VERIFY", True)),
"aristacv_from_cloudvision_default_site": "cloudvision_imported",
"aristacv_from_cloudvision_default_device_role": "network",
"aristacv_from_cloudvision_default_device_role_color": "ff0000",
"aristacv_delete_devices_on_sync": is_truthy(os.getenv("NAUTOBOT_ARISTACV_DELETE_ON_SYNC", False)),
"aristacv_apply_import_tag": is_truthy(os.getenv("NAUTOBOT_ARISTACV_IMPORT_TAG", True)),
"aristacv_import_active": is_truthy(os.getenv("NAUTOBOT_ARISTACV_IMPORT_ACTIVE", False)),
"aristacv_create_controller": is_truthy(os.getenv("NAUTOBOT_ARISTACV_CREATE_CONTROLLER", False)),
"aristacv_controller_site": os.getenv("NAUTOBOT_ARISTACV_CONTROLLER_SITE", ""),
"aristacv_hostname_patterns": [], # scrubbed
"aristacv_site_mappings": {}, # scrubbed
"aristacv_role_mappings": {}, # scrubbed
},
}
edit: was building from an older nautobot docker image (2.2.4). versions and logs updated to reflect latest 2.2.9 build.
Following up on this, today I upgraded Nautobot and SSOT plugin:
I am getting the same exception. Traceback:
[2024-08-14 19:20:50,391: INFO/MainProcess] Task nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731] received
19:20:50.413 DEBUG nautobot.extras.jobs jobs.py run_job() :
Running job nautobot_ssot.integrations.aristacv.jobs.CloudVisionDataSource
[2024-08-14 19:20:50,413: DEBUG/ForkPoolWorker-7] Running job nautobot_ssot.integrations.aristacv.jobs.CloudVisionDataSource
[2024-08-14 19:20:50,417: INFO/ForkPoolWorker-7] nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731]: Running job
[2024-08-14 19:20:50,440: INFO/ForkPoolWorker-7] nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731]: Loading current data from source adapter...
[2024-08-14 19:20:50,472: INFO/ForkPoolWorker-7] nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731]: Loading data from CloudVision
[2024-08-14 19:21:02,195: WARNING/ForkPoolWorker-7] Exception ignored in:
[2024-08-14 19:21:02,201: WARNING/ForkPoolWorker-7] <function _Rendezvous.__del__ at 0x7f4965703a60>
[2024-08-14 19:21:02,206: WARNING/ForkPoolWorker-7] Traceback (most recent call last):
[2024-08-14 19:21:02,210: WARNING/ForkPoolWorker-7] File "/opt/nautobot/.local/lib/python3.11/site-packages/grpc/_channel.py", line 561, in __del__
[2024-08-14 19:21:02,213: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,217: WARNING/ForkPoolWorker-7] with self._state.condition:
[2024-08-14 19:21:02,221: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,227: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,231: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,237: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,241: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,245: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,251: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,257: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,263: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,266: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,269: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,272: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,276: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,279: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,283: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,287: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,294: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,299: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,302: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,305: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,308: WARNING/ForkPoolWorker-7] AttributeError
[2024-08-14 19:21:02,311: WARNING/ForkPoolWorker-7] :
[2024-08-14 19:21:02,316: WARNING/ForkPoolWorker-7] '_MultiThreadedRendezvous' object has no attribute '_state'
[2024-08-14 19:21:02,319: WARNING/ForkPoolWorker-7] Exception ignored in:
[2024-08-14 19:21:02,322: WARNING/ForkPoolWorker-7] <function _Rendezvous.__del__ at 0x7f4965703a60>
[2024-08-14 19:21:02,325: WARNING/ForkPoolWorker-7] Traceback (most recent call last):
[2024-08-14 19:21:02,329: WARNING/ForkPoolWorker-7] File "/opt/nautobot/.local/lib/python3.11/site-packages/grpc/_channel.py", line 561, in __del__
[2024-08-14 19:21:02,332: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,335: WARNING/ForkPoolWorker-7] with self._state.condition:
[2024-08-14 19:21:02,338: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,342: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,345: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,349: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,356: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,360: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,363: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,366: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,369: WARNING/ForkPoolWorker-7]
[2024-08-14 19:21:02,374: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,377: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,380: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,383: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,386: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,393: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,396: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,400: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,402: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,405: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,408: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,411: WARNING/ForkPoolWorker-7] AttributeError
[2024-08-14 19:21:02,415: WARNING/ForkPoolWorker-7] :
[2024-08-14 19:21:02,418: WARNING/ForkPoolWorker-7] '_Rendezvous' object has no attribute '_state'
[2024-08-14 19:21:02,434: ERROR/ForkPoolWorker-7] Task nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731] raised unexpected: RpcError()
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
R = retval = fun(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
return self.run(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/nautobot/extras/jobs.py", line 1136, in run_job
result = job(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/nautobot/extras/jobs.py", line 149, in __call__
return self.run(*args, **deserialized_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/jobs.py", line 140, in run
super().run(dryrun=self.dryrun, memory_profiling=self.memory_profiling, *args, **kwargs)
File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/jobs/base.py", line 317, in run
self.sync_data(memory_profiling)
File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/jobs/base.py", line 136, in sync_data
self.load_source_adapter()
File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/jobs.py", line 119, in load_source_adapter
self.source_adapter.load()
File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 269, in load
self.load_devices()
File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 95, in load_devices
self.load_device_tags(device=new_device)
File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 234, in load_device_tags
system_tags = cloudvision.get_tags_by_type(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/utils/cloudvision.py", line 287, in get_tags_by_type
for resp in responses:
File "/opt/nautobot/.local/lib/python3.11/site-packages/grpc/_channel.py", line 543, in __next__
return self._next()
^^^^^^^^^^^^
File "/opt/nautobot/.local/lib/python3.11/site-packages/grpc/_channel.py", line 969, in _next
raise self
grpc.RpcError
@chambersh1129 Is this still an issue? Unfortunately, we don't have a CVP instance to test against at this time so we can't validate if this is still occurring or how to fix it.
@chambersh1129 Can you see if the PR #580 solves the issue you're experiencing? It should at least allow the code to bypass the RpcError that's being thrown.
Hi @jdrew82, we shifted course and started using the IPFabric integration which is working for us. I currently don't have the cycles nor a DEV environment to test the Arista CV integration and the IPFabric integration is vendor agnostic and all inclusive.
This can be kept open if you want to use it for tracking/reminder to test in the future. If I'm the only person who has brought up the issue then it can be closed/resolved.
@chambersh1129 Appreciate the response. The Arista team has reached out to provide a test instance for us to use. I believe they stated they were able to reproduce the issue so we should hopefully have a fix available in the near future. I'll leave this open for the time being while we work to resolve the bug.
Environment
Expected Behavior
Running the Arista SSoT successfully populates data into Nautobot.
Observed Behavior
A gRPC error is raised during the sync.
Steps to Reproduce