nautobot / nautobot-app-ssot

Single Source of Truth for Nautobot
https://docs.nautobot.com/projects/ssot/en/latest/
Other
38 stars 41 forks source link

Possible gRPC issue with Arista SSoT #467

Open qduk opened 5 months ago

qduk commented 5 months ago

Environment

Expected Behavior

Running the Arista SSoT successfully populates data into Nautobot.

Observed Behavior

A gRPC error is raised during the sync.

Steps to Reproduce

  1. Install SSoT app with Arista Integration
  2. Run sync job
qduk commented 5 months ago

Traceback:

[2024-05-09 23:53:26,378: WARNING/ForkPoolWorker-30] Exception ignored in:
[2024-05-09 23:53:26,382: WARNING/ForkPoolWorker-30] <function _Rendezvous.__del__ at 0x7fdf5d99e480>
[2024-05-09 23:53:26,386: WARNING/ForkPoolWorker-30] Traceback (most recent call last):
[2024-05-09 23:53:26,388: WARNING/ForkPoolWorker-30]   File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 561, in __del__
[2024-05-09 23:53:26,394: WARNING/ForkPoolWorker-30] with self._state.condition:
[2[2024-05-09 23:53:26,445: WARNING/ForkPoolWorker-30] AttributeError
[2024-05-09 23:53:26,447: WARNING/ForkPoolWorker-30] :
[2024-05-09 23:53:26,450: WARNING/ForkPoolWorker-30] '_MultiThreadedRendezvous' object has no attribute '_state'
[2024-05-09 23:53:26,452: WARNING/ForkPoolWorker-30] Exception ignored in:
[2024-05-09 23:53:26,454: WARNING/ForkPoolWorker-30] <function _Rendezvous.__del__ at 0x7fdf5d99e480>
[2024-05-09 23:53:26,457: WARNING/ForkPoolWorker-30] Traceback (most recent call last):
[2024-05-09 23:53:26,459: WARNING/ForkPoolWorker-30]   File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 561, in __del__
[2024-05-09 23:53:26,463: WARNING/ForkPoolWorker-30]
[2024-05-09 23:53:26,465: WARNING/ForkPoolWorker-30] with self._state.condition:
[2024-05-09 23:53:26,516: WARNING/ForkPoolWorker-30] AttributeError
[2024-05-09 23:53:26,518: WARNING/ForkPoolWorker-30] :
[2024-05-09 23:53:26,521: WARNING/ForkPoolWorker-30] '_Rendezvous' object has no attribute '_state'
[2024-05-09 23:53:26,544: ERROR/ForkPoolWorker-30] Task nautobot.extras.jobs.run_job[a2a6ea55-fc2f-440e-89c1-8c82761c563b] raised unexpected: RpcError()
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nautobot/extras/jobs.py", line 1136, in run_job
    result = job(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nautobot/extras/jobs.py", line 149, in __call__
    return self.run(*args, **deserialized_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/source/nautobot_ssot/integrations/aristacv/jobs.py", line 140, in run
    super().run(dryrun=self.dryrun, memory_profiling=self.memory_profiling, *args, **kwargs)
  File "/source/nautobot_ssot/jobs/base.py", line 317, in run
    self.sync_data(memory_profiling)
  File "/source/nautobot_ssot/jobs/base.py", line 136, in sync_data
    self.load_source_adapter()
  File "/source/nautobot_ssot/integrations/aristacv/jobs.py", line 119, in load_source_adapter
    self.source_adapter.load()
  File "/source/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 267, in load
    self.load_devices()
  File "/source/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 72, in load_devices
    for index, dev in enumerate(cloudvision.get_devices(client=self.conn.comm_channel, import_active=True), start=1):
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/source/nautobot_ssot/integrations/aristacv/utils/cloudvision.py", line 270, in get_devices
    for resp in responses:
  File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 543, in __next__
    return self._next()
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/grpc/_channel.py", line 969, in _next
    raise self
grpc.RpcError

This was an error brought to me by @chambersh1129 while at Cisco Live. I need dig into it more but I wanted to get the issue open. @chambersh1129 can you provide the python version you are using?

chambersh1129 commented 5 months ago

Python Version: 3.11.9

PLUGINS_CONFIG Arista SSoT Snippet:

PLUGINS_CONFIG = {
    "nautobot_ssot": {
        "enable_aristacv": True,
        "aristacv_cvp_token": os.getenv("NAUTOBOT_ARISTACV_TOKEN", ""),
        "aristacv_cvaas_url": "www.arista.io:443",
        "aristacv_verify": is_truthy(os.getenv("NAUTOBOT_ARISTACV_VERIFY", True)),
        "aristacv_from_cloudvision_default_site": "cloudvision_imported",
        "aristacv_from_cloudvision_default_device_role": "network",
        "aristacv_from_cloudvision_default_device_role_color": "ff0000",
        "aristacv_delete_devices_on_sync": is_truthy(os.getenv("NAUTOBOT_ARISTACV_DELETE_ON_SYNC", False)),
        "aristacv_apply_import_tag": is_truthy(os.getenv("NAUTOBOT_ARISTACV_IMPORT_TAG", True)),
        "aristacv_import_active": is_truthy(os.getenv("NAUTOBOT_ARISTACV_IMPORT_ACTIVE", False)),
        "aristacv_create_controller": is_truthy(os.getenv("NAUTOBOT_ARISTACV_CREATE_CONTROLLER", False)),
        "aristacv_controller_site": os.getenv("NAUTOBOT_ARISTACV_CONTROLLER_SITE", ""),
        "aristacv_hostname_patterns": [],  # scrubbed
        "aristacv_site_mappings": {},  # scrubbed
        "aristacv_role_mappings": {},  # scrubbed
    },
}
chambersh1129 commented 3 months ago

edit: was building from an older nautobot docker image (2.2.4). versions and logs updated to reflect latest 2.2.9 build.

Following up on this, today I upgraded Nautobot and SSOT plugin:

I am getting the same exception. Traceback:

[2024-08-14 19:20:50,391: INFO/MainProcess] Task nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731] received
19:20:50.413 DEBUG   nautobot.extras.jobs jobs.py                                run_job() :
  Running job nautobot_ssot.integrations.aristacv.jobs.CloudVisionDataSource
[2024-08-14 19:20:50,413: DEBUG/ForkPoolWorker-7] Running job nautobot_ssot.integrations.aristacv.jobs.CloudVisionDataSource
[2024-08-14 19:20:50,417: INFO/ForkPoolWorker-7] nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731]: Running job
[2024-08-14 19:20:50,440: INFO/ForkPoolWorker-7] nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731]: Loading current data from source adapter...
[2024-08-14 19:20:50,472: INFO/ForkPoolWorker-7] nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731]: Loading data from CloudVision
[2024-08-14 19:21:02,195: WARNING/ForkPoolWorker-7] Exception ignored in: 
[2024-08-14 19:21:02,201: WARNING/ForkPoolWorker-7] <function _Rendezvous.__del__ at 0x7f4965703a60>
[2024-08-14 19:21:02,206: WARNING/ForkPoolWorker-7] Traceback (most recent call last):
[2024-08-14 19:21:02,210: WARNING/ForkPoolWorker-7]   File "/opt/nautobot/.local/lib/python3.11/site-packages/grpc/_channel.py", line 561, in __del__
[2024-08-14 19:21:02,213: WARNING/ForkPoolWorker-7]     
[2024-08-14 19:21:02,217: WARNING/ForkPoolWorker-7] with self._state.condition:
[2024-08-14 19:21:02,221: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,227: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,231: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,237: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,241: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,245: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,251: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,257: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,263: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,266: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,269: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,272: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,276: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,279: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,283: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,287: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,294: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,299: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,302: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,305: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,308: WARNING/ForkPoolWorker-7] AttributeError
[2024-08-14 19:21:02,311: WARNING/ForkPoolWorker-7] : 
[2024-08-14 19:21:02,316: WARNING/ForkPoolWorker-7] '_MultiThreadedRendezvous' object has no attribute '_state'
[2024-08-14 19:21:02,319: WARNING/ForkPoolWorker-7] Exception ignored in: 
[2024-08-14 19:21:02,322: WARNING/ForkPoolWorker-7] <function _Rendezvous.__del__ at 0x7f4965703a60>
[2024-08-14 19:21:02,325: WARNING/ForkPoolWorker-7] Traceback (most recent call last):
[2024-08-14 19:21:02,329: WARNING/ForkPoolWorker-7]   File "/opt/nautobot/.local/lib/python3.11/site-packages/grpc/_channel.py", line 561, in __del__
[2024-08-14 19:21:02,332: WARNING/ForkPoolWorker-7]     
[2024-08-14 19:21:02,335: WARNING/ForkPoolWorker-7] with self._state.condition:
[2024-08-14 19:21:02,338: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,342: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,345: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,349: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,356: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,360: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,363: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,366: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,369: WARNING/ForkPoolWorker-7]  
[2024-08-14 19:21:02,374: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,377: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,380: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,383: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,386: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,393: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,396: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,400: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,402: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,405: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,408: WARNING/ForkPoolWorker-7] ^
[2024-08-14 19:21:02,411: WARNING/ForkPoolWorker-7] AttributeError
[2024-08-14 19:21:02,415: WARNING/ForkPoolWorker-7] : 
[2024-08-14 19:21:02,418: WARNING/ForkPoolWorker-7] '_Rendezvous' object has no attribute '_state'
[2024-08-14 19:21:02,434: ERROR/ForkPoolWorker-7] Task nautobot.extras.jobs.run_job[7413ea44-2668-49b6-87e2-bd4472e72731] raised unexpected: RpcError()
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 477, in trace_task
    R = retval = fun(*args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 760, in __protected_call__
    return self.run(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nautobot/extras/jobs.py", line 1136, in run_job
    result = job(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/nautobot/extras/jobs.py", line 149, in __call__
    return self.run(*args, **deserialized_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/jobs.py", line 140, in run
    super().run(dryrun=self.dryrun, memory_profiling=self.memory_profiling, *args, **kwargs)
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/jobs/base.py", line 317, in run
    self.sync_data(memory_profiling)
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/jobs/base.py", line 136, in sync_data
    self.load_source_adapter()
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/jobs.py", line 119, in load_source_adapter
    self.source_adapter.load()
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 269, in load
    self.load_devices()
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 95, in load_devices
    self.load_device_tags(device=new_device)
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/diffsync/adapters/cloudvision.py", line 234, in load_device_tags
    system_tags = cloudvision.get_tags_by_type(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/nautobot/.local/lib/python3.11/site-packages/nautobot_ssot/integrations/aristacv/utils/cloudvision.py", line 287, in get_tags_by_type
    for resp in responses:
  File "/opt/nautobot/.local/lib/python3.11/site-packages/grpc/_channel.py", line 543, in __next__
    return self._next()
           ^^^^^^^^^^^^
  File "/opt/nautobot/.local/lib/python3.11/site-packages/grpc/_channel.py", line 969, in _next
    raise self
grpc.RpcError
jdrew82 commented 1 month ago

@chambersh1129 Is this still an issue? Unfortunately, we don't have a CVP instance to test against at this time so we can't validate if this is still occurring or how to fix it.

jdrew82 commented 1 month ago

@chambersh1129 Can you see if the PR #580 solves the issue you're experiencing? It should at least allow the code to bypass the RpcError that's being thrown.

chambersh1129 commented 3 weeks ago

Hi @jdrew82, we shifted course and started using the IPFabric integration which is working for us. I currently don't have the cycles nor a DEV environment to test the Arista CV integration and the IPFabric integration is vendor agnostic and all inclusive.

This can be kept open if you want to use it for tracking/reminder to test in the future. If I'm the only person who has brought up the issue then it can be closed/resolved.

jdrew82 commented 3 weeks ago

@chambersh1129 Appreciate the response. The Arista team has reached out to provide a test instance for us to use. I believe they stated they were able to reproduce the issue so we should hopefully have a fix available in the near future. I'll leave this open for the time being while we work to resolve the bug.