Pyats yang.connector gnmi proto files collide with PyGNMI ?

[dmulyalin]

Hello,

After upgrading PyATS/Genie form version 22.1 to version 22.5 started getting this error while running "ping" api for Cisco IOS XE device:

            Traceback (most recent call last):
              File "/usr/local/lib/python3.9/site-packages/nornir/core/task.py", line 99, in start
                r = self.task(self, **self.params)
              File "/usr/local/lib/python3.9/site-packages/nornir_salt/utils/yangdantic.py", line 47, in wrapper
                return self.function(*args, **kwargs)
              File "/usr/local/lib/python3.9/site-packages/nornir_salt/plugins/tasks/pyats_genie_api.py", line 66, in pyats_genie_api
                result = getattr(device.api, api)(**kwargs)
              File "src/genie/conf/base/api.py", line 176, in genie.conf.base.api.API.__getattr__.wrapper_match
              File "src/genie/conf/base/api.py", line 162, in genie.conf.base.api.API.__getattr__.wrapper_match
              File "src/genie/conf/base/api.py", line 244, in genie.conf.base.api.API.get_api
              File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
                return _bootstrap._gcd_import(name[level:], package, level)
              File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
              File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
              File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
              File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
              File "<frozen importlib._bootstrap_external>", line 850, in exec_module
              File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
              File "/usr/local/lib/python3.9/site-packages/genie/libs/sdk/apis/iosxe/mpte_utils/mpte_utils.py", line 8, in <module>
                from yang.connector.gnmi import Gnmi
              File "/usr/local/lib/python3.9/site-packages/yang/connector/__init__.py", line 22, in <module>
                from .gnmi import Gnmi
              File "/usr/local/lib/python3.9/site-packages/yang/connector/gnmi.py", line 16, in <module>
                from cisco_gnmi import ClientBuilder
              File "/usr/local/lib/python3.9/site-packages/cisco_gnmi/__init__.py", line 27, in <module>
                from .client import Client
              File "/usr/local/lib/python3.9/site-packages/cisco_gnmi/client.py", line 30, in <module>
                from . import proto
              File "/usr/local/lib/python3.9/site-packages/cisco_gnmi/proto/__init__.py", line 25, in <module>
                from . import gnmi_pb2_grpc
              File "/usr/local/lib/python3.9/site-packages/cisco_gnmi/proto/gnmi_pb2_grpc.py", line 4, in <module>
                from . import gnmi_pb2 as gnmi__pb2
              File "/usr/local/lib/python3.9/site-packages/cisco_gnmi/proto/gnmi_pb2.py", line 19, in <module>
                from . import gnmi_ext_pb2 as gnmi__ext__pb2
              File "/usr/local/lib/python3.9/site-packages/cisco_gnmi/proto/gnmi_ext_pb2.py", line 19, in <module>
                DESCRIPTOR = _descriptor.FileDescriptor(
              File "/usr/local/lib64/python3.9/site-packages/google/protobuf/descriptor.py", line 1024, in __new__
                return _message.default_pool.AddSerializedFile(serialized_pb)
            TypeError: Couldn't build proto file into descriptor pool!
            Invalid proto descriptor for file "gnmi_ext.proto":
              gnmi_ext.proto: A file with this name is already in the pool.

Found this old github issue: https://github.com/protocolbuffers/protobuf/issues/3002

Turns out I have PyGNMI==0.6.9 installed on the same system and interesting enough after uninstalling PyGNMI package, PyATS api calls start working fine.

Could it be that proto files created by PyATS yang connector somehow overlap with proto files created by PyGNMI, could we implement some sort of shielding mechanism like put all PyATS proto files in dedicated directory as per this comment

At this point we have two options. Use pure python protobuf package, or set package directive in our proto files. I think this covers everything.

[dmulyalin]

Opened PyGNMI issue as well - https://github.com/akarneliuk/pygnmi/issues/62

[ademz]

Hello @dmulyalin

Thank you for bringing this. It seems it's not pyATS related issue.

I think it will help the following solution for your issue.

https://github.com/protocolbuffers/protobuf/issues/3002#issuecomment-314179893

Thanks,

[dmulyalin]

Doing pip uninstall protobuf following with pip install --no-binary protobuf protobuf did not help, getting new error:

            Traceback (most recent call last):
              File "/usr/local/lib/python3.9/site-packages/nornir/core/task.py", line 99, in start
                r = self.task(self, **self.params)
              File "/usr/local/lib/python3.9/site-packages/nornir_salt/utils/yangdantic.py", line 47, in wrapper
                return self.function(*args, **kwargs)
              File "/usr/local/lib/python3.9/site-packages/nornir_salt/plugins/tasks/pyats_genie_api.py", line 66, in pyats_genie_api
                result = getattr(device.api, api)(**kwargs)
              File "src/genie/conf/base/api.py", line 178, in genie.conf.base.api.API.__getattr__.wrapper_match
              File "src/genie/conf/base/api.py", line 164, in genie.conf.base.api.API.__getattr__.wrapper_match
              File "src/genie/conf/base/api.py", line 246, in genie.conf.base.api.API.get_api
              File "/usr/lib64/python3.9/importlib/__init__.py", line 127, in import_module
                return _bootstrap._gcd_import(name[level:], package, level)
              File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
              File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
              File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
              File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
              File "<frozen importlib._bootstrap_external>", line 850, in exec_module
              File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
              File "/usr/local/lib/python3.9/site-packages/genie/libs/sdk/apis/iosxe/mpte_utils/mpte_utils.py", line 8, in <module>
                from yang.connector.gnmi import Gnmi
              File "/usr/local/lib/python3.9/site-packages/yang/connector/__init__.py", line 22, in <module>
                from .gnmi import Gnmi
              File "/usr/local/lib/python3.9/site-packages/yang/connector/gnmi.py", line 16, in <module>
                from cisco_gnmi import ClientBuilder
              File "/usr/local/lib/python3.9/site-packages/cisco_gnmi/__init__.py", line 27, in <module>
                from .client import Client
              File "/usr/local/lib/python3.9/site-packages/cisco_gnmi/client.py", line 30, in <module>
                from . import proto
              File "/usr/local/lib/python3.9/site-packages/cisco_gnmi/proto/__init__.py", line 25, in <module>
                from . import gnmi_pb2_grpc
              File "/usr/local/lib/python3.9/site-packages/cisco_gnmi/proto/gnmi_pb2_grpc.py", line 4, in <module>
                from . import gnmi_pb2 as gnmi__pb2
              File "/usr/local/lib/python3.9/site-packages/cisco_gnmi/proto/gnmi_pb2.py", line 1834, in <module>
                google_dot_protobuf_dot_descriptor__pb2.FileOptions.RegisterExtension(gnmi_service)
              File "/usr/lib/python3.9/site-packages/google/protobuf/internal/python_message.py", line 795, in RegisterExtension
                cls.DESCRIPTOR.file.pool._AddExtensionDescriptor(extension_handle)
              File "/usr/lib/python3.9/site-packages/google/protobuf/descriptor_pool.py", line 343, in _AddExtensionDescriptor
                raise AssertionError(
            AssertionError: Extensions "gnmi.gnmi_service" and "gnmi.gnmi_service" both try to extend message type "google.protobuf.FileOptions" with field number 1001.

IMHO it is PyATS related issue, as it seems as a some form of clash causing PyATS to throw an error. Not sure how likely fix could be implemented, hence leaving this case closed.

[willbebuds]

Hello @dmulyalin

Thank you for bringing this. It seems it's not pyATS related issue.

I think it will help the following solution for your issue.

protocolbuffers/protobuf#3002 (comment)

Thanks,

This is definitely a pyATS issue...

As submitter pointed out, previous versions of pyATS doesn't have this problem. I ran into this today as well and can't seem to get it to work. @dmulyalin - did you just downgrade your pyATS version?

[dmulyalin]

Yeah, ended up downgrading pyats.

[willbebuds]

@dmulyalin just an fyi - i was told today there is a cisco-gnmi package available and didn't see any issue importing it. Will be messing around with it but seems promising! https://github.com/cisco-ie/cisco-gnmi-python