CiscoDevNet / ydk-gen

Generate model-driven APIs from YANG models
http://ciscodevnet.github.io/ydk-gen/
Apache License 2.0
136 stars 74 forks source link

CodecService fails to decode XML in python when referencing namespace prefix #1071

Closed ldacol closed 1 year ago

ldacol commented 2 years ago

Issue tracker is ONLY used for reporting bugs. Please use the YDK Community for any support issues.

Expected Behavior

Current Behavior

CodecService.decode fails to decode XML code in python when using idx BGP prefix in the names space

Steps to Reproduce

Attempt to decode below XML code via CodecService

<data xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" xmlns:nc="urn:ietf:params:xml:ns:netconf:base:1.0">
  <network-instances xmlns="http://openconfig.net/yang/network-instance">
   <network-instance>
    <name>default</name>
    <protocols>
     <protocol>
      <identifier xmlns:idx="http://openconfig.net/yang/policy-types">idx:BGP</identifier>
      <name>default</name>
      <config>
       <identifier xmlns:idx="http://openconfig.net/yang/policy-types">idx:BGP</identifier>
       <name>default</name>
      </config>
     </protocol>
    </protocols>
    <config>
     <name>default</name>
    </config>
   </network-instance>
  </network-instances>
</data> 

Your Script

CODEC = CodecService()

JSON_PROVIDER_CISCO_IOSXR_762 = CodecServiceProvider(type='json')
XML_PROVIDER_CISCO_IOSXR_762 = CodecServiceProvider(type='xml')
JSON_PROVIDER_CISCO_IOSXR_762.initialize('iosxr-762',
    '/Users/ldacol/ydk_vne/lib/python3.10/site-packages/ydk/models/cisco_ios_xr_762/_yang')
XML_PROVIDER_CISCO_IOSXR_762.initialize('iosxr-762',
    '/Users/ldacol/ydk_vne/lib/python3.10/site-packages/ydk/models/cisco_ios_xr_762/_yang')

JSON_PROVIDER = JSON_PROVIDER_CISCO_IOSXR_762
XML_PROVIDER = XML_PROVIDER_CISCO_IOSXR_762

    with open(xml_file, "rb") as f:
        config_xml = f.read()
    root = etree.fromstring(config_xml) 

    for child in root:
        decoded_xml = CODEC.decode(XML_PROVIDER, etree.tostring(child))
        yang_json = CODEC.encode(JSON_PROVIDER, decoded_xml)

Logs

Enable logging and post the logs below

Traceback (most recent call last):
  File "/Users/ldacol/ydk_vne/lib/python3.10/site-packages/ydk/errors/error_handler.py", line 50, in handle_runtime_error
    yield
  File "/Users/ldacol/ydk_vne/lib/python3.10/site-packages/ydk/services/codec_service.py", line 122, in _encode
    data_node = _get_data_node_from_entity(entity, root_schema)
  File "/Users/ldacol/ydk_vne/lib/python3.10/site-packages/ydk/types/py_types.py", line 231, in has_operation
    for v in value:
  File "/Users/ldacol/ydk_vne/lib/python3.10/site-packages/ydk/types/py_types.py", line 683, in __iter__
    return iter(self.entities())
  File "/Users/ldacol/ydk_vne/lib/python3.10/site-packages/ydk/types/py_types.py", line 781, in entities
    self._flush_cache()
  File "/Users/ldacol/ydk_vne/lib/python3.10/site-packages/ydk/types/py_types.py", line 753, in _flush_cache
    self._entity_map[self._key(entity)] = entity
TypeError: unhashable type: 'BGP'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/ldacol/Code/GNSA/cisco_convert_versions_xr.py", line 237, in <module>
    yang_xml_to_json(ARGS['xml_file'], JSON_PROVIDER, XML_PROVIDER)
  File "/Users/ldacol/Code/GNSA/cisco_convert_versions_xr.py", line 223, in yang_xml_to_json
    yang_json = CODEC.encode(JSON_PROVIDER, decoded_xml)
  File "/Users/ldacol/ydk_vne/lib/python3.10/site-packages/ydk/errors/error_handler.py", line 99, in helper
    return func(self, provider, entity, *args, **kwargs)
  File "/Users/ldacol/ydk_vne/lib/python3.10/site-packages/ydk/services/codec_service.py", line 92, in encode
    return self._encode(provider, entity_holder, pretty, subtree)
  File "/Users/ldacol/ydk_vne/lib/python3.10/site-packages/ydk/services/codec_service.py", line 121, in _encode
    with _handle_error():
  File "/usr/local/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/ldacol/ydk_vne/lib/python3.10/site-packages/ydk/errors/error_handler.py", line 69, in handle_runtime_error
    raise _exc
ydk.errors.YServiceError:  'BGP'

System Information

MacOS 12.6.1 Python 3.10.8

ydk @ file:///Users/ldacol/Code/GNSA/ydk-gen/gen-api/python/ydk/dist/ydk-0.8.6.3.tar.gz ydk-models-cisco-ios-xe-1651 @ file:///Users/ldacol/Code/GNSA/ydk-gen/gen-api/python/cisco_ios_xe_1651-bundle/dist/ydk-models-cisco-ios-xe-1651-16.5.1.tar.gz ydk-models-cisco-ios-xr-762 @ file:///Users/ldacol/Code/GNSA/ydk-gen/gen-api/python/cisco_ios_xr_762-bundle/dist/ydk-models-cisco-ios-xr-762-7.6.2.tar.gz ydk-models-ietf==0.1.6 ydk-models-juniper-junos-18-4R3 @ file:///Users/ldacol/Code/GNSA/ydk-gen/gen-api/python/juniper_junos_18_4R3-bundle/dist/ydk-models-juniper-junos-18-4R3-18.4.post3.tar.gz ydk-models-juniper-qfx-20-2R1 @ file:///Users/ldacol/Code/GNSA/ydk-gen/gen-api/python/juniper_qfx_20_2R1-bundle/dist/ydk-models-juniper-qfx-20-2R1-20.2.post1.tar.gz

ygorelik commented 1 year ago

Test script

from ydk.services import CodecService
from ydk.providers import CodecServiceProvider

import logging

def enable_logging(level):
    log = logging.getLogger('ydk')
    log.setLevel(level)
    handler = logging.StreamHandler()
    formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
    handler.setFormatter(formatter)
    log.addHandler(handler)
    return log

xml_payload = """
  <network-instances xmlns="http://openconfig.net/yang/network-instance">
   <network-instance>
    <name>default</name>
    <protocols>
     <protocol>
      <identifier xmlns:idx="http://openconfig.net/yang/policy-types">idx:BGP</identifier>
      <name>default</name>
      <config>
       <identifier xmlns:idx="http://openconfig.net/yang/policy-types">idx:BGP</identifier>
       <name>default</name>
      </config>
     </protocol>
    </protocols>
    <config>
     <name>default</name>
    </config>
   </network-instance>
  </network-instances>
"""

def xml_to_json(xml: str) -> str:
    codec = CodecService()
    provider = CodecServiceProvider(type='xml')
    json_provider = CodecServiceProvider(type='json')
    try:
        model = codec.decode(provider, xml)
        json = codec.encode(json_provider, model)
        return json
    except Exception as err:
        print(f"Exception: {err}")
        return None

if __name__ == "__main__":
    enable_logging(logging.INFO)
    json_payload = xml_to_json(xml_payload)
    if json_payload:
        print(json_payload)
    else:
        print("Failed to transform XML payload to JSON")

Output

/home/ubuntu/venv/bin/python3 /home/ubuntu/ydk-gen/scripts/issues/1071/xml_to_json.py 
{
  "openconfig-network-instance:network-instances": {
    "network-instance": [
      {
        "name": "default",
        "config": {
          "name": "default"
        },
        "protocols": {
          "protocol": [
            {
              "identifier": "openconfig-policy-types:BGP",
              "name": "default",
              "config": {
                "identifier": "openconfig-policy-types:BGP",
                "name": "default"
              }
            }
          ]
        }
      }
    ]
  }
}

Process finished with exit code 0