search

CiscoDevNet / ydk-gen

Generate model-driven APIs from YANG models
http://ciscodevnet.github.io/ydk-gen/
Apache License 2.0
135 stars 74 forks source link

ydk.path.codec.decode validates incorrect XML payload #1026

Closed tdesh10 closed 2 years ago

tdesh10 commented 4 years ago

Expected Behavior

If an XML payload contains an error in a leaf, list, or a container, ydk.path.code.decode() should throw an exception.

Current Behavior

If an XML payload contains an error in a leaf, list, or a container, ydk.path.code.decode() does not throw an exception.

Steps to Reproduce

Run my script below using this directory as Repository: pavarotti.zip

pyang output of Cisco-IOS-XR-um-interface-cfg.yang and Cisco-IOS-XR-um-if-ip-address-cfg.yang:

module: Cisco-IOS-XR-um-interface-cfg
  +--rw interfaces
     +--rw interface* [interface-name]
     |  +--rw interface-name        xr:Interface-name
     |  +--rw sub-interface-type
     |  |  +--rw l2transport!
     |  |  +--rw point-to-point!
     |  |  +--rw multipoint!
     |  +--rw ipv4
     |  |  +--rw um-if-ip-address-cfg:addresses
     |  |     +--rw um-if-ip-address-cfg:address!
     |  |     |  +--rw um-if-ip-address-cfg:address      inet:ipv4-address-no-zone
     |  |     |  +--rw um-if-ip-address-cfg:netmask      inet:ipv4-address-no-zone
     |  |     |  +--rw um-if-ip-address-cfg:route-tag?   uint32
     |  |     +--rw um-if-ip-address-cfg:secondaries
     |  |     |  +--rw um-if-ip-address-cfg:secondary* [address]
     |  |     |     +--rw um-if-ip-address-cfg:address      inet:ipv4-address-no-zone
     |  |     |     +--rw um-if-ip-address-cfg:netmask      inet:ipv4-address-no-zone
     |  |     |     +--rw um-if-ip-address-cfg:route-tag?   uint32
     |  |     +--rw um-if-ip-address-cfg:unnumbered?    xr:Interface-name
     |  |     +--rw um-if-ip-address-cfg:dhcp!
     |  +--rw ipv6
     |  |  +--rw um-if-ip-address-cfg:addresses
     |  |  |  +--rw um-if-ip-address-cfg:address* [address]
     |  |  |  |  +--rw um-if-ip-address-cfg:address          inet:ip-address-no-zone
     |  |  |  |  +--rw um-if-ip-address-cfg:prefix-length    xr:Ipv6-prefix-length
     |  |  |  |  +--rw um-if-ip-address-cfg:zone?            string
     |  |  |  |  +--rw um-if-ip-address-cfg:route-tag?       uint32
     |  |  |  +--rw um-if-ip-address-cfg:link-local-address!
     |  |  |  |  +--rw um-if-ip-address-cfg:address      inet:ip-address-no-zone
     |  |  |  |  +--rw um-if-ip-address-cfg:zone?        string
     |  |  |  |  +--rw um-if-ip-address-cfg:route-tag?   uint32
     |  |  |  +--rw um-if-ip-address-cfg:eui64-addresses
     |  |  |  |  +--rw um-if-ip-address-cfg:eui64-address* [address]
     |  |  |  |     +--rw um-if-ip-address-cfg:address          inet:ip-address-no-zone
     |  |  |  |     +--rw um-if-ip-address-cfg:prefix-length    xr:Ipv6-prefix-length
     |  |  |  |     +--rw um-if-ip-address-cfg:zone?            string
     |  |  |  |     +--rw um-if-ip-address-cfg:route-tag?       uint32
     |  |  |  +--rw um-if-ip-address-cfg:autoconfig!
     |  |  +--rw um-if-ip-address-cfg:enable!
     |  +--rw dampening!
     |  |  +--rw decay-half-life!
     |  |     +--rw value              uint32
     |  |     +--rw reuse-threshold!
     |  |        +--rw value                 uint32
     |  |        +--rw suppress-threshold!
     |  |           +--rw value                uint32
     |  |           +--rw max-suppress-time!
     |  |              +--rw value              uint32
     |  |              +--rw restart-penalty!
     |  |                 +--rw value    uint32
     |  +--rw encapsulation
     |  |  +--rw ppp!
     |  |  +--rw hdlc!
     |  |  +--rw mfr!
     |  |  +--rw frame-relay!
     |  |     +--rw IETF!
     |  +--rw shutdown!
     |  +--rw mtu?                  uint32
     |  +--rw logging
     |  |  +--rw events
     |  |     +--rw link-status!
     |  +--rw bandwidth?            uint32
     |  +--rw description?          string
     +--rw interface-preconfigure* [interface-name]
        +--rw interface-name        xr:Interface-name
        +--rw sub-interface-type
        |  +--rw l2transport!
        |  +--rw point-to-point!
        |  +--rw multipoint!
        +--rw ipv4
        |  +--rw um-if-ip-address-cfg:addresses
        |     +--rw um-if-ip-address-cfg:address!
        |     |  +--rw um-if-ip-address-cfg:address      inet:ipv4-address-no-zone
        |     |  +--rw um-if-ip-address-cfg:netmask      inet:ipv4-address-no-zone
        |     |  +--rw um-if-ip-address-cfg:route-tag?   uint32
        |     +--rw um-if-ip-address-cfg:secondaries
        |     |  +--rw um-if-ip-address-cfg:secondary* [address]
        |     |     +--rw um-if-ip-address-cfg:address      inet:ipv4-address-no-zone
        |     |     +--rw um-if-ip-address-cfg:netmask      inet:ipv4-address-no-zone
        |     |     +--rw um-if-ip-address-cfg:route-tag?   uint32
        |     +--rw um-if-ip-address-cfg:unnumbered?    xr:Interface-name
        |     +--rw um-if-ip-address-cfg:dhcp!
        +--rw ipv6
        |  +--rw um-if-ip-address-cfg:addresses
        |  |  +--rw um-if-ip-address-cfg:address* [address]
        |  |  |  +--rw um-if-ip-address-cfg:address          inet:ip-address-no-zone
        |  |  |  +--rw um-if-ip-address-cfg:prefix-length    xr:Ipv6-prefix-length
        |  |  |  +--rw um-if-ip-address-cfg:zone?            string
        |  |  |  +--rw um-if-ip-address-cfg:route-tag?       uint32
        |  |  +--rw um-if-ip-address-cfg:link-local-address!
        |  |  |  +--rw um-if-ip-address-cfg:address      inet:ip-address-no-zone
        |  |  |  +--rw um-if-ip-address-cfg:zone?        string
        |  |  |  +--rw um-if-ip-address-cfg:route-tag?   uint32
        |  |  +--rw um-if-ip-address-cfg:eui64-addresses
        |  |  |  +--rw um-if-ip-address-cfg:eui64-address* [address]
        |  |  |     +--rw um-if-ip-address-cfg:address          inet:ip-address-no-zone
        |  |  |     +--rw um-if-ip-address-cfg:prefix-length    xr:Ipv6-prefix-length
        |  |  |     +--rw um-if-ip-address-cfg:zone?            string
        |  |  |     +--rw um-if-ip-address-cfg:route-tag?       uint32
        |  |  +--rw um-if-ip-address-cfg:autoconfig!
        |  +--rw um-if-ip-address-cfg:enable!
        +--rw dampening!
        |  +--rw decay-half-life!
        |     +--rw value              uint32
        |     +--rw reuse-threshold!
        |        +--rw value                 uint32
        |        +--rw suppress-threshold!
        |           +--rw value                uint32
        |           +--rw max-suppress-time!
        |              +--rw value              uint32
        |              +--rw restart-penalty!
        |                 +--rw value    uint32
        +--rw encapsulation
        |  +--rw ppp!
        |  +--rw hdlc!
        |  +--rw mfr!
        |  +--rw frame-relay!
        |     +--rw IETF!
        +--rw shutdown!
        +--rw mtu?                  uint32
        +--rw logging
        |  +--rw events
        |     +--rw link-status!
        +--rw bandwidth?            uint32
        +--rw description?          string

Your Script

from ydk.path import Repository, Capability, Codec
from ydk.types import EncodingFormat

# import logging
# log = logging.getLogger('ydk')
# log.setLevel(logging.DEBUG)
# handler = logging.StreamHandler()
# log.addHandler(handler)

correct_xml = """
<interfaces xmlns="http://cisco.com/ns/yang/Cisco-IOS-XR-um-interface-cfg">
   <interface>
    <interface-name>Loopback0</interface-name>
    <description>interface description</description>
    <ipv4>
     <addresses xmlns="http://cisco.com/ns/yang/Cisco-IOS-XR-um-if-ip-address-cfg">
      <address>
       <address>172.16.255.101</address>
       <netmask>255.255.255.255</netmask>
      </address>
     </addresses>
    </ipv4>
   </interface>
</interfaces>"""

incorrect_xml = """
<interfaces xmlns="http://cisco.com/ns/yang/Cisco-IOS-XR-um-interface-cfg">
   <interfacez>
    <interface-namez>Loopback0</interface-namez>
    <description>interface description</description>
    <ipv4z>
     <addresses xmlns="http://cisco.com/ns/yang/Cisco-IOS-XR-um-if-ip-address-cfg">
      <address>
       <address>172.16.255.101</address>
       <netmask>255.255.255.255</netmask>
      </address>
     </addresses>
    </ipv4z>
   </interfacez>
</interfaces>"""

repo = Repository("pavarotti")

root_schema = repo.create_root_schema([Capability('Cisco-IOS-XR-um-interface-cfg', ''), Capability('Cisco-IOS-XR-um-if-ip-address-cfg', ''), Capability('Cisco-IOS-XR-um-if-ip-address-cfg', '')])
codec = Codec()

data_node = codec.decode(root_schema, incorrect_xml, EncodingFormat.XML)

Logs

Enable logging and post the logs below

Creating libyang context in path: pavarotti
[libyang] Extension plugin "/usr/local/lib/libyang/libyang_ext_test.so" successfully loaded.
[libyang] Extension plugin "/usr/local/lib/libyang/metadata.so" successfully loaded.
[libyang] Extension plugin "/usr/local/lib/libyang/nacm.so" successfully loaded.
[libyang] Reading module "ietf-yang-metadata".
[libyang] Module "ietf-yang-metadata@2016-08-05" successfully parsed as implemented.
[libyang] Reading module "yang".
[libyang] Resolving "yang" unresolved schema nodes and their constraints...
[libyang] All "yang" schema nodes and constraints resolved.
[libyang] Module "yang@2017-02-20" successfully parsed as implemented.
[libyang] Reading module "ietf-inet-types".
[libyang] Resolving derived type "union" failed, it will be attempted later.
[libyang] Resolving derived type "union" failed, it will be attempted later.
[libyang] Resolving derived type "union" failed, it will be attempted later.
[libyang] Resolving derived type "union" failed, it will be attempted later.
[libyang] Resolving "ietf-inet-types" unresolved schema nodes and their constraints...
[libyang] All "ietf-inet-types" schema nodes and constraints resolved.
[libyang] Module "ietf-inet-types@2013-07-15" successfully parsed as implemented.
[libyang] Reading module "ietf-yang-types".
[libyang] Module "ietf-yang-types@2013-07-15" successfully parsed as implemented.
[libyang] Reading module "ietf-datastores".
[libyang] Module "ietf-datastores@2017-08-17" successfully parsed as implemented.
[libyang] Reading module "ietf-yang-library".
[libyang] Module "ietf-yang-library@2017-08-17" successfully parsed as implemented.
Loading module 'Cisco-IOS-XR-um-interface-cfg', revision ''
[libyang] Searching for "Cisco-IOS-XR-um-interface-cfg" in /root/ydk-py/pathapi-taran/pavarotti.
[libyang] Searching for "Cisco-IOS-XR-um-interface-cfg" in /root/ydk-py/pathapi-taran.
[libyang] Loading schema from "/root/ydk-py/pathapi-taran/pavarotti/Cisco-IOS-XR-um-interface-cfg.yang" file.
[libyang] Searching for "cisco-semver" in /root/ydk-py/pathapi-taran/pavarotti.
[libyang] Searching for "cisco-semver" in /root/ydk-py/pathapi-taran.
[libyang] Loading schema from "/root/ydk-py/pathapi-taran/pavarotti/cisco-semver.yang" file.
[libyang] Module "cisco-semver@2019-03-13" successfully parsed as implemented.
[libyang] Searching for "Cisco-IOS-XR-types" in /root/ydk-py/pathapi-taran/pavarotti.
[libyang] Searching for "Cisco-IOS-XR-types" in /root/ydk-py/pathapi-taran.
[libyang] Loading schema from "/root/ydk-py/pathapi-taran/pavarotti/Cisco-IOS-XR-types.yang" file.
[libyang] Module "Cisco-IOS-XR-types@2019-04-05" successfully parsed as implemented.
[libyang] Searching for "tailf-common" in /root/ydk-py/pathapi-taran/pavarotti.
[libyang] Searching for "tailf-common" in /root/ydk-py/pathapi-taran.
[libyang] Loading schema from "/root/ydk-py/pathapi-taran/pavarotti/tailf-common.yang" file.
[libyang] Searching for "tailf-meta-extensions" in /root/ydk-py/pathapi-taran/pavarotti.
[libyang] Loading schema from "/root/ydk-py/pathapi-taran/pavarotti/tailf-meta-extensions@2017-03-08.yang" file.
[libyang] Resolving extension "(null)" failed, it will be attempted later.
[libyang] Resolving extension "(null)" failed, it will be attempted later.
[libyang] Resolving extension "(null)" failed, it will be attempted later.
[libyang] Resolving extension "(null)" failed, it will be attempted later.
[libyang] Resolving extension "(null)" failed, it will be attempted later.
[libyang] Resolving extension "(null)" failed, it will be attempted later.
[libyang] Resolving extension "(null)" failed, it will be attempted later.
[libyang] Resolving extension "(null)" failed, it will be attempted later.
[libyang] Resolving extension "(null)" failed, it will be attempted later.
[libyang] Resolving extension "(null)" failed, it will be attempted later.
[libyang] Resolving extension "(null)" failed, it will be attempted later.
[libyang] Resolving extension "(null)" failed, it will be attempted later.
[libyang] Submodule "tailf-meta-extensions" successfully parsed.
[libyang] Searching for "tailf-cli-extensions" in /root/ydk-py/pathapi-taran/pavarotti.
[libyang] Loading schema from "/root/ydk-py/pathapi-taran/pavarotti/tailf-cli-extensions@2018-09-11.yang" file.
[libyang] Submodule "tailf-cli-extensions" successfully parsed.
[libyang] Resolving "tailf-common" unresolved schema nodes and their constraints...
[libyang] All "tailf-common" schema nodes and constraints resolved.
[libyang] Module "tailf-common@2018-09-11" successfully parsed as implemented.
[libyang] Resolving "Cisco-IOS-XR-um-interface-cfg" unresolved schema nodes and their constraints...
[libyang] All "Cisco-IOS-XR-um-interface-cfg" schema nodes and constraints resolved.
[libyang] Module "Cisco-IOS-XR-um-interface-cfg@2019-06-10" successfully parsed as implemented.
Loading module 'Cisco-IOS-XR-um-if-ip-address-cfg', revision ''
[libyang] Searching for "Cisco-IOS-XR-um-if-ip-address-cfg" in /root/ydk-py/pathapi-taran/pavarotti.
[libyang] Searching for "Cisco-IOS-XR-um-if-ip-address-cfg" in /root/ydk-py/pathapi-taran.
[libyang] Loading schema from "/root/ydk-py/pathapi-taran/pavarotti/Cisco-IOS-XR-um-if-ip-address-cfg.yang" file.
[libyang] Resolving "Cisco-IOS-XR-um-if-ip-address-cfg" unresolved schema nodes and their constraints...
[libyang] All "Cisco-IOS-XR-um-if-ip-address-cfg" schema nodes and constraints resolved.
[libyang] Module "Cisco-IOS-XR-um-if-ip-address-cfg@2019-06-10" successfully parsed as implemented.
Loading module 'Cisco-IOS-XR-um-if-ip-address-cfg', revision ''
The module 'Cisco-IOS-XR-um-if-ip-address-cfg' schema present in Libyang repository, but not in YDK; consider populate
Populating new module schema 'ietf-yang-metadata'
Populating new module schema 'yang'
Populating new module schema 'ietf-inet-types'
Populating new module schema 'ietf-yang-types'
Populating new module schema 'ietf-datastores'
Populating new module schema 'ietf-yang-library'
Populating new module schema 'cisco-semver'
Populating new module schema 'Cisco-IOS-XR-types'
Populating new module schema 'tailf-common'
Populating new module schema 'Cisco-IOS-XR-um-interface-cfg'
Populating new module schema 'Cisco-IOS-XR-um-if-ip-address-cfg'
ydk::path::Codec: Decoding from XML formatted payload:

<interfaces xmlns="http://cisco.com/ns/yang/Cisco-IOS-XR-um-interface-cfg">
   <interfacez>
    <interface-namez>Loopback0</interface-namez>
    <description>interface description</description>
    <ipv4z>
     <addresses xmlns="http://cisco.com/ns/yang/Cisco-IOS-XR-um-if-ip-address-cfg">
      <address>
       <address>172.16.255.101</address>
       <netmask>255.255.255.255</netmask>
      </address>
     </addresses>
    </ipv4z>
   </interfacez>
</interfaces>
Populating new schema from payload:

<interfaces xmlns="http://cisco.com/ns/yang/Cisco-IOS-XR-um-interface-cfg">
   <interfacez>
    <interface-namez>Loopback0</interface-namez>
    <description>interface description</description>
    <ipv4z>
     <addresses xmlns="http://cisco.com/ns/yang/Cisco-IOS-XR-um-if-ip-address-cfg">
      <address>
       <address>172.16.255.101</address>
       <netmask>255.255.255.255</netmask>
      </address>
     </addresses>
    </ipv4z>
   </interfacez>
</interfaces>
Extracting module namespaces from XML payload
Failed to find namespace 'http://cisco.com/ns/yang/Cisco-IOS-XR-um-if-ip-address-cfg' in lookup table
Failed to find namespace 'http://cisco.com/ns/yang/Cisco-IOS-XR-um-interface-cfg' in lookup table
Performing decode operation

System Information

Python 3.5.2 in the ydk docker container, using ydk (0.8.5).

ygorelik commented 3 years ago

Fixed this error handling in Libyang.

Please recompile and reinstall YDK C++ and Python core components to address the issue.

Here is the test script:

from ydk.path import Repository, Capability, Codec
from ydk.types import EncodingFormat

import logging
import sys

incorrect_xml = """
<interfaces xmlns="http://cisco.com/ns/yang/Cisco-IOS-XR-um-interface-cfg">
   <interfacez>
    <interface-name>Loopback0</interface-name>
    <description>interface description</description>
    <ipv4z>
     <addresses xmlns="http://cisco.com/ns/yang/Cisco-IOS-XR-um-if-ip-address-cfg">
      <address>
       <address>172.16.255.101</address>
       <netmask>255.255.255.255</netmask>
      </address>
     </addresses>
    </ipv4z>
   </interfacez>
</interfaces>"""

def enable_logging(level):
    log = logging.getLogger('ydk')
    log.setLevel(level)
    handler = logging.StreamHandler()
    formatter = logging.Formatter(
            "%(asctime)s - %(name)s - %(levelname)s - %(message)s")
    handler.setFormatter(formatter)
    log.addHandler(handler)

def perform_operation(root, payload):
    codec = Codec()
    try:
        data_node = codec.decode(root, payload, EncodingFormat.XML)
        print("Correctly decoded payload to DataNode object")
    except RuntimeError as err:
        print("Exception occurred %s" % err)

if __name__ == '__main__':
    enable_logging(logging.INFO)
    repo = Repository("pavarotti")

    lookup_table = {'http://cisco.com/ns/yang/Cisco-IOS-XR-um-interface-cfg': Capability('Cisco-IOS-XR-um-interface-cfg', ''),
                    'http://cisco.com/ns/yang/Cisco-IOS-XR-um-if-ip-address-cfg': Capability('Cisco-IOS-XR-um-if-ip-address-cfg', ''),
                    }
    capabilities = [Capability('Cisco-IOS-XR-um-interface-cfg', ''),
                    Capability('Cisco-IOS-XR-um-if-ip-address-cfg', ''),
                    ]
    root_schema = repo.create_root_schema(lookup_table, capabilities)

    perform_operation(root_schema, incorrect_xml)

And run results:

2020-10-29 17:26:11,271 - ydk - ERROR - Data is invalid according to the yang model. Libyang error: Unknown element "interfacez". Path: '/Cisco-IOS-XR-um-interface-cfg:interfaces'
2020-10-29 17:26:11,272 - ydk - ERROR - Parsing failed with message Unknown element "interfacez".
Exception occurred YCoreError: YCodecError:Unknown element "interfacez".. Path: /Cisco-IOS-XR-um-interface-cfg:interfaces

Process finished with exit code 0

Please Note. Due to issue #998, all operations on DataNode objects must be separated from RootSchemaNode object. That is why the decode operation is isolated in a function _performoperation().

ygorelik commented 3 years ago

This issue has been fixed in Libyang code.