CiscoDevNet / ydk-gen

Generate model-driven APIs from YANG models
http://ciscodevnet.github.io/ydk-gen/
Apache License 2.0
135 stars 74 forks source link

cannot decode json blob in python without encoding it first #1089

Closed hyberdk closed 1 year ago

hyberdk commented 1 year ago

Issue tracker is ONLY used for reporting bugs. Please use the YDK Community for any support issues.

Expected Behavior

I should be able to decode a json blob with an access-list

Current Behavior

I can only decode the test_test access list json blob, if I have "encoded" an access-list before with the same name,

Steps to Reproduce

  1. run the script below, it works.
(venv) eslau@N503476:~/repos-wsl/ydk_poc$ python3 decode_err.py 
{
  "Cisco-IOS-XE-native:native": {
    "ip": {
      "access-list": {
        "Cisco-IOS-XE-acl:extended": [
          {
            "name": "TEST_TEST",
            "access-list-seq-rule": [
              {
                "sequence": "10",
                "ace-rule": {
                  "action": "permit",
                  "protocol": "tcp",
                  "any": [null],
                  "dst-any": [null]
                }
              },
              {
                "sequence": "20",
                "ace-rule": {
                  "action": "deny",
                  "protocol": "udp",
                  "any": [null],
                  "dst-any": [null]
                }
              },
              {
                "sequence": "30",
                "ace-rule": {
                  "action": "permit",
                  "protocol": "ip",
                  "any": [null],
                  "dst-any": [null]
                }
              }
            ]
          }
        ]
      }
    }
  }
}
  1. comment out the line: json = codec.encode(codec_provider, native) and run it again, now it fails..
(venv) eslau@N503476:~/repos-wsl/ydk_poc$ python3 decode_err.py 
Traceback (most recent call last):
  File "/home/eslau/repos-wsl/ydk_poc/decode_err.py", line 75, in <module>
    decoded = codec.decode(codec_provider, json)
  File "/home/eslau/repos-wsl/ydk_poc/venv/lib/python3.10/site-packages/ydk/errors/error_handler.py", line 99, in helper
    return func(self, provider, entity, *args, **kwargs)
  File "/home/eslau/repos-wsl/ydk_poc/venv/lib/python3.10/site-packages/ydk/services/codec_service.py", line 156, in decode
    return self._decode(provider, payload_holder, subtree)
  File "/home/eslau/repos-wsl/ydk_poc/venv/lib/python3.10/site-packages/ydk/services/codec_service.py", line 189, in _decode
    root_data_node = codec_service.decode(root_schema, payload, provider.encoding)
RuntimeError: YCoreError: YCodecError:Unknown element "extended".. Path: /Cisco-IOS-XE-native:native/ip/access-list

Your Script

from ydk.models.cisco_ios_xe import Cisco_IOS_XE_native as xe_native
from ydk.services import CodecService
from ydk.providers import CodecServiceProvider
import logging

if __name__ == "__main__":

    logger = logging.getLogger("ydk")
    logger.setLevel(logging.DEBUG)
    handler = logging.FileHandler('ydk.log')
    formatter = logging.Formatter(("%(asctime)s - %(name)s - "
                                    "%(levelname)s - %(message)s"))
    handler.setFormatter(formatter)
    logger.addHandler(handler)

    access_list = xe_native.Native.Ip.AccessList()
    acl = xe_native.Native.Ip.AccessList.Extended()
    acl.name = "TEST_TEST"
    access_list.extended.append(acl)
    ip = xe_native.Native.Ip()
    ip.access_list = access_list
    native = xe_native.Native()
    native.ip = ip

    codec_provider = CodecServiceProvider(type='json')
    codec = CodecService()
    json = codec.encode(codec_provider, native)

    json = '''
    {
    "Cisco-IOS-XE-native:native": {
        "ip": {
        "access-list": {
            "Cisco-IOS-XE-acl:extended": [
            {
                "name": "TEST_TEST",
                "access-list-seq-rule": [
                {
                    "sequence": "10",
                    "ace-rule": {
                    "action": "permit",
                    "protocol": "tcp",
                    "any": [null],
                    "dst-any": [null]
                    }
                },
                {
                    "sequence": "20",
                    "ace-rule": {
                    "action": "deny",
                    "protocol": "udp",
                    "any": [null],
                    "dst-any": [null]
                    }
                },
                {
                    "sequence": "30",
                    "ace-rule": {
                    "action": "permit",
                    "protocol": "ip",
                    "any": [null],
                    "dst-any": [null]
                    }
                }
                ]
            }
            ]
        }
        }
    }
    }
    '''

    decoded = codec.decode(codec_provider, json)

    print(codec.encode(codec_provider, decoded))

Logs

Enable logging and post the logs below (note that the link does not work!) See #1088

logs are attached as files.. ydk.log ydk-ok.log

System Information

(venv) eslau@N503476:~/repos-wsl/ydk_poc$ pip list
Package                        Version
------------------------------ ------------
bcrypt                         4.0.1
cffi                           1.15.1
cryptography                   41.0.1
lxml                           4.9.2
ncclient                       0.6.13
paramiko                       3.2.0
pip                            22.0.2
pyang                          1.6
pybind11                       2.6.2
pycparser                      2.21
PyNaCl                         1.5.0
setuptools                     59.6.0
six                            1.16.0
wheel                          0.40.0
ydk                            0.8.6.2
ydk-models-cisco-ios-xe        16.9.3.post1
ydk-models-cisco-ios-xe-native 17.6.1.dev5
ydk-models-ietf                0.1.6

(venv) eslau@N503476:~/repos-wsl/ydk_poc$ python3 --version
Python 3.10.6

(venv) eslau@N503476:~/repos-wsl/ydk_poc$ uname -a
Linux N503476 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
(venv) eslau@N503476:~/repos-wsl/ydk_poc$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.2 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.2 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
ygorelik commented 1 year ago

Please consider this piece of code before the actual decode as a workaround:

    acl = xe_native.Native.Ip.AccessList.Extended()
    acl.name = "TEST_TEST"
    native = xe_native.Native()
    native.ip.access_list.extended.append(acl)

    json = codec.encode(codec_provider, native)
ygorelik commented 1 year ago

Built test case in C++ and reproduced the issue:

#include <iostream>
#include <spdlog/spdlog.h>

#include <ydk/codec_service.hpp>
#include <ydk/codec_provider.hpp>

#include <ydk_cisco_iosxe_demo/Cisco_IOS_XE_native.hpp>

using namespace ydk;
using namespace std;

string json_payload = R"(
{
  "Cisco-IOS-XE-native:native":
  {
    "ip":
    {
      "access-list":
      {
        "Cisco-IOS-XE-acl:extended":
        [
          {
            "name": "TEST_TEST",
            "access-list-seq-rule":
            [
              {
                "sequence": "10",
                "ace-rule": {
                  "action": "permit",
                  "protocol": "tcp"
                }
              }
            ]
          }
        ]
      }
    }
  }
}
)";

int main(int argc, char* argv[])
{
    auto logger = spdlog::stdout_color_mt("ydk");
    logger->set_level(spdlog::level::debug);

    CodecService codec{};
    CodecServiceProvider xml_provider(EncodingFormat::XML);
    CodecServiceProvider json_provider(EncodingFormat::JSON);

    auto top = make_shared<cisco_iosxe_demo::Cisco_IOS_XE_native::Native>();
    auto model = codec.decode(json_provider, json_payload, top);

    auto xml = codec.encode(xml_provider, *model, true);
    cout << json << endl;

    return 0;
}

Found and fixed the bug. After the fix applied getting expected result:

[2023-09-07 08:23:32.134] [ydk] [info] Performing encode operation, resulting in <native xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-native">
  <ip>
    <access-list>
      <extended xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-acl">
        <name>TEST_TEST</name>
        <access-list-seq-rule>
          <sequence>10</sequence>
          <ace-rule>
            <action>permit</action>
            <protocol>tcp</protocol>
          </ace-rule>
        </access-list-seq-rule>
      </extended>
    </access-list>
  </ip>
</native>

Got the same result in Python API with original script (slightly modified):

from ydk.services import CodecService
from ydk.providers import CodecServiceProvider
import logging
import os

if __name__ == "__main__":

    logger = logging.getLogger("ydk")
    logger.setLevel(logging.INFO)
    handler = logging.StreamHandler()
    formatter = logging.Formatter("%(asctime)s - %(name)s - "
                                  "%(levelname)s - %(message)s")
    handler.setFormatter(formatter)
    logger.addHandler(handler)
    if os.path.exists('ydk.log'):
        with open('ydk.log', 'w'):
            pass

    codec_provider = CodecServiceProvider(type='json')
    codec = CodecService()

    json = '''
{
  "Cisco-IOS-XE-native:native":
  {
    "ip":
    {
      "access-list":
      {
        "Cisco-IOS-XE-acl:extended":
        [
          {
            "name": "TEST_TEST",
            "access-list-seq-rule":
            [
              {
                "sequence": "10",
                "ace-rule": {
                  "action": "permit",
                  "protocol": "tcp"
                }
              }
            ]
          }
        ]
      }
    }
  }
}
'''
    decoded = codec.decode(codec_provider, json)

    print(codec.encode(codec_provider, decoded))

Result:

/Users/ygorelik/venv3.10/bin/python3 /Users/ygorelik/ydk-gen/scripts/issues/1089/xe-native-access-list.py
{
  "Cisco-IOS-XE-native:native": {
    "ip": {
      "access-list": {
        "Cisco-IOS-XE-acl:extended": [
          {
            "name": "TEST_TEST",
            "access-list-seq-rule": [
              {
                "sequence": "10",
                "ace-rule": {
                  "action": "permit",
                  "protocol": "tcp"
                }
              }
            ]
          }
        ]
      }
    }
  }
}

Process finished with exit code 0

Hence closing the issue.