search

CiscoDevNet / ydk-gen

Generate model-driven APIs from YANG models
http://ciscodevnet.github.io/ydk-gen/
Apache License 2.0
135 stars 74 forks source link

Incorrect loading of submodules into YANG repository #1094

Closed hyberdk closed 10 months ago

hyberdk commented 10 months ago

When YDK-GEN downloads the models and put them into the repository (I guess by default in ~/.ydk/<ip>/ it seems like sub-modules are named using the model@revision.yang notation. I guess this should be fine, excepts that the versions seems to be wrong.

Expected Behavior

The revision of the sub-module are versioned with the latest revision in the sub-module, or not versioned at all.

Current Behavior

From the looks of it, it seems like the revision of the sub-module are coming from the parent (belongs-to) module. These revisions are not always the same, just an example here:

https://github.com/YangModels/yang/blob/43f84815fdbf4b374ec58e7a424610bea8c492f1/vendor/cisco/xe/1771/Cisco-IOS-XE-native.yang#L45 vs. https://github.com/YangModels/yang/blob/43f84815fdbf4b374ec58e7a424610bea8c492f1/vendor/cisco/xe/1771/Cisco-IOS-XE-parser.yang#L27C11-L27C22

there are also different revisions on the devices itself which are not in the public yang repo on github.

Steps to Reproduce

this is just an example, it does actually not need a working model to see it, but Im usng this code.

from ydk.services import NetconfService, Datastore
from ydk.providers import NetconfServiceProvider
from ydk.models.cisco_ios_xe import (
    Cisco_IOS_XE_native as xe_native
    )
from ydk.filters import YFilter

import logging

if __name__ == "__main__":
    logger = logging.getLogger("ydk")
    logger.setLevel(logging.DEBUG)
    handler = logging.StreamHandler()
    file_handler = logging.FileHandler("ydk.log")
    formatter = logging.Formatter(("%(asctime)s - %(name)s - "
                                    "%(levelname)s - %(message)s"))
    handler.setFormatter(formatter)
    logger.addHandler(handler)
    logger.addHandler(file_handler)

    provider = NetconfServiceProvider(address="<ip>",
                                      port=830,
                                      username="<username>",
                                      password="<password>",
                                      protocol="ssh")
    netconf = NetconfService()
    native = xe_native.Native()
    native.ip.access_list.yfilter = YFilter.read
    native.object_group.yfilter = YFilter.read

    result = netconf.get_config(provider, Datastore.running, read_filter=native)  # read running config
    exit()

once that has run, go to the repo folder and do a yanglint Cisco-IOS-XE-native.yang and observe that it complains about version mismatch. Or simply look at the revisions in the models :-)

here is an example for my ISR1K router running 17.6.5

eslau@N503476:~/.ydk/10.12.35.56$ cat Cisco-IOS-XE-native.yang |grep revision | head -1
  revision 2021-08-10 {
eslau@N503476:~/.ydk/10.12.35.56$ cat Cisco-IOS-XE-ip@2021-08-10.yang |grep revision | head -1
  revision 2021-07-10 {

System Information

(venv) eslau@N503476:~/repos-wsl/ydk_poc$ python --version
Python 3.10.12
(venv) eslau@N503476:~/repos-wsl/ydk_poc$ uname -a
Linux N503476 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
(venv) eslau@N503476:~/repos-wsl/ydk_poc$ 
(venv) eslau@N503476:~/repos-wsl/ydk_poc$ pip list
Package                       Version
----------------------------- -----------
alabaster                     0.7.13
Babel                         2.13.1
certifi                       2023.7.22
charset-normalizer            3.3.2
docutils                      0.18.1
gitdb                         4.0.11
GitPython                     3.1.40
idna                          3.4
imagesize                     1.4.1
Jinja2                        3.1.2
lxml                          4.9.3
MarkupSafe                    2.1.3
packaging                     23.2
pip                           22.0.2
pyang                         2.6.0
pybind11                      2.6.2
Pygments                      2.16.1
requests                      2.31.0
rstr                          3.2.2
setuptools                    59.6.0
smmap                         5.0.1
snowballstemmer               2.2.0
Sphinx                        7.2.6
sphinx-rtd-theme              1.3.0
sphinxcontrib-applehelp       1.0.7
sphinxcontrib-devhelp         1.0.5
sphinxcontrib-htmlhelp        2.0.4
sphinxcontrib-jquery          4.1
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.6
sphinxcontrib-serializinghtml 1.1.9
urllib3                       2.0.7
wheel                         0.41.3
ydk                           0.8.6.5
ydk-models-cisco-ios-xe       17.6.5.dev3
ygorelik commented 10 months ago

That must be an issue in the model on the device. The YDK gets list of modules from device capabilities. When needed the module is requested to be downloaded from the device.

As a workaround I suggest removing the revision from the model name in the repo. When YDK finds the model in the repo, it will not download it from the device:

cd ~/.ydk/10.12.35.56
mv Cisco-IOS-XE-ip@2021-08-10.yang Cisco-IOS-XE-ip.yang
hyberdk commented 10 months ago

If the model exist, but without the revision, it downloads the model again with revision, so my only option is to copy the model edit the yang-model update the revision for a workaround..

But if I use this script, to get the modules list, it does clearly states the correct version numbers.

from ncclient import manager
import xml.dom.minidom

device_params = {
    "host": "<ip>",
    "port": 830,
    "username": "<username>",
    "password": "<password>",
}

filter = ('subtree', """
    <modules-state xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-library">
    </modules-state>
    """)

with manager.connect(**device_params, hostkey_verify=False) as netconf_session:
    netconf_response = netconf_session.get(filter=filter)
    print(xml.dom.minidom.parseString(netconf_response.xml).toprettyxml())

here is a snip (output very long, but attached here: modules_output.xml.txt ) from the native model

                        <module>
                                <name>Cisco-IOS-XE-native</name>
                                <revision>2021-08-10</revision>
                                <schema>http://localhost:9938/restconf/tailf/modules/Cisco-IOS-XE-native/2021-08-10</schema>
                                <namespace>http://cisco.com/ns/yang/Cisco-IOS-XE-native</namespace>
                                <deviation>
                                        <name>Cisco-IOS-XE-cts-routing-deviation</name>
                                        <revision>2020-08-26</revision>
                                </deviation>
                                <deviation>
                                        <name>Cisco-IOS-XE-dialer-deviation</name>
                                        <revision>2020-07-01</revision>
                                </deviation>
                                <deviation>
                                        <name>Cisco-IOS-XE-ethernet-deviation</name>
                                        <revision>2021-07-01</revision>
                                </deviation>
                                <deviation>
                                        <name>Cisco-IOS-XE-lisp-deviation</name>
                                        <revision>2021-07-10</revision>
                                </deviation>
                                <deviation>
                                        <name>Cisco-IOS-XE-nd-deviation</name>
                                        <revision>2020-06-09</revision>
                                </deviation>
                                <deviation>
                                        <name>Cisco-IOS-XE-ospfv3-deviation</name>
                                        <revision>2021-01-08</revision>
                                </deviation>
                                <deviation>
                                        <name>Cisco-IOS-XE-policy-deviation</name>
                                        <revision>2021-07-01</revision>
                                </deviation>
                                <deviation>
                                        <name>Cisco-IOS-XE-port-channel-deviation</name>
                                        <revision>2020-11-01</revision>
                                </deviation>
                                <deviation>
                                        <name>Cisco-IOS-XE-power-deviation</name>
                                        <revision>2019-04-17</revision>
                                </deviation>
                                <deviation>
                                        <name>Cisco-IOS-XE-sanet-deviation</name>
                                        <revision>2020-07-01</revision>
                                </deviation>
                                <conformance-type>implement</conformance-type>
                                <submodule>
                                        <name>Cisco-IOS-XE-interfaces</name>
                                        <revision>2021-07-10</revision>
                                        <schema>http://localhost:9938/restconf/tailf/modules/Cisco-IOS-XE-interfaces/2021-07-10</schema>
                                </submodule>
                                <submodule>
                                        <name>Cisco-IOS-XE-ip</name>
                                        <revision>2021-07-10</revision>
                                        <schema>http://localhost:9938/restconf/tailf/modules/Cisco-IOS-XE-ip/2021-07-10</schema>
                                </submodule>
                                <submodule>
                                        <name>Cisco-IOS-XE-ipv6</name>
                                        <revision>2021-07-01</revision>
                                        <schema>http://localhost:9938/restconf/tailf/modules/Cisco-IOS-XE-ipv6/2021-07-01</schema>
                                </submodule>
                                <submodule>
                                        <name>Cisco-IOS-XE-license</name>
                                        <revision>2020-11-01</revision>
                                        <schema>http://localhost:9938/restconf/tailf/modules/Cisco-IOS-XE-license/2020-11-01</schema>
                                </submodule>
                                <submodule>
                                        <name>Cisco-IOS-XE-line</name>
                                        <revision>2021-07-01</revision>
                                        <schema>http://localhost:9938/restconf/tailf/modules/Cisco-IOS-XE-line/2021-07-01</schema>
                                </submodule>
                                <submodule>
                                        <name>Cisco-IOS-XE-logging</name>
                                        <revision>2020-11-01</revision>
                                        <schema>http://localhost:9938/restconf/tailf/modules/Cisco-IOS-XE-logging/2020-11-01</schema>
                                </submodule>
                                <submodule>
                                        <name>Cisco-IOS-XE-parser</name>
                                        <revision>2019-07-01</revision>
                                        <schema>http://localhost:9938/restconf/tailf/modules/Cisco-IOS-XE-parser/2019-07-01</schema>
                                </submodule>
                        </module>

As you can see the revisions are fine, but when I look in the tmp folder, they are all the native parent modules revision..

at least the ones that are submodules to the native model image

I cant make this add up :-)

Esben

hyberdk commented 10 months ago

Hi @ygorelik,

Im not sure you get notified when the issue is closed, so Im trying to mention you directly. In my previous post, you can see that models from my router seems fine. I have been looking further into this, although I do not really do c++ code, I have narrowed it down to this section:

        extern "C" char* get_module_callback(const char* module_name, const char* module_rev, const char *submod_name, const char *sub_rev,
                                       void* user_data, LYS_INFORMAT* format, void (**free_module_data)(void *model_data))
        {
            YLOG_DEBUG("Getting module '{}' submodule '{}'", module_name, (submod_name?submod_name:"none"));
            *free_module_data = c_free_data;

            if(user_data != nullptr){
                ModelProvider::Format m_format = ModelProvider::Format::YANG;
                *format = LYS_IN_YANG;
                auto repo = reinterpret_cast<const RepositoryPtr*>(user_data);

                //first check our directory for a file of the form <module-module_name>@<module_rev-date>.yang
                YLOG_DEBUG("Looking for file in folder: {}", repo->path);
                std::string yang_file_path{repo->path};
                std::string yang_file_path_no_revision{repo->path};
                yang_file_path += '/';
                yang_file_path += (submod_name?submod_name:module_name);
                yang_file_path_no_revision += yang_file_path;

                if(submod_name && sub_rev){
                    yang_file_path += "@";
                    yang_file_path += sub_rev;
                }
                else if(module_name && module_rev){
                    yang_file_path += "@";
                    yang_file_path += module_rev;
                }
                yang_file_path += ".yang";
                YLOG_DEBUG("Opening file '{}'", yang_file_path);

                if (file_exists(yang_file_path) || file_exists(yang_file_path_no_revision)) {
                    if (file_exists(yang_file_path))
                        YLOG_DEBUG("Path found with revision: {}", yang_file_path);
                    else
                        YLOG_DEBUG("Path found without revision: {}", yang_file_path_no_revision);

                    //open the file read the data and return it
                    std::string model_data {""};
                    std::ifstream yang_file {yang_file_path};
                    if(yang_file.is_open()) {
                        std::string line;
                        while(std::getline(yang_file, line)){
                            model_data+=line;
                            model_data+='\n';
                        }

                        yang_file.close();

                        return get_enlarged_data(model_data, yang_file_path);
                    } else {
                        YLOG_ERROR("Cannot open file '{}'", yang_file_path);
                        throw(YIllegalStateError("Cannot open file " + yang_file_path));
                    }
                }
                else {
                    YLOG_DEBUG("File '{}' is not found in repository", yang_file_path);
                }

                for(auto model_provider : repo->get_model_providers()) {
                    std::string model_data{};
                    if(submod_name)
                    {
                        YLOG_DEBUG("Getting submodule '{}' using get-schema", submod_name);
                        model_data = model_provider->get_model(submod_name, sub_rev != nullptr ? sub_rev : "", m_format);
                    }
                    else
                    {
                        YLOG_DEBUG("Getting module '{}' using get-schema", module_name);
                        model_data = model_provider->get_model(module_name, module_rev != nullptr ? module_rev : "", m_format);
                    }
                    if(!model_data.empty()){
                        sink_to_file(yang_file_path, model_data);
                        return get_enlarged_data(model_data, yang_file_path);
                    }
                }
            }
            YLOG_ERROR("Cannot find model with module name '{}'", module_name);
            //throw(YServiceProviderError("Cannot find model with module name: " + std::string(module_name)));
            return {};
        }
    }
}

and I do agree that it seems like the code is supposed to download the revision of the submodule, but that is not what I am seeing..

here is a snip from the log:

2023-11-13 21:49:39,333 - ydk - DEBUG - Getting module 'Cisco-IOS-XE-native' submodule 'Cisco-IOS-XE-ip'
2023-11-13 21:49:39,334 - ydk - DEBUG - Looking for file in folder: /home/eslau/.ydk/10.12.35.56
2023-11-13 21:49:39,334 - ydk - DEBUG - Opening file '/home/eslau/.ydk/10.12.35.56/Cisco-IOS-XE-ip@2021-08-10.yang'
2023-11-13 21:49:39,335 - ydk - DEBUG - File '/home/eslau/.ydk/10.12.35.56/Cisco-IOS-XE-ip@2021-08-10.yang' is not found in repository
2023-11-13 21:49:39,335 - ydk - DEBUG - Getting submodule 'Cisco-IOS-XE-ip' using get-schema
2023-11-13 21:49:39,336 - ydk - DEBUG - NetconfSSHClient: NC session status: 1
2023-11-13 21:49:39,336 - ydk - DEBUG - Trace: Missing message-id in rpc.
2023-11-13 21:49:39,336 - ydk - DEBUG - Netconf SSH Client: sending RPC
2023-11-13 21:49:39,336 - ydk - DEBUG - Trace: Writing message (session 25): 
<?xml version="1.0"?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="2">
  <get-schema xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring">
    <identifier>Cisco-IOS-XE-ip</identifier>
    <format>yang</format>
  </get-schema>
</rpc>

2023-11-13 21:49:39,337 - ydk - DEBUG - NetconfSSHClient: NC session status: 1
2023-11-13 21:49:39,337 - ydk - DEBUG - Netconf SSH Client: receiving reply RPC
2023-11-13 21:49:39,425 - ydk - DEBUG - Trace: Received message (session 25): 
<?xml version="1.0" encoding="UTF-8"?>
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="2"><data xmlns='urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring'><![CDATA[submodule Cisco-IOS-XE-ip {
  belongs-to Cisco-IOS-XE-native {
    prefix ios;
  }

  import cisco-semver {
    prefix cisco-semver;
  }
  import ietf-inet-types {
    prefix inet;
  }
  import Cisco-IOS-XE-types {
    prefix ios-types;
  }
  import Cisco-IOS-XE-interface-common {
    prefix ios-ifc;
  }
  import Cisco-IOS-XE-features {
    prefix ios-features;
  }
  include Cisco-IOS-XE-interfaces;

  organization
    "Cisco Systems, Inc.";
  contact
    "Cisco Systems, Inc.
     Customer Service

     Postal: 170 W Tasman Drive
     San Jose, CA 95134

     Tel: +1 1800 553-NETS

     E-mail: cs-yang@cisco.com";
  description
    "Cisco XE Native IP Yang Model.
     Copyright (c) 2016-2021 by Cisco Systems, Inc.
     All rights reserved.";

  revision 2021-07-10 {
    description
      "- Added container vxlan for Tenant Routed Multicast Data MDT support";
    cisco-semver:module-version "7.0.17060(m)";
  }
  revision 2021-07-01 {
    description
      "- Added ip ssh bulk-mode support
       - Fixed configuration roll-back ordering for ip prefix-lists in controller mode
       - Added new allow-evpn leaf
       - Added support to include interface module for fixing yang errors
       - Changed 'ip ssh source-interface' from leaf to container
       - Fixed the must constraints of mdt model";
    cisco-semver:module-version "7.0.0";
  }

You can see that it is trying to open file Opening file '/home/eslau/.ydk/10.12.35.56/Cisco-IOS-XE-ip@2021-08-10.yang', but if you look further down in the actual yang model, the first revision is revision 2021-07-10 and not revision 2021-08-10

Also if it tried to download a revision 2021-08-10 from my router it would fail (I tried manually with ncclient)..

Further up in the log file, I do also see this section:

        <submodule>
          <name>Cisco-IOS-XE-ip</name>
          <revision>2021-07-10</revision>
          <schema>http://localhost:9938/restconf/tailf/modules/Cisco-IOS-XE-ip/2021-07-10</schema>
        </submodule>

which is also the correct version... I have attached the full log output here: log.txt

I really dont see how the YDK comes to the conclusion that the correct revision is 2021-08-10.. I also tried to place the Cisco-IOS-XE-ip.yang file in the repo, but it does not find/use that file, which is also a little strange as the c++ code seems to suggest that it would..

Please can you have another look at it :-)

Thanks a lot!

Esben

hyberdk commented 10 months ago

Hi @ygorelik,

So I did some more testing..

I modified the code to look like this (added more logging):

        extern "C" char* get_module_callback(const char* module_name, const char* module_rev, const char *submod_name, const char *sub_rev,
                                       void* user_data, LYS_INFORMAT* format, void (**free_module_data)(void *model_data))
        {
            YLOG_DEBUG("submod_name '{}' sub_rev '{}'", (submod_name?submod_name:"unknown"), (sub_rev?sub_rev:"unknown"));
            YLOG_DEBUG("Getting module '{}' submodule '{}'", module_name, (submod_name?submod_name:"none"));
            *free_module_data = c_free_data;

            if(user_data != nullptr){
                ModelProvider::Format m_format = ModelProvider::Format::YANG;
                *format = LYS_IN_YANG;
                auto repo = reinterpret_cast<const RepositoryPtr*>(user_data);

                //first check our directory for a file of the form <module-module_name>@<module_rev-date>.yang
                YLOG_DEBUG("Looking for file in folder: {}", repo->path);
                std::string yang_file_path{repo->path};
                std::string yang_file_path_no_revision{repo->path};
                yang_file_path += '/';
                yang_file_path += (submod_name?submod_name:module_name);
                yang_file_path_no_revision += yang_file_path;

                if(submod_name && sub_rev){
                    YLOG_DEBUG("found submodule with revision: submodule '{}' and revision '{}'", submod_name, sub_rev);
                    yang_file_path += "@";
                    yang_file_path += sub_rev;
                }
                else if(module_name && module_rev){
                    YLOG_DEBUG("found module with revision: module '{}' and revision '{}'", module_name, module_rev);
                    yang_file_path += "@";
                    yang_file_path += module_rev;
                }
                yang_file_path += ".yang";
                YLOG_DEBUG("Opening file '{}'", yang_file_path);

                if (file_exists(yang_file_path) || file_exists(yang_file_path_no_revision)) {
                    if (file_exists(yang_file_path))

here is the log output..

2023-11-13 23:51:58,354 - ydk - DEBUG - submod_name 'Cisco-IOS-XE-ip' sub_rev 'unknown'
2023-11-13 23:51:58,354 - ydk - DEBUG - Getting module 'Cisco-IOS-XE-native' submodule 'Cisco-IOS-XE-ip'
2023-11-13 23:51:58,354 - ydk - DEBUG - Looking for file in folder: /home/eslau/.ydk/10.12.35.56
2023-11-13 23:51:58,355 - ydk - DEBUG - found module with revision: module 'Cisco-IOS-XE-native' and revision '2021-08-10'
2023-11-13 23:51:58,355 - ydk - DEBUG - Opening file '/home/eslau/.ydk/10.12.35.56/Cisco-IOS-XE-ip@2021-08-10.yang'
2023-11-13 23:51:58,355 - ydk - DEBUG - File '/home/eslau/.ydk/10.12.35.56/Cisco-IOS-XE-ip@2021-08-10.yang' is not found in repository
2023-11-13 23:51:58,355 - ydk - DEBUG - Getting submodule 'Cisco-IOS-XE-ip' using get-schema
2023-11-13 23:51:58,355 - ydk - DEBUG - NetconfSSHClient: NC session status: 1
2023-11-13 23:51:58,355 - ydk - DEBUG - Trace: Missing message-id in rpc.
2023-11-13 23:51:58,355 - ydk - DEBUG - Netconf SSH Client: sending RPC
2023-11-13 23:51:58,356 - ydk - DEBUG - Trace: Writing message (session 38):

So as you can see, when the submodule are to be downloaded, no sub_rev is sent to the function. This means that if(submod_name && sub_rev) is always false.. and it goes to the else if statement..

and since that is true (there is both a module and submodule) it actually grabs the revision from the "main" module and puts that into the submodule..

Its getting late here, but I guess the quick fix would be to do something like:

                if(submod_name && sub_rev){
                    YLOG_DEBUG("found submodule with revision: submodule '{}' and revision '{}'", submod_name, sub_rev);
                    yang_file_path += "@";
                    yang_file_path += sub_rev;
                }
                else if(submod_name){
                    YLOG_DEBUG("found submodule without revision: submodule '{}'", submod_name);
                }
                else if(module_name && module_rev){
                    YLOG_DEBUG("found module with revision: module '{}' and revision '{}'", module_name, module_rev);
                    yang_file_path += "@";
                    yang_file_path += module_rev;
                }

but I do not know if that will break something else.. I will see if I can find time tomorrow for more test..

Esben

ygorelik commented 10 months ago

Hi Esben You code change might be OK, but there are a bit more (check the commit). Could you please try to run updated version of YDK-0.8.6.6 from my fork master branch? Let me know the results. Yan

hyberdk commented 10 months ago

Hi Yan,

thanks for looking into this I've been looking though the code, I think I follow your process here..

looking though the code, I think it makes sense. I will for sure check it out and give feedback!

Just a sorta off topic comment, if you are going to release a new maintenance version, it might make sense to bump pyang to version 2.6.0 as there are some fixes in there regarding scoping of submodules and augmenting models with when statements. see mbj4668/pyang@dc9755382194d59484f439b557b6110b138a3bd0 it was essential for me to make the Cisco-IOS-XE-*.yang models pass linting and generate the models..

Also it seems that the latest version of libyang (CESNET/libyang@7e5ea21030fe6632b6faad30c0de8d9669503773) fixes the problem #1083, #1093. I am not 100% sure yet, but it seems like at least yang v1.1 allows enum without quotes (see https://datatracker.ietf.org/doc/html/rfc7950#section-9.6.5). Anyways, I need to do more testing, but it seems like libyang v2.1.128 fixes my problem with the enum named +

As I said before I am not a c++ programmer, so I would not know how to set the depending version of libyang in the c code. :-)

Anyways, thanks for looking into this for me.. I seem to be on the verge of a breakthrough getting YDK-GEN to work for me and my Cisco XE 17.x.x versions..

hyberdk commented 10 months ago

Okay, so I did some testing..

darn, I seem to be hitting the "enum +" bug again, I was sure I tested it before and it worked.. well I guess more testing is needed.. (could ydk-gen have installed an old version of libyang?)

Anywho.. regarding the case of filename.. yup, it seems to do the trick.. now it downloads the submodules without any revision..

2023-11-15 23:28:36,501 - ydk - DEBUG - Getting module 'Cisco-IOS-XE-native' submodule 'Cisco-IOS-XE-ip' with revision none
2023-11-15 23:28:36,501 - ydk - DEBUG - Looking for file in folder: /home/eslau/.ydk/10.12.35.56
2023-11-15 23:28:36,501 - ydk - DEBUG - Opening file '/home/eslau/.ydk/10.12.35.56/Cisco-IOS-XE-ip.yang'
2023-11-15 23:28:36,501 - ydk - DEBUG - File '/home/eslau/.ydk/10.12.35.56/Cisco-IOS-XE-ip.yang' is not found in repository
2023-11-15 23:28:36,501 - ydk - DEBUG - Getting submodule 'Cisco-IOS-XE-ip' using get-schema
2023-11-15 23:28:36,501 - ydk - DEBUG - NetconfSSHClient: NC session status: 1
2023-11-15 23:28:36,501 - ydk - DEBUG - Trace: Missing message-id in rpc.
2023-11-15 23:28:36,501 - ydk - DEBUG - Netconf SSH Client: sending RPC
2023-11-15 23:28:36,501 - ydk - DEBUG - Trace: Writing message (session 32): 
<?xml version="1.0"?>
<rpc xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="11">
  <get-schema xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring">
    <identifier>Cisco-IOS-XE-ip</identifier>
    <format>yang</format>
  </get-schema>
</rpc>

2023-11-15 23:28:36,501 - ydk - DEBUG - NetconfSSHClient: NC session status: 1
2023-11-15 23:28:36,501 - ydk - DEBUG - Netconf SSH Client: receiving reply RPC
2023-11-15 23:28:36,579 - ydk - DEBUG - Trace: Received message (session 32): 
<?xml version="1.0" encoding="UTF-8"?>
<rpc-reply xmlns="urn:ietf:params:xml:ns:netconf:base:1.0" message-id="11"><data xmlns='urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring'><![CDATA[submodule Cisco-IOS-XE-ip {
  belongs-to Cisco-IOS-XE-native {
    prefix ios;
  }

  import cisco-semver {
    prefix cisco-semver;
  }
  import ietf-inet-types {
    prefix inet;
  }

and the 2nd time around (even if I modified the file)

2023-11-15 23:31:21,060 - ydk - DEBUG - Getting module 'Cisco-IOS-XE-native' submodule 'Cisco-IOS-XE-ip' with revision none
2023-11-15 23:31:21,060 - ydk - DEBUG - Looking for file in folder: /home/eslau/.ydk/10.12.35.56
2023-11-15 23:31:21,060 - ydk - DEBUG - Opening file '/home/eslau/.ydk/10.12.35.56/Cisco-IOS-XE-ip.yang'
2023-11-15 23:31:21,060 - ydk - DEBUG - Path found with revision: /home/eslau/.ydk/10.12.35.56/Cisco-IOS-XE-ip.yang

This is expected behavior, nice!

here are the full log files: no_local_files.txt with_local_files.txt

hyberdk commented 10 months ago

okay, it is the

        leaf offset {
          status obsolete;
          type enumeration {
            enum "+";
            enum "-";
          }

in the Cisco-IOS-XE-native.yang module that is causing the problems. It is a little strange, if I generate the python package with the module like above (with the quotes) it works fine.. but if I leave it like its in the models from Cisco it fails.

Funny thing is though.. it is only during the request and its using the model in the python package.. once data is returned from the router it parses the data just fine even though the downloaded model is without the quotes on "+" and "-"

Im doing this:

    result = netconf.get_config(provider, Datastore.running, read_filter=native)  # read running config
    codec_provider = CodecServiceProvider(type='json')
    codec = CodecService()
    json_data = codec.encode(codec_provider, result, subtree=True)
    print(json.dumps(json.loads(json_data), indent=4))
2023-11-16 00:13:35,178 - ydk.types.Entity - DEBUG - Leaf data name: "ipv4-host", value: "9.9.9.9", yfilter: "YFilter.not_set", is_set: "True"
2023-11-16 00:13:35,178 - ydk - DEBUG - JsonCodec: Children count for 'host[ipv4-host='9.9.9.9']': 0
{
    "Cisco-IOS-XE-native:native": {
        "ip": {
            "access-list": {
                "Cisco-IOS-XE-acl:extended": [
                    {
                        "access-list-seq-rule": [
                            {
                                "ace-rule": {
                                    "action": "permit",
                                    "dst-any": null,
                                    "ipv4-address": "185.32.192.0",
                                    "mask": "0.0.0.255",
                                    "protocol": "ip"
                                },
                                "sequence": 10
                            },
                            {
                                "ace-rule": {
                                    "action": "permit",
                                    "any": null,
                                    "dest-ipv4-address": "185.32.192.0",
                                    "dest-mask": "0.0.0.255",
                                    "protocol": "ip"
                                },
                                "sequence": 20
                            },
                            {
                                "ace-rule": {
<snip>

That is really strange!

anyways, its late I better hit the sack.. again thanks for your help!

Esben

ygorelik commented 10 months ago

Hi Esben

You are right, the YDK uses older version of YDK. The upgrade would require significant makeover of the C++ code. This possibly a task for future release. In the meanwhile I will upgrade pyang to the latest version.

For the last comment. I do not understand what is the issue. And the XML load is not visible. If true, please open new issue with all the supporting materials.

Thanks for the verification. Yan

ygorelik commented 10 months ago

The resolution of this bug is in my private fork, see details in commit 0f63489.