robshakir / pyangbind

A plugin for pyang that creates Python bindings for a YANG model.
Other
198 stars 120 forks source link

Presence container support #336

Closed ianbarrere closed 4 months ago

ianbarrere commented 5 months ago

I came upon this thread from some years ago which indicates that presence containers are now supported in pyangbind, but I can't get my binding to recognize them. The model I'm working with is Cisco-IOS-XR-um-interface-cfg.yang from here. There is a presence container "shutdown" (why they chose to do it that way is beyond me...) which is not showing up as an attribute when I create my binding. Everything else seems to be recognized.

Is there something I need to do to enable presence containers to be created in the binding when I create it?

Relevant versions:

pyang 2.6.0 pyangbind 0.8.4.post1

Some output:

>>> from network_tools.bindings.xr_um_interfaces import interfaces
>>> interfaces = interfaces.interfaces()
>>> my_interface = interfaces.interface.add("TenGigE0/0/0/1")
>>> my_interface.get()
{'interface-name': 'TenGigE0/0/0/1', 'sub-interface-type': {}, 'ipv4': {'addresses': {'address': {'address': '', 'netmask': '', 'route-tag': 0, 'algorithm': 0}, 'secondaries': {'secondary': OrderedDict()}, 'unnumbered': ''}}, 'ipv6': {'addresses': {'ipv6-address': OrderedDict(), 'link-local-address': {'address': '', 'zone': '', 'route-tag': 0}, 'eui64-addresses': {'eui64-address': OrderedDict()}}}, 'dampening': {'decay-half-life': {'value': 0, 'reuse-threshold': {'value': 0, 'suppress-threshold': {'value': 0, 'max-suppress-time': {'value': 0, 'restart-penalty': {'value': 0}}}}}}, 'encapsulation': {'frame-relay': {}}, 'mtu': 0, 'logging': {'events': {}}, 'bandwidth': 0, 'description': '', 'vrf': '', 'address-family': {'ipv4': {'multicast': {'topologies': {'topology': OrderedDict()}}}, 'ipv6': {'multicast': {'topologies': {'topology': OrderedDict()}}}}}
>>> my_interface.shutdown
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'YANGBaseClass' object has no attribute 'shutdown'
>>>

Here is the command I used to generate the binding:

    pyang --plugindir ${PYBINDPLUGIN} --split-class-dir network_tools/bindings/xr_um_interfaces \
      --use-xpathhelper \
      -p /tmp/yang/standard/ietf/RFC \
      /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-types.yang \
      /tmp/yang/vendor/cisco/xr/7921/cisco-semver.yang \
      /tmp/yang/vendor/cisco/xr/7921/tailf-common.yang \
      /tmp/yang/vendor/cisco/xr/7921/tailf-meta-extensions.yang \
      /tmp/yang/vendor/cisco/xr/7921/tailf-cli-extensions.yang \
      /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-um-if-vrf-cfg.yang \
      /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-um-if-ip-address-cfg.yang \
      -f pybind \
      /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-um-interface-cfg.yang
ianbarrere commented 5 months ago

OK, I see the issue. It appears that when there is a presence container with nothing in it then it doesn't show up in the binding:

    container main {
        container presence-container {
            presence "This is a test container";
        }
        leaf foo {
            type string;
        }
    }

Results in this:

>>> from yang_test import test
>>> test = test()
>>> test.get()
{'main': {'foo': ''}}
>>>

But if I add a leaf inside the presence container then it shows up:

    container main {
        container presence-container {
            presence "This is a test container";
            leaf bar {
                type string;
            }
        }
        leaf foo {
            type string;
        }
    }
>>> from yang_test import test
>>> test = test()
>>> test.get()
{'main': {'presence-container': {'bar': ''}, 'foo': ''}}
>>>

I personally think it's kind of silly that the IOS-XR unified models, for example, go this route for signaling whether an interface is shutdown or not, but I of course have no power over that decision. Is it possible to extend support in pyangbind for presence containers that have no child elements?

JoseIgnacioTamayo commented 5 months ago

Give the Unit Tests for Presence a look

https://github.com/robshakir/pyangbind/blob/master/tests/presence/presence.yang

From there I learned that calling pyang with '--presence' might help you:

pyang --plugindir $PLUGIN -f pybind --presence tests/presence/presence.yang -o bind.py

import bind p = bind.presence() p.get() {'empty-container': {}, 'parent': {'child': {}}, 'np-container': {'s': ''}, 'pp': {'s': ''}}

Hope it helps

ianbarrere commented 5 months ago

I stumbled upon that flag also, but could not get it working before. Comparing my steps to yours, it appears the issue is when using the "--split-class-dir" flag to generate the binding, I get this when trying to instantiate the object:

(venv) ianb@mac tools % pyang --plugindir ${PYBINDPLUGIN} --split-class-dir bind -f pybind presence.yang --presence
(venv) ianb@mac tools % python3
Python 3.11.5 (main, Aug 24 2023, 15:09:45) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bind
>>> p = bind.presence()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ianb/tools/bind/__init__.py", line 58, in __init__
    self.__empty_container = YANGDynClass(base=empty_container.empty_container, is_container='container', presence=True, yang_name="empty-container", parent=self, path_helper=self._path_helper, extmethods=self._extmethods, register_paths=True, extensions=None, namespace='http://rob.sh/yang/test/presence', defining_module='presence', yang_type='container', is_config=True)
                                               ^^^^^^^^^^^^^^^
NameError: name 'empty_container' is not defined. Did you mean: 'np_container'?
>>>

But you're right, if I use the -o flag instead it seems to work:

(venv) ianb@mac tools % pyang --plugindir ${PYBINDPLUGIN} -o bind.py -f pybind presence.yang --presence
(venv) ianb@mac tools % python3
Python 3.11.5 (main, Aug 24 2023, 15:09:45) [Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import bind
>>> p = bind.presence()
>>> p.get()
{'empty-container': {}, 'parent': {'child': {}}, 'np-container': {'s': ''}, 'pp': {'s': ''}}
>>>

However, when trying to do the -o approach with the actual IOS-XR UM model I'm working with like so:

    pyang --plugindir ${PYBINDPLUGIN} -o dv_network_tools/bindings/xr_um_interfaces.py \
      --use-xpathhelper \
      -p /tmp/yang/standard/ietf/RFC \
      /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-types.yang \
      /tmp/yang/vendor/cisco/xr/7921/cisco-semver.yang \
      /tmp/yang/vendor/cisco/xr/7921/tailf-common.yang \
      /tmp/yang/vendor/cisco/xr/7921/tailf-meta-extensions.yang \
      /tmp/yang/vendor/cisco/xr/7921/tailf-cli-extensions.yang \
      /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-um-if-vrf-cfg.yang \
      /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-um-if-ip-address-cfg.yang \
      /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-um-l2-ethernet-cfg.yang \
      /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-um-if-l2transport-cfg.yang \
      -f pybind \
      /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-um-interface-cfg.yang --presence

Then it doesn't seem to pick up the VRF import, as the "vrf" leaf that's augmented in from Cisco-IOS-XR-um-if-vrf-cfg.yang is no longer in the binding. Strange. I may just resort to adding a dummy leaf in the empty presence container for now.

ianbarrere commented 5 months ago

Alright, so adding a dummy leaf didn't do the trick. But what's even more troublesome is that I can't seem to remove the presence container after it's been created either. If I build the binding as I described above with the --presence flag:

>>> from network_tools.bindings.xr_um_interfaces import Cisco_IOS_XR_um_interface_cfg
>>> bind = Cisco_IOS_XR_um_interface_cfg().interfaces.interface
>>> my_int = bind.add("TenGigE0/0/0/1")
>>> import pyangbind.lib.pybindJSON as pybindJSON
>>> pybindJSON.dumps(my_int, mode="ietf", indent=None)
'{"Cisco-IOS-XR-um-interface-cfg:interface-name": "TenGigE0/0/0/1"}'
>>> my_int.shutdown._set_present()
>>> pybindJSON.dumps(my_int, mode="ietf", indent=None)
'{"Cisco-IOS-XR-um-interface-cfg:interface-name": "TenGigE0/0/0/1", "Cisco-IOS-XR-um-interface-cfg:shutdown": {}}'
>>> my_int.shutdown._set_present(False)
>>> pybindJSON.dumps(my_int, mode="ietf", indent=None)
'{"Cisco-IOS-XR-um-interface-cfg:interface-name": "TenGigE0/0/0/1", "Cisco-IOS-XR-um-interface-cfg:shutdown": {}}'
>>>

But hmm, this seems to only be an issue when using mode="ietf", removing that results in correct behavior:

>>> my_int.shutdown._set_present(False)
>>> pybindJSON.dumps(my_int, indent=None)
'{"interface-name": "TenGigE0/0/0/1"}'
>>> my_int.shutdown._set_present(True)
>>> pybindJSON.dumps(my_int, indent=None)
'{"interface-name": "TenGigE0/0/0/1", "shutdown": {}}'
>>>
JoseIgnacioTamayo commented 5 months ago

I was trying a bit and I guess we have 2 issues:

For the Presence elements to show up in the binding, it needs to be created with --presence. But this only works with a single file output. When output is a dir of classes, several imports are missing that make reference to presence containers. This is a bug and I will try to see how to fix it.

Regardless, once the binding is working and in use, the presence of the container can be set with ._set_present(). This changes a property in the object which does not seem to be read by the IETF Json and Xml Serializers (somehow the Json serializer does).

A workaround is calling the parent to remove the not-present container with '._unset_shutdown()'

Here I wonder what is the correct implementation of setting a container, possibly not-empty, to NotPresent... it seems it is right to remove the container (like calling the parent's unset()

ianbarrere commented 5 months ago

Cool, thank you for the details. Indeed _unset_shutdown() seems to work fine, so I will use that for now. As for the right approach, I agree, this is how it's done in Cisco NSO as well and is rather intuitive.

The one issue I am still have is this missing "vrf" leaf when creating the binding with the --presence flag. If I create it with this command:

pyang --plugindir ${PYBINDPLUGIN} -o tools/bindings/xr_um_interfaces.py \
          --use-xpathhelper \
          -p /tmp/yang/standard/ietf/RFC \
          /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-types.yang \
          /tmp/yang/vendor/cisco/xr/7921/cisco-semver.yang \
          /tmp/yang/vendor/cisco/xr/7921/tailf-common.yang \
          /tmp/yang/vendor/cisco/xr/7921/tailf-meta-extensions.yang \
          /tmp/yang/vendor/cisco/xr/7921/tailf-cli-extensions.yang \
          /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-um-if-vrf-cfg.yang \
          /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-um-if-ip-address-cfg.yang \
          /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-um-l2-ethernet-cfg.yang \
          /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-um-if-l2transport-cfg.yang \
          -f pybind \
          /tmp/yang/vendor/cisco/xr/7921/Cisco-IOS-XR-um-interface-cfg.yang

Then the "vrf" leaf is there as expected:

>>> from tools.bindings.xr_um_interfaces import Cisco_IOS_XR_um_interface_cfg
>>> ints = Cisco_IOS_XR_um_interface_cfg().interfaces.interface
>>> my = ints.add("TenGigE0/0/0/1")
>>> my.vrf
''
>>> my.vrf = "TEST"
>>> my.vrf
'TEST'
>>>

But then if I add the --presence flag to the binding creation command:

>>> from tools.bindings.xr_um_interfaces import Cisco_IOS_XR_um_interface_cfg
>>> ints = Cisco_IOS_XR_um_interface_cfg().interfaces.interface
>>> my = ints.add("TenGigE0/0/0/1")
>>> my.vrf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'YANGBaseClass' object has no attribute 'vrf'
>>>

Any idea what might be causing this? I will continue to look into it on my side as well.

ianbarrere commented 5 months ago

OK, so it seems like the binding is actually missing everything that's not a presence container when the --presence flag is given. I modified the Cisco-IOS-XR-um-interface-cfg.yang file and added another presence container called test at the main level, and then added a leaf underneath that. I then created the binding with the --presence flag, and when navigating the structure in python I see the test container, but not the leaf within it. Likewise all the "basic", non augmented structure like description, mtu, etc, are also missing.

JoseIgnacioTamayo commented 5 months ago

I tried some fixes in https://github.com/robshakir/pyangbind/tree/fix_presence, can you try this branch?

I think you could create a virtualenv without pyangbind installed from package, clone the pyangbind branch and point pyang to the cloned git with --plugindir.

With the UnitTests, I checked that the --presence option also works with --split-class-dir, and that the serialization also works after unsetting a presence.

ianbarrere commented 5 months ago

Nice, it looks good from my test. I was able to build the binding with split directories and toggle presence on and off with it reflecting properly in the IETF serialization.

The issue of the missing vrf leaf is still there though, and now it's even stranger. Before, only presence containers were visible, now it seems like everything except leafs are visible. So, for example, I can see the structure up to (using the IOS-XR UM model) /interfaces/interface/ipv4/addresses/address, but there is no address leaf and no netmask leaf underneath that. Likewise, the non-augmented leafs description and mtu are also not visible directly under the interface.

Unfortunately I can't seem to recreate it with a simple test case though, using the YANG from the pyanbind unit tests I can see all the leafs in there.

Thank you!

ianbarrere commented 5 months ago

OK, after a bit more digging it seems that the --presence flag doesn't play well with groupings. So if I modify the unit test YANG slightly:

module presence {
    yang-version "1";
    namespace "http://rob.sh/yang/test/presence";
    prefix "foo";
    organization "BugReports Inc";
    contact "A bug reporter";

    description
        "A test module";
    revision 2014-01-01 {
        description "april-fools";
        reference "fooled-you";
    }

    grouping my-grouping {
        leaf leaf-in-grouping {
            type string;
        }
    }

    container empty-container {
        presence "something implied implicity";
    }

    container parent {
        container child {
            presence "something else implicit";
        }
    }

    container np-container {
        description
            "np container";

        leaf s {
            type string;
        }
        uses my-grouping;
    }

    container pp {
        presence "implicit";

        leaf s {
            type string;
        }
        uses my-grouping;
        leaf non-grouping-leaf {
            type string;
        }
    }
}

And then generate the binding with pyang --plugindir ${PYBINDPLUGIN} --split-class-dir presence presence.yang --presence -f pybind, I get this structure:

>>> from presence import presence
>>> p = presence()
>>> p.get()
{'empty-container': {}, 'parent': {'child': {}}, 'np-container': {'s': ''}, 'pp': {'s': '', 'non-grouping-leaf': ''}}
>>>
JoseIgnacioTamayo commented 5 months ago

Alright! We are moving forward, little by little.

Thanks for testing and bringing feedback. I will take the UnitCase and see how I can fix it.

ianbarrere commented 5 months ago

No worries, I'm happy to help! Thank you for the fixes so far. Let me know what you find with the grouping issue, I will dig into the code a bit too, wouldn't mind familiarizing myself with it more.

JoseIgnacioTamayo commented 5 months ago

Hi again,

Could you please give the latest code in the branch a try?

If it all goes fine, I can make a PullRequest for it.

Thanks for your help testing it.

ianbarrere commented 5 months ago

Yes, looks good now! I just tested with the same IOS-XR model and I can see everything and also toggle the presence containers properly. Thank you!

JoseIgnacioTamayo commented 4 months ago

Fixed in https://github.com/robshakir/pyangbind/pull/339