failure in regex check due to python 2 to 3 conversion causes pybindIETFXMLDecoder to fail to decode a non string (restricted e.g. inet:ip-address) type

[igrush]

How I got here I am attempting to decode from XML into a pyangbind object. Here is a small code sample.

#!/usr/bin/env python

import pyangbind.lib.serialise as pyang_serial
import pybind_eg

XML_SAMPLE = '<?xml version="1.0"?><data xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
<ospf xmlns="urn:com:cxx:abc:yang:ospf">
  <instances>
    <instance>
      <router-id>1.2.3.4</router-id>
      <network-areas>
        <network-area>
          <area-id>24.0.0.1</area-id>
          <network-address>192.5.5.0/24</network-address>
        </network-area>
      </network-areas>
    </instance>
  </instances>
</ospf>
</data>'

# decode a block of XML into an etree object
config = pyang_serial.pybindIETFXMLDecoder.decode(XML_SAMPLE, pybind_eg, 'ospf')

pybind_eg is the directory containing the bindings. They were generated with --split-class-dir and --use-xpathhelper, so the binding code is mainly in the init.py files. I have attached pybind_eg.zip with the bindings.

This is the yang that describes router-id:

        list instance {
           description
          "Global configuration for the OSPF router instance";
       key "router-id" ;

           leaf router-id {
              type inet:ip-address;
                 description
                    "Local router id number of the ospf routing instance.  Uses
                    the 32-bit ip address type.";
            }

The setter method _set_router_id(self, v, load=False) in the binding is failing. The router-id is of type inet:ip-address for which there is no match of type to a restricted type in YANGDynClass.

    if hasattr(v, "_utype"):
      v = v._utype(v)
    try:
      t = YANGDynClass(v,base=[RestrictedClassType(base_type=six.text_type, restriction_dict={'pattern': '(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\\.){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\\p{N}\\p{L}]+)?'}),RestrictedClassType(base_type=six.text_type, restriction_dict={'pattern': '((:|[0-9a-fA-F]{0,4}):)([0-9a-fA-F]{0,4}:){0,5}((([0-9a-fA-F]{0,4}:)?(:|[0-9a-fA-F]{0,4}))|(((25[0-5]|2[0-4][0-9]|[01]?[0-9]?[0-9])\\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9]?[0-9])))(%[\\p{N}\\p{L}]+)?'}),], is_leaf=True, yang_name="router-id", parent=self, path_helper=self._path_helper, extmethods=self._extmethods, register_paths=True, is_keyval=True, namespace='urn:com:cxx:abc:yang:ospf', defining_module='ospf', yang_type='inet:ip-address', is_config=True)
    except (TypeError, ValueError):
      raise ValueError({
          'error-string': """router_id must be of a type compatible with inet:ip-address""",
          'defined-type': "inet:ip-address",
          'generated-type': """YANGDynClass(base=[RestrictedClassType(base_type=six.text_type, restriction_dict={'pattern': '(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\\.){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\\p{N}\\p{L}]+)?'}),RestrictedClassType(base_type=six.text_type, restriction_dict={'pattern': '((:|[0-9a-fA-F]{0,4}):)([0-9a-fA-F]{0,4}:){0,5}((([0-9a-fA-F]{0,4}:)?(:|[0-9a-fA-F]{0,4}))|(((25[0-5]|2[0-4][0-9]|[01]?[0-9]?[0-9])\\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9]?[0-9])))(%[\\p{N}\\p{L}]+)?'}),], is_leaf=True, yang_name="router-id", parent=self, path_helper=self._path_helper, extmethods=self._extmethods, register_paths=True, is_keyval=True, namespace='urn:com:cxx:abc:yang:ospf', defining_module='ospf', yang_type='inet:ip-address', is_config=True)""",
        })

    self.__router_id = t
    if hasattr(self, '_set'):
      self._set()

So this fails with the error message:

Traceback (most recent call last):
  File "/home/iang/PycharmProjects/abacus-cli/cnos-cli/pybind_eg/ospf_/instances/instance/__init__.py", line 117, in _set_router_id
    t = YANGDynClass(v,base=[RestrictedClassType(base_type=six.text_type, restriction_dict={'pattern': '(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\\.){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\\p{N}\\p{L}]+)?'}),RestrictedClassType(base_type=six.text_type, restriction_dict={'pattern': '((:|[0-9a-fA-F]{0,4}):)([0-9a-fA-F]{0,4}:){0,5}((([0-9a-fA-F]{0,4}:)?(:|[0-9a-fA-F]{0,4}))|(((25[0-5]|2[0-4][0-9]|[01]?[0-9]?[0-9])\\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9]?[0-9])))(%[\\p{N}\\p{L}]+)?'}),], is_leaf=True, yang_name="router-id", parent=self, path_helper=self._path_helper, extmethods=self._extmethods, register_paths=True, is_keyval=True, namespace='urn:com:cxx:abc:yang:ospf', defining_module='ospf', yang_type='inet:ip-address', is_config=True)
  File "/home/iang/PycharmProjects/pyangbind/pyangbind/lib/yangtypes.py", line 977, in YANGDynClass
    raise TypeError("did not find a valid type using the argument as a" + " hint")
TypeError: did not find a valid type using the argument as a hint

I get a couple of follow up errors with this message.

  File "/home/iang/PycharmProjects/pyangbind/pyangbind/lib/yangtypes.py", line 731, in __set
    raise KeyError("key value must be valid, %s" % m)
KeyError: 'key value must be valid, {\'error-string\': \'router_id must be of a type compatible with inet:ip-address\', \'defined-type\': \'inet:ip-address\', \'generated-type\': \'YANGDynClass(base=[RestrictedClassType(base_type=six.text_type, restriction_dict={\\\'pattern\\\': \\\'(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\\\\.){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\\\\p{N}\\\\p{L}]+)?\\\'}),RestrictedClassType(base_type=six.text_type, restriction_dict={\\\'pattern\\\': \\\'((:|[0-9a-fA-F]{0,4}):)([0-9a-fA-F]{0,4}:){0,5}((([0-9a-fA-F]{0,4}:)?(:|[0-9a-fA-F]{0,4}))|(((25[0-5]|2[0-4][0-9]|[01]?[0-9]?[0-9])\\\\.){3}(25[0-5]|2[0-4][0-9]|[01]?[0-9]?[0-9])))(%[\\\\p{N}\\\\p{L}]+)?\\\'}),], is_leaf=True, yang_name="router-id", parent=self, path_helper=self._path_helper, extmethods=self._extmethods, register_paths=True, is_keyval=True, namespace=\\\'urn:com:cxx:abc:yang:ospf\\\', defining_module=\\\'ospf\\\', yang_type=\\\'inet:ip-address\\\', is_config=True)\'}'

pybind_eg.zip

[igrush]

I have discovered the issue. This seems to be an artifact of the 2 to 3 python conversion. In the RestrictedClassType in yangtypes.py, in the method mp_check there is a type check that looks for six types.

               def mp_check(value):
                    if not isinstance(value, six.string_types + (six.text_type,)):
                        return False
                    if regex.match(convert_regexp(regexp), value):
                        return True
                    return False

                return mp_check

However if the type passed in is a Restricted type it is passed in as an lxml.objectify.StringElement. This fails the check. If instead of the lxml.objectify.StringElement you use the lxml.objectify.StringElement.pyval, it will work.

My solution was to add a type check before the call to mp_check. in class RestrictedClass(base_type):

            if val is not False:
                for test in self._restriction_tests:
                    passed = False
                    if isinstance(val, objectify.StringElement):
                        val = val.pyval
                    if test(val) is not False:
                        passed = True
                        break
                if not passed:
                    raise ValueError("%s does not match a restricted type" % val)

[JoseIgnacioTamayo]

Hi,

Could you please try again with recent versions of pyangbind?

Python2 was deprecated, and pyangbind is working fine with python3.

Thanks.

[JoseIgnacioTamayo]

Closing issue without recent updates.