CiscoDevNet / ydk-gen

Generate model-driven APIs from YANG models
http://ciscodevnet.github.io/ydk-gen/
Apache License 2.0
137 stars 74 forks source link

Need remove '^' and '$' from all instances of pattern string in openconfig yang modules #964

Closed ygorelik closed 5 years ago

ygorelik commented 5 years ago

Current Behavior

Parsing of data based on openconfig models is causing YModelError exception due to failure to match string value with model defined patterns. The errors are similar to this:

ydk.errors.YModelError:  Value "173.31.1.0/24" does not satisfy the constraint "^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])/(([0-9])|([1-2][0-9])|(3[0-2]))$" (range, length, or pattern). Path: /openconfig-acl:acl/acl-sets/acl-set[name='ACL3'][type='openconfig-acl:ACL_IPV4']/acl-entries/acl-entry[sequence-id='20']/ipv4/config/source-address

The Google OpenConfig group has adopted POSIX regular expressions for coding string patterns in YANG modules "due to the lack of widely available software implementations for the W3C standard".

As per RFC 6020, the pattern string should follow the rules according to the XSD types https://tools.ietf.org/html/rfc6020#page-119, according to which the '^' and '$' character is taken literally as part of the regex and not as an indication of head and tail.

The Google team claimed that they are not going to change their YANG models and any such issues must be resolved locally in YANG model parsers. So the '^' character in the beginning of pattern string and '$' at the end needs to be removed automatically.

Steps to Reproduce

Try to run sample example from here using standard installation of ydk-models-openconfig package.

Logs

Enable logging and post the logs below

Observe this or similar error in the log:

2019-10-18 13:22:47,441 - ydk - ERROR - Data is invalid according to the yang model. Libyang error: Value "173.31.1.0/24" does not satisfy the constraint "^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])/(([0-9])|([1-2][0-9])|(3[0-2]))$" (range, length, or pattern). Path: '/openconfig-acl:acl/acl-sets/acl-set[name='ACL3'][type='openconfig-acl:ACL_IPV4']/acl-entries/acl-entry[sequence-id='20']/ipv4/config/source-address'
Traceback (most recent call last):
  File "acl2.py", line 92, in <module>
    main()
  File "acl2.py", line 84, in main
    print(codec.encode(provider, acl))
  File "/usr/local/lib/python3.6/dist-packages/ydk/errors/error_handler.py", line 112, in helper
    return func(self, provider, entity, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ydk/services/codec_service.py", line 78, in encode
    return self._encode(provider, entity_holder, pretty, subtree)
  File "/usr/local/lib/python3.6/dist-packages/ydk/services/codec_service.py", line 110, in _encode
    return result
  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/dist-packages/ydk/errors/error_handler.py", line 82, in handle_runtime_error
    _raise(_exc)
  File "/usr/local/lib/python3.6/dist-packages/ydk/errors/error_handler.py", line 54, in _raise
    exec("raise exc from None")
  File "<string>", line 1, in <module>
ydk.errors.YModelError:  Value "173.31.1.0/24" does not satisfy the constraint "^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])/(([0-9])|([1-2][0-9])|(3[0-2]))$" (range, length, or pattern). Path: /openconfig-acl:acl/acl-sets/acl-set[name='ACL3'][type='openconfig-acl:ACL_IPV4']/acl-entries/acl-entry[sequence-id='20']/ipv4/config/source-address

System Information

YDK-0.8.4

ygorelik commented 5 years ago

Fixed the issue in libyang. After recompiling YDK with libyang code changes and building python core package repeated the above test:

#!/usr/bin/env python
#
# Copyright 2016 Cisco Systems, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

"""
Encode configuration for model openconfig-acl.

usage: cd-encode-oc-acl-30-ydk.py [-h] [-v] device

optional arguments:
  -h, --help     show this help message and exit
  -v, --verbose  print debugging messages
"""

from argparse import ArgumentParser

import logging
import sys
if sys.version_info > (3,):
    from urllib.parse import urlparse
else:
    from urlparse import urlparse

from ydk.services import CodecService
from ydk.providers import CodecServiceProvider
from ydk.models.openconfig import openconfig_acl \
    as oc_acl

def config_acl(acl):
    """Add config data to acl object."""
    # acl-set configuration
    acl_set = acl.AclSets.AclSet()
    acl_set.name = "ACL1"
    acl_set.type = oc_acl.ACLIPV4()
    acl_set.config.name = "ACL1"
    acl_set.config.type = oc_acl.ACLIPV4()
    acl_set.acl_entries = acl_set.AclEntries()

    # acl-entry with sequence number 10
    acl_entry = acl_set.acl_entries.AclEntry()
    acl_entry.sequence_id = 10
    acl_entry.config.sequence_id = 10
    acl_set.acl_entries.acl_entry.append(acl_entry)

    # acl-entry with sequence number 20
    acl_entry = acl_set.acl_entries.AclEntry()
    acl_entry.sequence_id = 20
    acl_entry.config.sequence_id = 20
    acl_entry.actions.config.forwarding_action = oc_acl.ACCEPT()
    acl_entry.ipv4.config.source_address= "172.31.255.1/32"
    acl_set.acl_entries.acl_entry.append(acl_entry)

    # acl-entry with sequence number 30
    acl_entry = acl_set.acl_entries.AclEntry()
    acl_entry.sequence_id = 30
    acl_entry.config.sequence_id = 30
    acl_entry.actions.config.forwarding_action = oc_acl.REJECT()
    acl_set.acl_entries.acl_entry.append(acl_entry)
    acl.acl_sets.acl_set.append(acl_set)

if __name__ == "__main__":
    """Execute main program."""
    parser = ArgumentParser()
    parser.add_argument("-v", "--verbose", help="print debugging messages",
                        action="store_true")
    args = parser.parse_args()

    # log debug messages if verbose argument specified
    if args.verbose:
        logger = logging.getLogger("ydk")
        logger.setLevel(logging.INFO)
        handler = logging.StreamHandler()
        formatter = logging.Formatter(("%(asctime)s - %(name)s - "
                                      "%(levelname)s - %(message)s"))
        handler.setFormatter(formatter)
        logger.addHandler(handler)

    # create codec provider
    provider = CodecServiceProvider(type="xml")

    # create codec service
    codec = CodecService()

    acl = oc_acl.Acl()  # create object
    config_acl(acl)  # add object configuration

    # encode and print object
    print(codec.encode(provider, acl))

    exit()

Run results:

(venv) ygorelik@bionic:~/ydk-gen$ python scripts/samples/cd-encode-oc-acl-30-ydk.py -v
<acl xmlns="http://openconfig.net/yang/acl">
  <acl-sets>
    <acl-set>
      <name>ACL1</name>
      <type>ACL_IPV4</type>
      <config>
        <name>ACL1</name>
        <type>ACL_IPV4</type>
      </config>
      <acl-entries>
        <acl-entry>
          <sequence-id>10</sequence-id>
          <config>
            <sequence-id>10</sequence-id>
          </config>
        </acl-entry>
        <acl-entry>
          <sequence-id>20</sequence-id>
          <config>
            <sequence-id>20</sequence-id>
          </config>
          <ipv4>
            <config>
              <source-address>172.31.255.1/32</source-address>
            </config>
          </ipv4>
          <actions>
            <config>
              <forwarding-action>ACCEPT</forwarding-action>
            </config>
          </actions>
        </acl-entry>
        <acl-entry>
          <sequence-id>30</sequence-id>
          <config>
            <sequence-id>30</sequence-id>
          </config>
          <actions>
            <config>
              <forwarding-action>REJECT</forwarding-action>
            </config>
          </actions>
        </acl-entry>
      </acl-entries>
    </acl-set>
  </acl-sets>
</acl>

The codec worked with no issues.

horseinthesky commented 4 years ago

The clean installation of ydk 0.8.4 still has this issue.

ygorelik commented 4 years ago

In order to get the fix, you need recompile the libydk from source and install it. Please follow this procedure.

horseinthesky commented 4 years ago

Generate and install libydk library completed successfully

Code generation and installation completed successfully!

Total time taken: 5 minutes 32 seconds

Generate and install libydk_gnmi library (optional) failed

python3 generate.py -is --cpp --service profiles/services/gnmi-0.4.0.json
-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake System: Ubuntu
-- Found LibXml2: /usr/lib/x86_64-linux-gnu/libxml2.so (found version "2.9.4") 
CMake Error at /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find Protobuf (missing: PROTOBUF_LIBRARY PROTOBUF_INCLUDE_DIR
  PROTOBUF_PROTOC_EXECUTABLE)
Call Stack (most recent call first):
  /usr/share/cmake-3.10/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  CMakeModules/FindProtobuf.cmake:125 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:166 (find_package)

-- Configuring incomplete, errors occurred!
See also "/home/horseinthesky/ydk-gen/gen-api/cpp/ydk-service-gnmi/build/CMakeFiles/CMakeOutput.log".

ERROR: Failed to configure build!

It's said that is is optional so I've tried to build an ACL using openconfig model and it is still yelling at me:

python3 openconfig_acl.py                                   
Traceback (most recent call last):
  File "openconfig_acl.py", line 49, in <module>
    print(codec.encode(provider, acl))
  File "/usr/local/lib/python3.6/dist-packages/ydk/errors/error_handler.py", line 112, in helper
    return func(self, provider, entity, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ydk/services/codec_service.py", line 78, in encode
    return self._encode(provider, entity_holder, pretty, subtree)
  File "/usr/local/lib/python3.6/dist-packages/ydk/services/codec_service.py", line 110, in _encode
    return result
  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.6/dist-packages/ydk/errors/error_handler.py", line 82, in handle_runtime_error
    _raise(_exc)
  File "/usr/local/lib/python3.6/dist-packages/ydk/errors/error_handler.py", line 54, in _raise
    exec("raise exc from None")
  File "<string>", line 1, in <module>
ydk.errors.YModelError:  Value "172.31.255.1/32" does not satisfy the constraint "^(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])/(([0-9])|([1-2][0-9])|(3[0-2]))$" (range, length, or pattern). Path: /openconfig-acl:acl/acl-sets/acl-set[name='ACL1'][type='openconfig-acl:ACL_IPV4']/acl-entries/acl-entry[sequence-id='20']/ipv4/config/source-address
ygorelik commented 4 years ago

The build for _libydkgnmi failed, because the gNMI Requirements were not satisfied on your system. The python script failed, because you also need reinstall python core package after libydk got changed (sorry, I should mention this previously). Simply run:

cd ydk-gen
python3 generate.py --core -is

Note. You should not use '-s' (sudo) flag if you employ python virtual environment.

Alternatively you can install YDK from PyPi:

pip3 install -U ydk

As this is not a bug, let's continue this conversation in private or on Cisco Community website.