nuagenetworks / nuage-metroae

Nuage Networks Metro Automation Engine
http://devops.nuagenetworks.net
Apache License 2.0
44 stars 17 forks source link

MetroaAE Config Excel: template variable of type "choice" returns an error when string value is selected in xlsx file #1683

Closed medometo closed 3 years ago

medometo commented 3 years ago

I have noticed in two occasions that a template variable of type "choice" which is actually a string causes an error when the value of the variable is set in the Excel file/template.

The two occasions that I have tested are:

1) When creating a Policy Group using an Excel template and select the "Policy group type" from the drop-down list (for example setting it to "SOFTWARE"), will cause an error when validating or creating the Policy Group.

policy_group_excel

[mmetwall@metroae ~]$ metroae config validate excel/user_data_form_2.xlsx --debug
ERROR : Could not parse VSD API specification downloadprogress.spec: 'resource_name' missing in specification {u'resource_name': None, u'description': u'An object representing the progress of gateway image download', u'entity_name': u'DownloadProgress', u'get': True, u'create': False, u'package': u'infrastructure', u'update': True, u'userlabel': u'Download Progress', u'rest_name': None, u'extends': [], u'template': None, u'allowed_job_commands': None, u'root': None, u'delete': False}
ERROR : Could not parse VSD API specification forwardingclass.spec: 'resource_name' missing in specification {u'resource_name': None, u'description': u'Contains the Forwarding Class and its usage for load balancing.', u'entity_name': u'ForwardingClass', u'get': True, u'create': False, u'package': None, u'update': True, u'userlabel': u'Forwarding Class', u'rest_name': None, u'extends': [], u'template': None, u'allowed_job_commands': None, u'root': None, u'delete': False}
ERROR : Could not parse VSD API specification sysmonuplinkconnection.spec: 'resource_name' missing in specification {u'resource_name': None, u'description': u'Models the connection between a VRS and the controller.', u'entity_name': u'SysmonUplinkConnection', u'get': True, u'create': False, u'package': None, u'update': True, u'userlabel': u'Sysmon Uplink Connection', u'rest_name': None, u'extends': [], u'template': None, u'allowed_job_commands': None, u'root': None, u'delete': False}
OUTPUT: Device: Nuage Networks VSD 5.4.1
ERROR : In template Policy Group, variable policy_group_type: is not a string
ERROR : Stack trace
Traceback (most recent call last):
  File "/source/nuage-metroae-config/metroae_config.py", line 386, in run
    self.apply_templates()
  File "/source/nuage-metroae-config/metroae_config.py", line 721, in apply_templates
    config.add_template_data(template_name, **template_data)
  File "/source/nuage-metroae-config/nuage_metroae_config/configuration.py", line 69, in add_template_data
    template.validate_template_data(**template_data)
  File "/source/nuage-metroae-config/nuage_metroae_config/template.py", line 162, in validate_template_data
    self._validate_data(template_data)
  File "/source/nuage-metroae-config/nuage_metroae_config/template.py", line 461, in _validate_data
    self._validate_against_variable_info(var_info, name, value)
  File "/source/nuage-metroae-config/nuage_metroae_config/template.py", line 495, in _validate_against_variable_info
    var_type)
  File "/source/nuage-metroae-config/nuage_metroae_config/template.py", line 529, in _validate_variable_value
    self._raise_value_error(var_name, "is not a string")
  File "/source/nuage-metroae-config/nuage_metroae_config/template.py", line 577, in _raise_value_error
    self.get_name(), var_name, message))
VariableValueError: In template Policy Group, variable policy_group_type: is not a string

Error
----
In template Policy Group, variable policy_group_type: is not a string
[mmetwall@metroae ~]$

_A workaround is to modify the "policy_group_type" variable type in standard-templates/templates/policy_group.yml to "string" instead of "choice":_

variables:
  - name: enterprise_name
    description: Name of the Enterprise in which to create this Policy Group.
    type: reference
  - name: domain_name
    description: If configuring on L3 Domain, the name of the L3 Domain in which to
      create this Policy Group.
    type: reference
    optional: true
  - name: l2_domain_name
    description: If configuring on L2 Domain, the name of the L2 Domain in which to
      create this Policy Group.
    type: reference
    optional: true
  - name: policy_group_name
    description: Name of this Policy Group.
    type: string
  - name: description
    description: Optional description of this Policy Group. Defaults to "Policy Group
      policy_group_name".
    type: string
    optional: true
  - name: policy_group_type
    description: Type (SOFTWARE or HARDWARE) of this Policy Group. Defaults to SOFTWARE.
    type: choice    <<<< **Change to (string)**
    choices:
      - SOFTWARE
      - HARDWARE
    optional: true
  - name: policy_group_category_name
    description: Optional name of a Policy Group Category which is used for contextual
      filtering of Policy Groups.
    type: reference
    optional: true

Now the validation/creation succeeds:

[mmetwall@metroae ~]$ metroae config validate excel/user_data_form_2.xlsx --debug
ERROR : Could not parse VSD API specification downloadprogress.spec: 'resource_name' missing in specification {u'resource_name': None, u'description': u'An object representing the progress of gateway image download', u'entity_name': u'DownloadProgress', u'get': True, u'create': False, u'package': u'infrastructure', u'update': True, u'userlabel': u'Download Progress', u'rest_name': None, u'extends': [], u'template': None, u'allowed_job_commands': None, u'root': None, u'delete': False}
ERROR : Could not parse VSD API specification forwardingclass.spec: 'resource_name' missing in specification {u'resource_name': None, u'description': u'Contains the Forwarding Class and its usage for load balancing.', u'entity_name': u'ForwardingClass', u'get': True, u'create': False, u'package': None, u'update': True, u'userlabel': u'Forwarding Class', u'rest_name': None, u'extends': [], u'template': None, u'allowed_job_commands': None, u'root': None, u'delete': False}
ERROR : Could not parse VSD API specification sysmonuplinkconnection.spec: 'resource_name' missing in specification {u'resource_name': None, u'description': u'Models the connection between a VRS and the controller.', u'entity_name': u'SysmonUplinkConnection', u'get': True, u'create': False, u'package': None, u'update': True, u'userlabel': u'Sysmon Uplink Connection', u'rest_name': None, u'extends': [], u'template': None, u'allowed_job_commands': None, u'root': None, u'delete': False}
OUTPUT: Device: Nuage Networks VSD 5.4.1
DEBUG : *** Validate ***
DEBUG : Reading [select Enterprise (name of new-test-ent-2)] (template: Policy Group)
DEBUG : Reading [select Domain (name of l3dom1)] (template: Policy Group)
DEBUG : Reading PolicyGroup (template: Policy Group)
DEBUG : Reading [set values] (template: Policy Group)
DEBUG : Reading [select Enterprise (name of new-test-ent-2)] (template: Policy Group)
DEBUG : Reading [select Domain (name of l3dom1)] (template: Policy Group)
DEBUG : Reading PolicyGroup (template: Policy Group)
DEBUG : Reading [set values] (template: Policy Group)
DEBUG : Configuration
DEBUG :     [select Enterprise (name of new-test-ent-2)]
DEBUG :         [select Domain (name of l3dom1)]
DEBUG :             PolicyGroup
DEBUG :                 type = 'SOFTWARE'
DEBUG :                 description = 'PG 1'
DEBUG :                 external = False
DEBUG :                 name = 'pg1'
DEBUG :             PolicyGroup
DEBUG :                 type = 'SOFTWARE'
DEBUG :                 description = 'PG 2'
DEBUG :                 external = False
DEBUG :                 name = 'pg2'
DEBUG :
DEBUG : Session start {'username': 'csproot', 'password': 'Cor@#$lor1', 'api_url': 'https://10.216.80.94:8443', 'enterprise': 'csp'}
DEBUG : Select object Enterprise name = new-test-ent-2 [None]
DEBUG : Select object Domain name = l3dom1 [Root / Enterprise (ID=None)]
DEBUG : Create object PolicyGroup [Enterprise (ID=None) / Domain (ID=None)]
DEBUG : Set value [Domain (ID=None) / PolicyGroup (ID=None) **] = {'type': 'SOFTWARE', 'description': 'PG 1', 'external': False, 'name': 'pg1'}
DEBUG : Creating child [Domain (ID=None) / PolicyGroup (name=pg1, ID=None) **]
DEBUG : Saved [Domain (ID=None) / PolicyGroup (name=pg1, ID=None)]
DEBUG : Create object PolicyGroup [Enterprise (ID=None) / Domain (ID=None)]
DEBUG : Set value [Domain (ID=None) / PolicyGroup (ID=None) **] = {'type': 'SOFTWARE', 'description': 'PG 2', 'external': False, 'name': 'pg2'}
DEBUG : Creating child [Domain (ID=None) / PolicyGroup (name=pg2, ID=None) **]
DEBUG : Saved [Domain (ID=None) / PolicyGroup (name=pg2, ID=None)]
DEBUG : Session stopping
DEBUG : *** Writing ***
Configuration
    [select Enterprise (name of new-test-ent-2)]
        [select Domain (name of l3dom1)]
            PolicyGroup
                type = 'SOFTWARE'
                description = 'PG 1'
                external = False
                name = 'pg1'
            PolicyGroup
                type = 'SOFTWARE'
                description = 'PG 2'
                external = False
                name = 'pg2'

>>> All actions valid
[mmetwall@metroae ~]$

2) Similarly, when creating an L3 Domain using an Excel template and select the "Underlay enabled" attribute from the drop-down list (for example setting it to "ENABLED"), will cause an error when validating or creating the L3 Domain. Similar error is given like in the case of policy group type.

If we set the "underlay_enabled" variable in standard-templates/templates/l3_domain.yml to "string" instead of "choice", the validation/creation succeeds.

  - name: underlay_enabled
    description: Optional enablement of underlay access from the overlay. Defaults to
      DISABLED.
    type: choice     <<<< change this to (string)
    choices:
      - ENABLED
      - DISABLED
      - INHERITED
    optional: true

I know the above workarounds are just a "desperate man's" workaround and are unlikely to be the solution.

While looking at the code in source/nuage-metroae-config/nuage_metroae_config/template.py, I suspect that the issue is related to the part checking that var of type "choice" is a string in Python 3 using isinstance(value, str). I'm not a programmer, so I can be totally wrong.

527         elif var_type == "choice":
528             if not isinstance(value, str):
529                 self._raise_value_error(var_name, "is not a string")
530             choices = self._get_required_field(var_schema, "choices")
531             upper_choices = [x.upper() for x in choices]
532             if value.upper() in upper_choices:
533                 return True
534             else:
535                 self._raise_value_error(var_name, "is not a valid choice")
medometo commented 3 years ago

So, I have done some more research and I think the issue is related to how the isinstance(value, str) function checks for type string in Python 2 vs Python 3.

I tried two solutions to solve this.

Solution 1) Do a check of Python version in which I split the below code section:

    527         elif var_type == "choice":
    528             if not isinstance(value, str):
    529                 self._raise_value_error(var_name, "is not a string")
    530             choices = self._get_required_field(var_schema, "choices")
    531             upper_choices = [x.upper() for x in choices]
    532             if value.upper() in upper_choices:
    533                 return True
    534             else:
    535                 self._raise_value_error(var_name, "is not a valid choice")

and replace it with the following code in which I use isinstance(value, str) in Python 3 and use isinstance(value, unicode) in Python 2:

    527         elif var_type == "choice" and sys.version_info == (3,):
    528             if not isinstance(value, str):
    529                 self._raise_value_error(var_name, "is not a string")
    530             choices = self._get_required_field(var_schema, "choices")
    531             upper_choices = [x.upper() for x in choices]
    532             if value.upper() in upper_choices:
    533                 return True
    534             else:
    535                 self._raise_value_error(var_name, "is not a valid choice")
    536         elif var_type == "choice" and sys.version_info < (3,):
    537             if not isinstance(value, unicode):
    538                 self._raise_value_error(var_name, "is not a string")
    539             choices = self._get_required_field(var_schema, "choices")
    540             upper_choices = [x.upper() for x in choices]
    541             if value.upper() in upper_choices:
    542                 return True
    543             else:
    544                 self._raise_value_error(var_name, "is not a valid choice")

Solution 2) Follow the recommendation mentioned here, and use the six library to be both Python 2 and Python 3 compatible using one piece of code.

And this is by: a) Importing the six library:

import six

b) Rreplacing the following code (using isinstance(value, str):

    527         elif var_type == "choice":
    528             if not isinstance(value, str):
    529                 self._raise_value_error(var_name, "is not a string")
    530             choices = self._get_required_field(var_schema, "choices")
    531             upper_choices = [x.upper() for x in choices]
    532             if value.upper() in upper_choices:
    533                 return True
    534             else:
    535                 self._raise_value_error(var_name, "is not a valid choice")

by the following code; using isinstance(value, six.string_types)

    528         elif var_type == "choice":
    529             if not isinstance(value, six.string_types):
    530                 self._raise_value_error(var_name, "is not a string")
    531             choices = self._get_required_field(var_schema, "choices")
    532             upper_choices = [x.upper() for x in choices]
    533             if value.upper() in upper_choices:
    534                 return True
    535             else:
    536                 self._raise_value_error(var_name, "is not a valid choice")

Both solutions were tested successfully with the following templates that include different "string" choice var_type (of course while keeping the type of the variable as "choice" in the yml template file, i.e., no workaround as previously mentioned in the original comment reporting the issue):

medometo commented 3 years ago

Closing the issue here as it is reported by mistake in nuage-metroae. New issue created correctly under nuage-metroae-config project.