cisco.mso.mso_schema_site_anp unhandled exception with "MSO Error 400: Bad Request: (0)(0)(0)(0)/deploymentImmediacy error.path.missing"

phil-dotchon commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

We have a somewhat large and reasonably complex set of Ansible playbooks in development, and have recently discovered what appears to be an unexpected behaviour in this module.

The issue only occurs when the play is run for the second time; the issue does not occur on the first run of the play (after cleaning up NDO), but does occur on subsequent runs. The first run is successful and creates the tenant, schema, template and site anp, template and site epg, associates the vmm domain as expected. Without any other change a subsequent run will fail with the following error.

TASK [Create Application Profile in sites] **************************************************************************************************************************************************
failed: [e_us1_mso] (item=e-us-site1) => {"ansible_loop_var": "item", "changed": true, "current": {"anpRef": {"anpName": "recreate_bug", "schemaId": "62e812bc1f00007827a85276", "templateName": "e-us1"}, "epgs": [{"domainAssociations": [{"allowMicroSegmentation": false, "deployImmediacy": "lazy", "dn": "uni/vmmp-VMware/dom-dvs_tl2_bdc6_ci3b", "domainType": "vmmDomain", "epgLagPol": {"enhancedLagPol": {"dn": "uni/vmmp-VMware/dom-dvs_tl2_bdc6_ci3b/vswitchpolcont/enlacplagp-lag-pol-3b", "name": "lag-pol-3b"}}, "resolutionImmediacy": "pre-provision", "vlanEncapMode": "dynamic"}], "epgRef": "/schemas/62e812bc1f00007827a85276/templates/e-us1/anps/recreate_bug/epgs/Test_EPG", "selectors": [], "staticLeafs": [], "staticPorts": [], "subnets": [], "uSegAttrs": []}]}, "data": [{"op": "replace", "path": "/sites/60ec79a99e9dae6a5997ced9-e-us1/anps/recreate_bug", "value": {"anpRef": {"anpName": "recreate_bug", "schemaId": "62e812bc1f00007827a85276", "templateName": "e-us1"}, "epgs": [{"domainAssociations": [{"allowMicroSegmentation": false, "deployImmediacy": "lazy", "dn": "uni/vmmp-VMware/dom-dvs_tl2_bdc6_ci3b", "domainType": "vmmDomain", "epgLagPol": {"enhancedLagPol": {"dn": "uni/vmmp-VMware/dom-dvs_tl2_bdc6_ci3b/vswitchpolcont/enlacplagp-lag-pol-3b", "name": "lag-pol-3b"}}, "resolutionImmediacy": "pre-provision", "vlanEncapMode": "dynamic"}], "epgRef": "/schemas/62e812bc1f00007827a85276/templates/e-us1/anps/recreate_bug/epgs/Test_EPG", "selectors": [], "staticLeafs": [], "staticPorts": [], "subnets": [], "uSegAttrs": []}]}}], "info": {"body": {"code": 400, "message": "Bad Request: (0)(0)(0)(0)/deploymentImmediacy error.path.missing"}, "connection": "close", "content-language": "en", "content-length": "89", "content-type": "application/json", "error": {"code": -1, "message": {"code": 400, "message": "Bad Request: (0)(0)(0)(0)/deploymentImmediacy error.path.missing"}}, "method": "PATCH", "msg": "OK (89 bytes)", "server": "nginx", "status": 400, "strict-transport-security": "max-age=31536000; includeSubDomains", "url": "https://e-us1-ndo.fis.lab:443/mso/api/v1/schemas/62e812bc1f00007827a85276?validate=false", "x-ratelimit-limit": "-1", "x-ratelimit-remaining": "-1", "x-ratelimit-reset": "1559582945"}, "item": "e-us-site1", "msg": "MSO Error 400: Bad Request: (0)(0)(0)(0)/deploymentImmediacy error.path.missing", "payload": {"code": 400, "message": "Bad Request: (0)(0)(0)(0)/deploymentImmediacy error.path.missing"}, "status": 400}

If we do not associate the EPG with a vmm domain the problem does not appear. I have created a minimal example playbook to demonstrate the problem

Affected Module Name(s):

cisco.mso.mso_schema_site_anp

MSO version and MSO Platform

Orchestrator Version: 3.7(1j) on Nexus Dashboard Version 2.1(1e)

APIC version and APIC Platform for Site Level Resources

APIC Version 5.2(4d) on-premise. We do not need to deploy the schema in order to see the problem

Collection versions

cisco.mso 2.0.0

Output/ Error message

failed: [e_us1_mso] (item=e-us-site1) => {"ansible_loop_var": "item", "changed": true, "current": {"anpRef": {"anpName": "recreate_bug", "schemaId": "62e812bc1f00007827a85276", "templateName": "e-us1"}, "epgs": [{"domainAssociations": [{"allowMicroSegmentation": false, "deployImmediacy": "lazy", "dn": "uni/vmmp-VMware/dom-dvs_tl2_bdc6_ci3b", "domainType": "vmmDomain", "epgLagPol": {"enhancedLagPol": {"dn": "uni/vmmp-VMware/dom-dvs_tl2_bdc6_ci3b/vswitchpolcont/enlacplagp-lag-pol-3b", "name": "lag-pol-3b"}}, "resolutionImmediacy": "pre-provision", "vlanEncapMode": "dynamic"}], "epgRef": "/schemas/62e812bc1f00007827a85276/templates/e-us1/anps/recreate_bug/epgs/Test_EPG", "selectors": [], "staticLeafs": [], "staticPorts": [], "subnets": [], "uSegAttrs": []}]}, "data": [{"op": "replace", "path": "/sites/60ec79a99e9dae6a5997ced9-e-us1/anps/recreate_bug", "value": {"anpRef": {"anpName": "recreate_bug", "schemaId": "62e812bc1f00007827a85276", "templateName": "e-us1"}, "epgs": [{"domainAssociations": [{"allowMicroSegmentation": false, "deployImmediacy": "lazy", "dn": "uni/vmmp-VMware/dom-dvs_tl2_bdc6_ci3b", "domainType": "vmmDomain", "epgLagPol": {"enhancedLagPol": {"dn": "uni/vmmp-VMware/dom-dvs_tl2_bdc6_ci3b/vswitchpolcont/enlacplagp-lag-pol-3b", "name": "lag-pol-3b"}}, "resolutionImmediacy": "pre-provision", "vlanEncapMode": "dynamic"}], "epgRef": "/schemas/62e812bc1f00007827a85276/templates/e-us1/anps/recreate_bug/epgs/Test_EPG", "selectors": [], "staticLeafs": [], "staticPorts": [], "subnets": [], "uSegAttrs": []}]}}], "info": {"body": {"code": 400, "message": "Bad Request: (0)(0)(0)(0)/deploymentImmediacy error.path.missing"}, "connection": "close", "content-language": "en", "content-length": "89", "content-type": "application/json", "error": {"code": -1, "message": {"code": 400, "message": "Bad Request: (0)(0)(0)(0)/deploymentImmediacy error.path.missing"}}, "method": "PATCH", "msg": "OK (89 bytes)", "server": "nginx", "status": 400, "strict-transport-security": "max-age=31536000; includeSubDomains", "url": "https://x.x.x.x:443/mso/api/v1/schemas/62e812bc1f00007827a85276?validate=false", "x-ratelimit-limit": "-1", "x-ratelimit-remaining": "-1", "x-ratelimit-reset": "1559582945"}, "item": "e-us-site1", "msg": "MSO Error 400: Bad Request: (0)(0)(0)(0)/deploymentImmediacy error.path.missing", "payload": {"code": 400, "message": "Bad Request: (0)(0)(0)(0)/deploymentImmediacy error.path.missing"}, "status": 400}

Expected Behavior

essentially idempotent, so nothing should have happened on the subsequent runs

Actual Behavior

The module returns an unhandled exception with error message above, and will continue to do so until we at least remove the vmm domain from the EPG (through the NDO GUI). The play will then run correctly once again, but only once, and will again fail on subsequent runs:

$ ansible-playbook -i inventory_bug_recreate.yml play-bug-recreate.yml
[WARNING]: The required python-logstash/python3-logstash is not installed. pip install python3-logstash for Python 3

PLAY [Create tenant] ************************************************************************************************************************************************************************

TASK [Set mso_connection anchor] ************************************************************************************************************************************************************
ok: [e_us1_mso]

TASK [Set schema_template anchor] ***********************************************************************************************************************************************************
ok: [e_us1_mso]

TASK [Set anp_name] *************************************************************************************************************************************************************************
ok: [e_us1_mso]

TASK [Set all_sites] ************************************************************************************************************************************************************************
ok: [e_us1_mso]

TASK [Create Tenant] ************************************************************************************************************************************************************************
changed: [e_us1_mso]

TASK [adding templates and schema] **********************************************************************************************************************************************************
changed: [e_us1_mso]

TASK [adding site to template] **************************************************************************************************************************************************************
changed: [e_us1_mso] => (item=e-us-site1)
changed: [e_us1_mso] => (item=e-us-site2)

TASK [Create Application Profile] ***********************************************************************************************************************************************************
changed: [e_us1_mso]

TASK [Create Application Profile in sites] **************************************************************************************************************************************************
changed: [e_us1_mso] => (item=e-us-site1)
changed: [e_us1_mso] => (item=e-us-site2)

TASK [create EPG] ***************************************************************************************************************************************************************************
changed: [e_us1_mso]

TASK [create EPG in sites] ******************************************************************************************************************************************************************
changed: [e_us1_mso] => (item=e-us-site1)
changed: [e_us1_mso] => (item=e-us-site2)

TASK [EPG to VMM Domain] ********************************************************************************************************************************************************************
changed: [e_us1_mso] => (item=Adding dvs_tl2_bdc6_ci3b to e-us-site1})
changed: [e_us1_mso] => (item=Adding dvs_tl2_bdd6_ci3b to e-us-site2})

PLAY RECAP **********************************************************************************************************************************************************************************
e_us1_mso                  : ok=12   changed=8    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

$ 
$ 
$ ansible-playbook -i inventory_bug_recreate.yml play-bug-recreate.yml
[WARNING]: The required python-logstash/python3-logstash is not installed. pip install python3-logstash for Python 3

PLAY [Create tenant] ************************************************************************************************************************************************************************

TASK [Set mso_connection anchor] ************************************************************************************************************************************************************
ok: [e_us1_mso]

TASK [Set schema_template anchor] ***********************************************************************************************************************************************************
ok: [e_us1_mso]

TASK [Set anp_name] *************************************************************************************************************************************************************************
ok: [e_us1_mso]

TASK [Set all_sites] ************************************************************************************************************************************************************************
ok: [e_us1_mso]

TASK [Create Tenant] ************************************************************************************************************************************************************************
ok: [e_us1_mso]

TASK [adding templates and schema] **********************************************************************************************************************************************************
ok: [e_us1_mso]

TASK [adding site to template] **************************************************************************************************************************************************************
ok: [e_us1_mso] => (item=e-us-site1)
ok: [e_us1_mso] => (item=e-us-site2)

TASK [Create Application Profile] ***********************************************************************************************************************************************************
ok: [e_us1_mso]

TASK [Create Application Profile in sites] **************************************************************************************************************************************************
failed: [e_us1_mso] (item=e-us-site1) => {"ansible_loop_var": "item", "changed": true, "current": {"anpRef": {"anpName": "recreate_bug", "schemaId": "62e823ac200000953625970f", "templateName": "e-us1"}, "epgs": [{"domainAssociations": [{"allowMicroSegmentation": false, "deployImmediacy": "lazy", "dn": "uni/vmmp-VMware/dom-dvs_tl2_bdc6_ci3b", "domainType": "vmmDomain", "epgLagPol": {"enhancedLagPol": {"dn": "uni/vmmp-VMware/dom-dvs_tl2_bdc6_ci3b/vswitchpolcont/enlacplagp-lag-pol-3b", "name": "lag-pol-3b"}}, "resolutionImmediacy": "pre-provision", "vlanEncapMode": "dynamic"}], "epgRef": "/schemas/62e823ac200000953625970f/templates/e-us1/anps/recreate_bug/epgs/Test_EPG", "selectors": [], "staticLeafs": [], "staticPorts": [], "subnets": [], "uSegAttrs": []}]}, "data": [{"op": "replace", "path": "/sites/60ec79a99e9dae6a5997ced9-e-us1/anps/recreate_bug", "value": {"anpRef": {"anpName": "recreate_bug", "schemaId": "62e823ac200000953625970f", "templateName": "e-us1"}, "epgs": [{"domainAssociations": [{"allowMicroSegmentation": false, "deployImmediacy": "lazy", "dn": "uni/vmmp-VMware/dom-dvs_tl2_bdc6_ci3b", "domainType": "vmmDomain", "epgLagPol": {"enhancedLagPol": {"dn": "uni/vmmp-VMware/dom-dvs_tl2_bdc6_ci3b/vswitchpolcont/enlacplagp-lag-pol-3b", "name": "lag-pol-3b"}}, "resolutionImmediacy": "pre-provision", "vlanEncapMode": "dynamic"}], "epgRef": "/schemas/62e823ac200000953625970f/templates/e-us1/anps/recreate_bug/epgs/Test_EPG", "selectors": [], "staticLeafs": [], "staticPorts": [], "subnets": [], "uSegAttrs": []}]}}], "info": {"body": {"code": 400, "message": "Bad Request: (0)(0)(0)(0)/deploymentImmediacy error.path.missing"}, "connection": "close", "content-language": "en", "content-length": "89", "content-type": "application/json", "error": {"code": -1, "message": {"code": 400, "message": "Bad Request: (0)(0)(0)(0)/deploymentImmediacy error.path.missing"}}, "method": "PATCH", "msg": "OK (89 bytes)", "server": "nginx", "status": 400, "strict-transport-security": "max-age=31536000; includeSubDomains", "url": "https://x.x.x.x:443/mso/api/v1/schemas/62e823ac200000953625970f?validate=false", "x-ratelimit-limit": "-1", "x-ratelimit-remaining": "-1", "x-ratelimit-reset": "1559582945"}, "item": "e-us-site1", "msg": "MSO Error 400: Bad Request: (0)(0)(0)(0)/deploymentImmediacy error.path.missing", "payload": {"code": 400, "message": "Bad Request: (0)(0)(0)(0)/deploymentImmediacy error.path.missing"}, "status": 400}
failed: [e_us1_mso] (item=e-us-site2) => {"ansible_loop_var": "item", "changed": true, "current": {"anpRef": {"anpName": "recreate_bug", "schemaId": "62e823ac200000953625970f", "templateName": "e-us1"}, "epgs": [{"domainAssociations": [{"allowMicroSegmentation": false, "deployImmediacy": "lazy", "dn": "uni/vmmp-VMware/dom-dvs_tl2_bdd6_ci3b", "domainType": "vmmDomain", "epgLagPol": {"enhancedLagPol": {"dn": "uni/vmmp-VMware/dom-dvs_tl2_bdd6_ci3b/vswitchpolcont/enlacplagp-lag-pol-3b", "name": "lag-pol-3b"}}, "resolutionImmediacy": "pre-provision", "vlanEncapMode": "dynamic"}], "epgRef": "/schemas/62e823ac200000953625970f/templates/e-us1/anps/recreate_bug/epgs/Test_EPG", "selectors": [], "staticLeafs": [], "staticPorts": [], "subnets": [], "uSegAttrs": []}]}, "data": [{"op": "replace", "path": "/sites/60ec79b59e9dae6a5997ceda-e-us1/anps/recreate_bug", "value": {"anpRef": {"anpName": "recreate_bug", "schemaId": "62e823ac200000953625970f", "templateName": "e-us1"}, "epgs": [{"domainAssociations": [{"allowMicroSegmentation": false, "deployImmediacy": "lazy", "dn": "uni/vmmp-VMware/dom-dvs_tl2_bdd6_ci3b", "domainType": "vmmDomain", "epgLagPol": {"enhancedLagPol": {"dn": "uni/vmmp-VMware/dom-dvs_tl2_bdd6_ci3b/vswitchpolcont/enlacplagp-lag-pol-3b", "name": "lag-pol-3b"}}, "resolutionImmediacy": "pre-provision", "vlanEncapMode": "dynamic"}], "epgRef": "/schemas/62e823ac200000953625970f/templates/e-us1/anps/recreate_bug/epgs/Test_EPG", "selectors": [], "staticLeafs": [], "staticPorts": [], "subnets": [], "uSegAttrs": []}]}}], "info": {"body": {"code": 400, "message": "Bad Request: (1)(0)(0)(0)/deploymentImmediacy error.path.missing"}, "connection": "close", "content-language": "en", "content-length": "89", "content-type": "application/json", "error": {"code": -1, "message": {"code": 400, "message": "Bad Request: (1)(0)(0)(0)/deploymentImmediacy error.path.missing"}}, "method": "PATCH", "msg": "OK (89 bytes)", "server": "nginx", "status": 400, "strict-transport-security": "max-age=31536000; includeSubDomains", "url": "https://x.x.x.x:443/mso/api/v1/schemas/62e823ac200000953625970f?validate=false", "x-ratelimit-limit": "-1", "x-ratelimit-remaining": "-1", "x-ratelimit-reset": "1559582945"}, "item": "e-us-site2", "msg": "MSO Error 400: Bad Request: (1)(0)(0)(0)/deploymentImmediacy error.path.missing", "payload": {"code": 400, "message": "Bad Request: (1)(0)(0)(0)/deploymentImmediacy error.path.missing"}, "status": 400}

PLAY RECAP **********************************************************************************************************************************************************************************
e_us1_mso                  : ok=8    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   

$

Playbook tasks to Reproduce

---

- name: Create tenant
  hosts: e_us1_mso
  gather_facts: false
  tasks:
    - name: Set mso_connection anchor
      set_fact:
        mso_connection: &mso_connection
          host: "{{ ansible_host | mandatory }}"
          username: "{{ ansible_user | mandatory }}"
          password: "{{ ansible_password | mandatory }}"
          validate_certs: false
          use_ssl: true
          use_proxy: false
          timeout: "{{ mso_timeout | default(omit) }}"
        mso_retry: 2
        mso_retry_delay: 5

    - name: Set schema_template anchor
      set_fact:
        schema_template: &schema_template
          schema: "recreate_bug"
          template: "e-us1"

    - name: Set anp_name
      set_fact:
        anp_name: "recreate_bug"

    - name: Set all_sites
      set_fact:
        all_sites: ['e-us-site1', 'e-us-site2']

    - name: Create Tenant
      cisco.mso.mso_tenant:
        <<: *mso_connection
        tenant: "recreate_bug"
        display_name: "recreate_bug"
        users:
          - "{{ ansible_user }}"
        sites: "{{ all_sites }}"
      register: tenant_info
      retries: "{{ mso_retry }}"
      delay: "{{ mso_retry_delay }}"
      until: tenant_info is not failed

    - name: adding templates and schema
      cisco.mso.mso_schema_template:
        <<: *mso_connection
        <<: *schema_template
        tenant: "recreate_bug"
        state: present
      retries: "{{ mso_retry }}"
      delay: "{{ mso_retry_delay }}"
      register: mso_template
      until: mso_template is not failed

    - name: adding site to template
      cisco.mso.mso_schema_site:
        <<: *mso_connection
        <<: *schema_template
        site: "{{ item }}"
        state: present
      loop: "{{ all_sites }}"
      register: mso_site
      retries: "{{ mso_retry }}"
      delay: "{{ mso_retry_delay }}"
      until: mso_site is not failed

    - name: Create Application Profile
      cisco.mso.mso_schema_template_anp:
        <<: *mso_connection
        <<: *schema_template
        anp: "{{ anp_name }}"
        display_name: "{{ anp_name }}"

    - name: Create Application Profile in sites
      cisco.mso.mso_schema_site_anp:
        <<: *mso_connection
        <<: *schema_template
        anp: "{{ anp_name }}"
        site: "{{ item }}"
      loop: "{{ all_sites }}"

    - name: create EPG
      cisco.mso.mso_schema_template_anp_epg:
        <<: *mso_connection
        <<: *schema_template
        anp: "{{ anp_name }}"
        epg: "Test_EPG"
        display_name: "Test EPG"
        state: "present"
        bd:
          name: "application_e-us1"
          schema: Common Tenant Infrastructure
          template: "Infrastructure_e-us1"
      register: create_epg

    - name: create EPG in sites
      cisco.mso.mso_schema_site_anp_epg:
        <<: *mso_connection
        <<: *schema_template
        anp: "{{ anp_name }}"
        epg: "Test_EPG"
        site: "{{ item }}"
        state: "present"
      loop: "{{ all_sites }}"

    - name: EPG to VMM Domain
      cisco.mso.mso_schema_site_anp_epg_domain:
        <<: *mso_connection
        <<: *schema_template
        site: "{{vmm.0.name }}"
        anp: "{{ anp_name }}"
        epg: "Test_EPG"
        domain_association_type: vmmDomain
        domain_profile: "{{ vmm.1}}"
        deployment_immediacy: lazy
        resolution_immediacy: pre-provision
        state: present
        vlan_encap_mode: dynamic
        enhanced_lagpolicy_name: "lag-pol-3b"
        enhanced_lagpolicy_dn: "uni/vmmp-VMware/dom-{{ vmm.1 }}/vswitchpolcont/enlacplagp-lag-pol-3b"
      retries: "{{ mso_retry }}"
      delay: "{{ mso_retry_delay }}"
      register: configure_vmware
      until: configure_vmware is not failed
      loop: "{{ vmm_deployment_list | subelements('domains') }}"
      loop_control:
        loop_var: vmm
        label: Adding {{vmm.1}} to {{vmm.0.name }}}
      vars:
        vmm_deployment_list:
          - name: e-us-site1
            domains:
              - dvs_tl2_bdc6_ci3b
          - name: e-us-site2
            domains:
              - dvs_tl2_bdd6_ci3b

all:
  hosts:
    e_us1_mso:
      ansible_host: x.x.x.x
      ansible_user: someone
      ansible_network_os: cisco.nd.nd
      ansible_connection: ansible.netcommon.httpapi
      ansible_httpapi_validate_certs: false
      ansible_httpapi_use_ssl: true
      ansible_httpapi_use_proxy: false
      ansible_password: !vault |
                $ANSIBLE_VAULT;1.1;AES256
                 00000000000000000000000

Important Factoids

References

xinyuezhao commented 2 years ago

@phil-dotchon Thanks for providing the playbook, we are looking into this issue and will update with you once we fix it

xinyuezhao commented 2 years ago

@phil-dotchon we are trying to reproduce your issue and seems like that there are some configurations missing, like BD 'application_e-us1' and VMM Domains. Would you mind sharing these with us?

phil-dotchon commented 2 years ago

Hi @xinyuezhao, we have the BD and VRF in common tenant, which is created by a separate code base and is shared across all application instances. I am attaching the NDO schema extract for common tenant that shows the BD, though I have remove other elements. I'm also attaching extracts from the 2 site APICs that shows the vmm domains. I hope that helps and is enough, but let me know BDs and VMM domains.zip .

phil-dotchon commented 2 years ago

Hi @xinyuezhao , I'm wondering if you were able to recreate the issue?

akinross commented 2 years ago

HI @phil-dotchon, I was able to recreate the issue and am currently working on fixing it. I will keep you posted on my progress.

akinross commented 2 years ago

Hi @phil-dotchon,

I think I have isolated the issue to the deploymentImmediacy attribute in domainAssociations which is named different on response deployImmediacy. The assumption was made in the code that the retrieved response attributes of the child (domainAssociations) are equal to one in the POST, but it seems MSO has changed this somehow over time. So I think in every module where this child object is put in the replace operation the issue will occur, (verified this issue occurs on bot mso_schema_site_anp and mso_schema_site_anp_epg when domainAssociations is present).

The question now remains on where/how to solve this, since arguably this could be marked as MSO API bug. I will discuss this in my team and will see if I can get in touch with TAC, (saw you had a case opened already). I will get back to you on this as soon as I have more information.

phil-dotchon commented 1 year ago

Hi @akinross , I see you have created a PR but that it isn't yet merged; do you have an outlook for that? Thanks :) -phil

akinross commented 1 year ago

Hi @phil-dotchon, no I don't know any timelines for merge. Regards, Akini

CiscoDevNet / ansible-mso