search

CiscoDevNet / ansible-aci

Cisco ACI Ansible Collection
https://galaxy.ansible.com/cisco/aci
GNU General Public License v3.0
138 stars 94 forks source link

Modify listify filter to accept mutli-level dictionaries #574

Closed aj-cruz closed 7 months ago

aj-cruz commented 8 months ago

Community Note

Description

It looks like the listify plugin only works with lists of dictionaries (one deep).
I would like the listify plugin to support ACI YAML topologies with multi-level dictionaries.

I like to organize my playbooks as close to the APIC GUI as possible including creating nested keys that match the GUI folders instead of semi-flattening everything. Something like this for example:

    policies:
        protocol:
          bfd:
            - name: BFD-ON
              description: Enable BFD
              admin_state: enabled
              detection_multiplier: 3
              min_tx_interval: 50
              min_rx_interval: 50
              echo_rx_interval: 50
              echo_admin_state: enabled
              sub_interface_optimization_state: enabled
          ospf: ...

I'm not really a developer so this may be totally wrong or inefficient, but here's what I did to listify.py to add this functionality:

def listify_worker(d, keys, depth, cache, prefix):
    # The prefix in the code is used to store the path of keys traversed in the nested dictionary,
    # which helps to generate unique keys for each value when flattening the dictionary.
    prefix += keys[depth] + "_"

    if isinstance(d[list(d.keys())[0]], dict) and keys[depth] in d and keys[depth + 1] in d[keys[depth]]:
      '''
      If the value of the dictionary key 0 is also a dictionary
        and the key we're on (from the keys list) is in the dictionary keys
        and the next key in the keys list is in the value dictionary

      Means this is a nested/multi-level dictionary that matches the path we provided to listify
      '''
      cache_work = cache.copy()
      for k, v in d.items():
          if isinstance(v, dict) and keys[depth + 1] == list(v.keys())[0]:
              '''
              If the dictionary value is also a dictionary
                and the next key in the keys list is in the value dictionary keys

              means this is a nested/multi-level dictionary, add it to the cache but it will have no data
              '''
              cache_key = prefix + list(v.keys())[0]
              cache_value = None
              cache_work[cache_key] = cache_value
      # If we're at the deepest level of keys
      if len(keys) - 1 == depth:
          yield cache_work
      else:
          for k, v in v.items():
              if k == keys[depth + 1] and isinstance(v, (dict, list)):
                  for result in listify_worker({k:v}, keys, depth + 1, cache_work, prefix):
                      yield result
    elif keys[depth] in d:
      '''
      Else if the key we're on (from the keys list) is in the dictionary keys

      Means this is potentially a dictionary who's value is a list with the data we're looking for
      '''
      for item in d[keys[depth]]:
          cache_work = cache.copy()
          if isinstance(item, dict):
              for k, v in item.items():
                  if isinstance(v, list) and all(isinstance(x, (str, int, float, bool, bytes)) for x in v) or not isinstance(v, (dict, list)):
                      # The cache in this code is a temporary storage that holds key-value pairs as the function navigates through the nested dictionary.
                      # It helps to generate the final output by remembering the traversed path in each recursive call.
                      cache_key = prefix + k
                      cache_value = v
                      cache_work[cache_key] = cache_value
              # If we're at the deepest level of keys
              if len(keys) - 1 == depth:
                  yield cache_work
              else:
                  for k, v in item.items():
                      if k == keys[depth + 1] and isinstance(v, (dict, list)):
                          for result in listify_worker({k: v}, keys, depth + 1, cache_work, prefix):
                              yield result

New or Affected Module(s):

listify.py

APIC version and APIC Platform

Collection versions

References

shrsr commented 8 months ago

@aj-cruz Thank you for opening the issue. We recently added the listify plugin to the collection which was already written by a contributor several years ago. Let me consult with the team to see if it would be good to broaden the plugin's functionality as mentioned in the use case above.

shrsr commented 8 months ago

@aj-cruz I just had a discussion with the team, can you please provide us with a complete example of the data structure along with the tasks you're using to extract the listified objects.

aj-cruz commented 8 months ago

Here is my full ACI Topology File.

Here are a couple tasks I use to extract the listified objects:

In this example networking is a dictionary key that doesn't have a list of data, its value is another dictionary

- name: Configure VRFs
  aci_vrf:
    host: "{{ apic_host }}"
    username: "{{ apic_user }}"
    password: "{{ apic_pass }}"
    use_proxy: "{{ apic_use_proxy }}"
    validate_certs: "{{ apic_validate_certs }}"
    annotation: "{% if disable_default_annotation %}{{item.tenants_networking_vrfs_annotation | default('')}}{% else %}orchestrator:ansible{% endif %}"
    state: "{{ item.tenants_networking_vrfs_state | default('present') }}"
    tenant: "{{ item.tenants_name }}"
    vrf: "{{ item.tenants_networking_vrfs_name }}"
    descr: "{{ item.tenants_networking_vrfs_description }}"
    policy_control_preference: "{% if item.tenants_networking_vrfs_policy_control_enforce %}enforced{% else %}unenforced{% endif %}"
    policy_control_direction: "{{ item.tenants_networking_vrfs_policy_control_enforcement_direction }}"
    preferred_group: "{% if item.tenants_networking_vrfs_preferred_group_enable %}enabled{% else %}disabled{% endif %}"
    ip_data_plane_learning: "{% if item.tenants_networking_vrfs_enable_dataplane_learning %}enabled{% else %}disabled{% endif %}"
  delegate_to: localhost
  with_items: '{{ vars | cisco.aci.aci_listify("tenants","networking","vrfs") }}'
  loop_control:
    label: "Tenant '{{ item.tenants_name }}', VRF '{{ item.tenants_networking_vrfs_name }}''"
  when: item.tenants_state | default('present') == 'present'

In this example both policies and protocol are keys without list values

- name: Configure BFD Protocol Policy
  aci_rest:
    <<: *apic_login
    path: "/api/node/mo/uni/tn-{{ item.tenants_name }}/bfdIfPol-{{ item.tenants_policies_protocol_bfd_name }}.json"
    method: post
    content:
      bfdIfPol:
        attributes:
          dn: "uni/tn-{{ item.tenants_name }}/bfdIfPol-{{ item.tenants_policies_protocol_bfd_name }}"
          name: "{{ item.tenants_policies_protocol_bfd_name }}"
          descr: "{{ item.tenants_policies_protocol_bfd_description }}"
          adminSt: "{{ item.tenants_policies_protocol_bfd_admin_state }}"
          detectMult: "{{ item.tenants_policies_protocol_bfd_detection_multiplier }}"
          minTxIntvl: "{{ item.tenants_policies_protocol_bfd_min_tx_interval }}"
          minRxIntvl: "{{ item.tenants_policies_protocol_bfd_min_rx_interval }}"
          echoRxIntvl: "{{ item.tenants_policies_protocol_bfd_echo_rx_interval }}"
          echoAdminSt: "{{ item.tenants_policies_protocol_bfd_echo_admin_state }}"
          ctrl: "{% if item.tenants_policies_protocol_bfd_sub_interface_optimization_state == 'enabled' %}opt-subif{% else %}{% endif %}"
          rn: "bfdIfPol-{{ item.tenants_policies_protocol_bfd_name }}"
          status: "{% if item.tenants_policies_protocol_bfd_state is defined and item.tenants_policies_protocol_bfd_state == 'absent' %}deleted{% else %}created,modified{% endif %}"
  delegate_to: localhost
  with_items: '{{ vars | cisco.aci.aci_listify("tenants","policies","protocol","bfd") }}'
  loop_control:
    label: "Tenant '{{ item.tenants_name }}', BFD Interface Policy '{{ item.tenants_policies_protocol_bfd_name }}'"
  when: item.tenants_state | default('present') == 'present'

Since I posted this request I also made another change. I added the following condition to the value check before populating the cache: or (isinstance(v, dict) and not bool(set(keys).intersection(set(v.keys())))):

Basically if the value is a dictionary with no keys I'm looking for it will populate the cache with the full value. I see a potential problem with that if you use the same key names it might not work as expected. Also it does gather some additional possibly unnecessary data. The last value in the cache "tree" will have the full structure for the remainder of the topology. For example, using my topology if I give listify the keys: ("tenants","networking","l3outs") Cache key item.tenants_networking_l3outs will have a key/value pair that contains the "epgs" key and a list of all the EPGs.

Probably not what we want in most cases but, I don't think it hurts and it's actually beneficial (and the reason I did it) because it means I can access that data using dot notation. Here's an example of the task I use to do that for the L3Out OSPF configuration. You can see I'm pulling the attributes data using dot notation to go deeper into the structure:

- name: Configure OSPF if igp_routing_protocol = 'ospf'
  aci_rest:
    host: "{{ apic_host }}"
    username: "{{ apic_user }}"
    password: "{{ apic_pass }}"
    use_proxy: "{{ apic_use_proxy }}"
    validate_certs: "{{ apic_validate_certs }}"
    path: "/api/node/mo/uni/tn-{{ item.tenants_name }}/out-{{ item.tenants_networking_l3outs_name }}/ospfExtP.json"
    method: post
    content:
      ospfExtP:
        attributes:
          dn: "uni/tn-{{ item.tenants_name }}/out-{{ item.tenants_networking_l3outs_name }}/ospfExtP"
          areaId: "{{ item.tenants_networking_l3outs_ospf_config.area_id }}"
          areaType: "{{ item.tenants_networking_l3outs_ospf_config.area_type }}"
          areaCost: "{{ item.tenants_networking_l3outs_ospf_config.area_cost }}"
          areaCtrl: "{{ area_control }}"
  vars:
    area_control: |-
      {% if item.tenants_networking_l3outs_ospf_config.send_redistributed_lsas_into_nssa_area %}redistribute,{% endif %}
      {% if item.tenants_networking_l3outs_ospf_config.originate_summary_lsa %}summary,{% endif %}
      {% if item.tenants_networking_l3outs_ospf_config.suppress_forarding_address_in_translated_lsa %}suppress-fa,{% endif %}
  delegate_to: localhost
  with_items: '{{ vars | cisco.aci.aci_listify("tenants","networking","l3outs") }}'
  loop_control:
    label: "Tenant '{{ item.tenants_name }}', VRF '{{ item.tenants_networking_l3outs_vrf }}', L3Out {{ item.tenants_networking_l3outs_name }}'"
  when: item.tenants_state | default('present') == 'present'
        and item.tenants_networking_l3outs_state | default('present') == 'present'
        and item.tenants_networking_l3outs_igp_routing_protocol == 'ospf'
akinross commented 7 months ago

Hi @aj-cruz,

I am not sure I understand all your use cases and code but have came up with a slightly different solution. Would you be able to have a look whether this matches your expectations?

Code changes can be found here: https://github.com/CiscoDevNet/ansible-aci/pull/614/files#diff-10ea53b66748407e9ff4b7150292477165b43e287705e7abe8b8ac82ccf033fdR283-R285

As an example as input data I have taken the yaml structure below:

aci_model_data:
  tenant:
  - name: ansible_test2
    description: Created using listify
    app:
    - name: app_test2
      epg:
      - name: web2
        bd: web_bd2
      - name: app2
        bd: app_bd2
    policies:
      protocol:
        bfd:
        - name: BFD-ON
          description: Enable BFD
          admin_state: enabled
          detection_multiplier: 3
          min_tx_interval: 50
          min_rx_interval: 50
          echo_rx_interval: 50
          echo_admin_state: enabled
          sub_interface_optimization_state: enabled
        ospf:
          interface:
          - name: OSPF-P2P-IntPol
            network_type: p2p
            priority: 1
          - name: OSPF-Broadcast-IntPol
            network_type: bcast
            priority: 1

See below tasks for setting facts based on the yaml above and asserting the listified output:


- name: Set facts for nested dictionaries
  ansible.builtin.set_fact:
    bfd_listify_output: '{{ aci_model_data|cisco.aci.aci_listify("tenant", "policies", "protocol", "bfd") }}'
    ospf_listify_output: '{{ aci_model_data|cisco.aci.aci_listify("tenant", "policies", "protocol", "ospf", "interface") }}'

- name: Validate listify for nested dictionaries
  ansible.builtin.assert:
    that:
      - bfd_listify_output.0.tenant_name == "ansible_test2"
      - bfd_listify_output.0.tenant_description == "Created using listify"
      - bfd_listify_output.0.tenant_policies_protocol_bfd_admin_state == "enabled"
      - bfd_listify_output.0.tenant_policies_protocol_bfd_description == "Enable BFD"
      - bfd_listify_output.0.tenant_policies_protocol_bfd_detection_multiplier == 3
      - bfd_listify_output.0.tenant_policies_protocol_bfd_echo_admin_state == "enabled"
      - bfd_listify_output.0.tenant_policies_protocol_bfd_echo_rx_interval == 50
      - bfd_listify_output.0.tenant_policies_protocol_bfd_min_rx_interval == 50
      - bfd_listify_output.0.tenant_policies_protocol_bfd_min_tx_interval == 50
      - bfd_listify_output.0.tenant_policies_protocol_bfd_name == "BFD-ON"
      - bfd_listify_output.0.tenant_policies_protocol_bfd_sub_interface_optimization_state == "enabled"
      - ospf_listify_output.0.tenant_name == "ansible_test2"
      - ospf_listify_output.0.tenant_description == "Created using listify"
      - ospf_listify_output.0.tenant_policies_protocol_ospf_interface_name == "OSPF-P2P-IntPol"
      - ospf_listify_output.0.tenant_policies_protocol_ospf_interface_network_type == "p2p"
      - ospf_listify_output.0.tenant_policies_protocol_ospf_interface_priority == 1
      - ospf_listify_output.1.tenant_policies_protocol_ospf_interface_name == "OSPF-Broadcast-IntPol"
      - ospf_listify_output.1.tenant_policies_protocol_ospf_interface_network_type == "bcast"
      - ospf_listify_output.1.tenant_policies_protocol_ospf_interface_priority == 1