search

hardaker / nb2an

Converts or compares information in NetBox to Ansible
2 stars 1 forks source link

foreach_create_dict fails with "failed to call function foreach_create_dict" on more complex arrays #1

Closed ronnyaa closed 1 year ago

ronnyaa commented 1 year ago

Hello.

when trying the foreach_create_dict example from https://nb2an.readthedocs.io/en/latest/advanced_syntax.html everything works as expected. But when trying the same with interfaces the section do not get written. DEBUG shows the message "failed to call function foreach_create_dict" . i tried making the structure and function as simple as possible. have also tried replacing display with id or name, with same result. 

test3:
  __function: foreach_create_dict
  array: interfaces
  keyname: display
  structure:
    testing: display

the netbox database is barebones, a few devices using this device_type https://github.com/netbox-community/devicetype-library/blob/master/device-types/Dell/networking-s5248f-on.yaml

is there some other information i can provide in order to troubleshoot further ?

hardaker commented 1 year ago

Hmmm... Did it give you a stack trace too by chance?

hardaker commented 1 year ago

I'll note that your example yaml has a 'name' keyword for the interface, but not 'display'. So if nothing else it looks like you need to change display in the above to name most likely.

hardaker commented 1 year ago

I just pushed 0.9.10 that contains more debugging output. Can you try again, and make sure to enable --ll debug?

ronnyaa commented 1 year ago

thanks!

With this new version there is a stack trace.
have tried with display, name, id in sample.yml, same result. have tried with a test device, with a single interface. as well as multiple devices with 48 interfaces in netbox.

--

$ nb-update-ansible -c sample.yml --devices jalla-test --ll debug

DEBUG     :     loading config from /home/fits/.nb2an
DEBUG     :     loading config from /home/fits/.nb2an
DEBUG     :     fetching: https://netbox.intern/api/dcim/devices/
DEBUG     :     Starting new HTTPS connection (1): netbox.intern:443
DEBUG     :     https://netbox.intern:443 "GET /api/dcim/devices/ HTTP/1.1" 200 17605
DEBUG     :     fetching: https://netbox.intern/api/dcim/interfaces/
DEBUG     :     Starting new HTTPS connection (1): netbox.intern:443
DEBUG     :     https://netbox.intern:443 "GET /api/dcim/interfaces/ HTTP/1.1" 200 57340
DEBUG     :     fetching: https://netbox.intern/api/ipam/ip-addresses/?family=4
DEBUG     :     Starting new HTTPS connection (1): netbox.intern:443
DEBUG     :     https://netbox.intern:443 "GET /api/ipam/ip-addresses/?family=4 HTTP/1.1" 200 52
DEBUG     :     fetching: https://netbox.intern/api/ipam/ip-addresses/?family=6
DEBUG     :     Starting new HTTPS connection (1): netbox.intern:443
DEBUG     :     https://netbox.intern:443 "GET /api/ipam/ip-addresses/?family=6 HTTP/1.1" 200 52
DEBUG     :     returning cached: https://netbox.intern/api/dcim/devices/
DEBUG     :     fetching: https://netbox.intern/api/dcim/power-outlets/
DEBUG     :     Starting new HTTPS connection (1): netbox.intern:443
DEBUG     :     https://netbox.intern:443 "GET /api/dcim/power-outlets/ HTTP/1.1" 200 52
DEBUG     :     fetching: https://netbox.intern/api/dcim/power-ports/
DEBUG     :     Starting new HTTPS connection (1): netbox.intern:443
DEBUG     :     https://netbox.intern:443 "GET /api/dcim/power-ports/ HTTP/1.1" 200 15498
DEBUG     :     starting: a2-l1.intern
INFO      :     modifying /home/fits/nb2an/host_vars/a2-l1.intern.yml
DEBUG     :     looking for device a2-l1.intern by name
DEBUG     :     returning cached: https://netbox.intern/api/dcim/power-outlets/
DEBUG     :     returning cached: https://netbox.intern/api/dcim/power-ports/
ERROR     :     failed to call function foreach_create_dict for item test3
Traceback (most recent call last):
  File "/home/fits/.local/lib/python3.9/site-packages/nb2an/tools/update_ansible.py", line 94, in process_changes
    value = fn(dn, yaml_struct, changes[item], item)
  File "/home/fits/.local/lib/python3.9/site-packages/nb2an/plugins/update_ansible.py", line 55, in fn_foreach_create_dict
    fn_foreach_augment_dict(dn, yaml_struct, definition, item)
  File "/home/fits/.local/lib/python3.9/site-packages/nb2an/plugins/update_ansible.py", line 16, in _wrap
    return function(*args, **kwargs)
  File "/home/fits/.local/lib/python3.9/site-packages/nb2an/plugins/update_ansible.py", line 73, in fn_foreach_augment_dict
    keyvalue = dn.get(f"{array_name}.{n}.{definition['keyname']}")
  File "/home/fits/.local/lib/python3.9/site-packages/nb2an/dotnest.py", line 24, in get
    raise ValueError(f"key #{n} '{k}' not found in data")
ValueError: key #1 '0' not found in data

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/fits/.local/bin/nb-update-ansible", line 8, in <module>
    sys.exit(main())
  File "/home/fits/.local/lib/python3.9/site-packages/nb2an/tools/update_ansible.py", line 184, in main
    process_devices(nb, ansible_directory, racks=args.racks, changes=changes)
  File "/home/fits/.local/lib/python3.9/site-packages/nb2an/tools/update_ansible.py", line 159, in process_devices
    process_host(nb, name, device_yaml, changes=changes)
  File "/home/fits/.local/lib/python3.9/site-packages/nb2an/tools/update_ansible.py", line 140, in process_host
    process_changes(changes, yaml_struct, nb_data)
  File "/home/fits/.local/lib/python3.9/site-packages/nb2an/tools/update_ansible.py", line 97, in process_changes
    errors = traceback.format_exception(exp)
TypeError: format_exception() missing 2 required positional arguments: 'value' and 'tb'
ocess_host(nb, name, device_yaml, changes=changes)
  File "/home/fits/.local/lib/python3.9/site-packages/nb2an/tools/update_ansible.py", line 140, in process_host
    process_changes(changes, yaml_struct, nb_data)
  File "/home/fits/.local/lib/python3.9/site-packages/nb2an/tools/update_ansible.py", line 97, in process_changes
    errors = traceback.format_exception(exp)
TypeError: format_exception() missing 2 required positional arguments: 'value' and 'tb'

sample.yml

test:
  asn: custom_fields.custom_bgp_asn.asn

test3:
  __function: foreach_create_dict
  array: interfaces
  keyname: display
  structure:
    testing: "display"

https://netbox.intern/api/dcim/interfaces/487/?format=json

{"id":487,"url":"https://netbox.intern/api/dcim/interfaces/487/?format=json","display":"1","device":{"id":13,"url":"https://netbox.intern/api/dcim/devices/13/?format=json","display":"jalla-test","name":"jalla-test"},"module":null,"name":"1","label":"","type":{"value":"25gbase-x-sfp28","label":"SFP28 (25GE)"},"enabled":true,"parent":null,"bridge":null,"lag":null,"mtu":null,"mac_address":null,"speed":null,"duplex":null,"wwn":null,"mgmt_only":false,"description":"","mode":null,"rf_role":null,"rf_channel":null,"poe_mode":null,"poe_type":null,"rf_channel_frequency":null,"rf_channel_width":null,"tx_power":null,"untagged_vlan":null,"tagged_vlans":[],"mark_connected":false,"cable":null,"cable_end":"","wireless_link":null,"link_peers":[],"link_peers_type":null,"wireless_lans":[],"vrf":null,"l2vpn_termination":null,"connected_endpoints":null,"connected_endpoints_type":null,"connected_endpoints_reachable":null,"tags":[],"custom_fields":{},"created":"2022-11-24T08:15:45.733161Z","last_updated":"2022-11-24T11:50:42.191499Z","count_ipaddresses":0,"count_fhrp_groups":0,"_occupied":false}

$ nb-device 13

#
# device: #13
#
airflow: null
asset_tag: null
cluster: null
comments: ''
config_context: {}
created: '2022-11-24T08:15:45.686997Z'
custom_fields:
  custom_backup_destination: null
  custom_bgp_asn:
    asn: 65121
    display: AS65121
    id: 1
    url: https://netbox.intern/api/ipam/asns/1/
  custom_vlt_mac: null
device_role:
  display: Leaf Switch
  id: 1
  name: Leaf Switch
  slug: leafswitch
  url: https://netbox.intern/api/dcim/device-roles/1/
device_type:
  display: S5248F-ON
  id: 2
  manufacturer:
    display: Dell
    id: 1
    name: Dell
    slug: dell
    url: https://netbox.intern/api/dcim/manufacturers/1/
  model: S5248F-ON
  slug: s5248f-on
  url: https://netbox.intern/api/dcim/device-types/2/
display: jalla-test
face:
  label: Front
  value: front
id: 13
last_updated: '2022-11-24T08:19:22.663430Z'
local_context_data: null
location: null
name: jalla-test
parent_device: null
platform: null
position: 43.0
power_ports:
- _occupied: false
  allocated_draw: 310
  cable: null
  cable_end: ''
  connected_endpoints: null
  connected_endpoints_reachable: null
  connected_endpoints_type: null
  created: '2022-11-24T08:15:45.721059Z'
  custom_fields: {}
  description: ''
  device:
    display: jalla-test
    id: 13
    name: jalla-test
    url: https://netbox.intern/api/dcim/devices/13/
  display: PS1
  id: 21
  label: ''
  last_updated: '2022-11-24T08:15:45.721080Z'
  link_peers: []
  link_peers_type: null
  mark_connected: false
  maximum_draw: 647
  module: null
  name: PS1
  tags: []
  type:
    label: C14
    value: iec-60320-c14
  url: https://netbox.intern/api/dcim/power-ports/21/
- _occupied: false
  allocated_draw: 310
  cable: null
  cable_end: ''
  connected_endpoints: null
  connected_endpoints_reachable: null
  connected_endpoints_type: null
  created: '2022-11-24T08:15:45.721172Z'
  custom_fields: {}
  description: ''
  device:
    display: jalla-test
    id: 13
    name: jalla-test
    url: https://netbox.intern/api/dcim/devices/13/
  display: PS2
  id: 22
  label: ''
  last_updated: '2022-11-24T08:15:45.721179Z'
  link_peers: []
  link_peers_type: null
  mark_connected: false
  maximum_draw: 647
  module: null
  name: PS2
  tags: []
  type:
    label: C14
    value: iec-60320-c14
  url: https://netbox.intern/api/dcim/power-ports/22/
primary_ip: null
primary_ip4: null
primary_ip6: null
rack:
  display: A1
  id: 2
  name: A1
  url: https://netbox.intern/api/dcim/racks/2/
serial: ''
site:
  display: Site1
  id: 1
  name: Site1
  slug: BF
  url: https://netbox.intern/api/dcim/sites/1/
status:
  label: Active
  value: active
tags: []
tenant:
  display: Datasenter
  id: 1
  name: Datasenter
  slug: datasenter
  url: https://netbox.intern/api/tenancy/tenants/1/
url: https://netbox.intern/api/dcim/devices/13/
vc_position: null
vc_priority: null
virtual_chassis: null
ronnyaa commented 1 year ago

made a pull request #3 that fixes this problem for me.

hardaker commented 1 year ago

fixed in PR#3