search

nornir-automation / nornir

Pluggable multi-threaded framework with inventory management to help operate collections of devices
https://nornir.readthedocs.io/
Apache License 2.0
1.35k stars 226 forks source link

Unable to filter based netbox custom field in dict format #270

Closed walidamer711 closed 5 years ago

walidamer711 commented 5 years ago

Along with the host I receive a custom field using netbox inventory in dict format.

'vlandomain': {'value': 1, 'label': 'dc1'},

I am not able to filter based on this attribute, I have tried F filter with nested option as the following but it seems that work only when this information is attached to dict called "data" to search inside.

inv = nr.inventory.filter(F(vlandomain__value=1))

I got this error: AttributeError: 'NoneType' object has no attribute 'get'

however, I can access this directly from the host. In [28]: inv.hosts['rtr1'].get('vlandomain').get('value')
Out[28]: 1

dbarrosop commented 5 years ago

it seems that work only when this information is attached to dict called "data" to search inside

in 2.0.0 that's the only place where user data can exist, but that's handled by the plugin itself so you don't have to worry about it.

Which version of nornir are you running?

Can you also run the following code?

for host, data in inv.hosts.items():
    print(f"{host}")
    print(data.get("vlandomain").get("value"))

I suspect the issue is that some host may not have that "vlandomain" data but I need to confirm before writing a patch.

walidamer711 commented 5 years ago 
for host, data in inv.hosts.items():
    print(f"{host}")
    print(data.get("vlandomain").get("value"))

I suspect the issue is that some host may not have that "vlandomain" data but I need to confirm before writing a patch.

I am running 2.0.0 version. You are right not all the hosts have this "vlandomain" set and I got error for hosts that don't have this data when I run above code. AttributeError: 'Inventory' object has no attribute 'inventory'

dbarrosop commented 5 years ago

After some testing I can only reproduce your problem if some host has vlandomain = None. Is that your case?

dbarrosop commented 5 years ago

If I understood finally your issue (my previous comment) following PR should fix your issue: #272

Could you install from that branch and test? Thanks!

walidamer711 commented 5 years ago

After some testing I can only reproduce your problem if some host has vlandomain = None. Is that your case?

Yes this is the case. Other hosts have None value for "vlandomain". 'vlandomain': None,

walidamer711 commented 5 years ago

If I understood finally your issue (my previous comment) following PR should fix your issue: #272

Could you install from that branch and test? Thanks!

I'll test and confirm. Thanks

dbarrosop commented 5 years ago

I merged the PR so I will be closing this issue, please, if it doesn't solve your issue reopen it and let us know. Thanks!

walidamer711 commented 5 years ago

I used to initialize netbox inventory with the following in ver. 2.0.0: nr = InitNornir(num_workers=100, inventory="nornir.plugins.inventory.netbox.NBInventory", NBInventory={"nb_url": "http://xxxx", "nb_token": "xxx"}) but in this branch "fix-270", I initialize as the following: nr = InitNornir(core={"num_workers": 100}, inventory={"plugin": "nornir.plugins.inventory.netbox.NBInventory", "options": {"nb_url": "http://xxxx", "nb_token": "xxxx"}}) and I got these errors for all netbox custom fields for all hosts: hosts -> N5K_DC_ACC_01 -> dc_vlan_domain extra fields not permitted (type=value_error.extra) hosts -> N5K_DC_ACC_01 -> has_fex extra fields not permitted (type=value_error.extra) hosts -> N5K_DC_ACC_01 -> vlandomain extra fields not permitted (type=value_error.extra)

walidamer711 commented 5 years ago

Is it documented how to initialize netbox inventory in version 2.0.0-beta? I am not able to find it.

dbarrosop commented 5 years ago

What you are doing looks correct so you may have found a bug (that's why it's in beta :) ). Let me look at it. In order to reproduce and fix, would you mind sending me a sample of the call?

GET $NB_URL/api/dcim/devices/?limit=0

I need to see the json blob returned by that query (feel free to modify private/important information).

walidamer711 commented 5 years ago

it will be a huge json file if I attach 41 devices in the call. So, I have attached here two sample devices, one with custom filed set and one with None. { "id": 1, "name": "N5K_DC_ACC_01", "display_name": "N5K_DC_ACC_01", "device_type": { "id": 1, "url": "http://url/api/dcim/device-types/1/", "manufacturer": { "id": 2, "url": "http://url/api/dcim/manufacturers/2/", "name": "Cisco", "slug": "cisco" }, "model": "N5K-C5548UP", "slug": "n5k-c5548up" }, "device_role": { "id": 15, "url": "http://url/api/dcim/device-roles/15/", "name": "DC Access", "slug": "dc-access" }, "tenant": { "id": 2, "url": "http://url9/api/tenancy/tenants/2/", "name": "MZA-Infra", "slug": "mza-infra" }, "platform": { "id": 2, "url": "http://url/api/dcim/platforms/2/", "name": "Cisco NX-OS", "slug": "nxos" }, "serial": "", "asset_tag": null, "site": { "id": 2, "url": "http://url/api/dcim/sites/2/", "name": "MV1", "slug": "mv1" }, "rack": { "id": 1, "url": "http://url/api/dcim/racks/1/", "name": "DCC12", "display_name": "DCC12" }, "position": 20, "face": { "label": "Rear", "value": 1 }, "parent_device": null, "status": { "label": "Active", "value": 1 }, "primary_ip": { "id": 2, "url": "http://url/api/ipam/ip-addresses/2/", "family": 4, "address": "" }, "primary_ip4": { "id": 2, "url": "http://url/api/ipam/ip-addresses/2/", "family": 4, "address": "" }, "primary_ip6": null, "cluster": null, "virtual_chassis": null, "vc_position": null, "vc_priority": null, "comments": "", "tags": [], "custom_fields": { "vlandomain": { "value": 1, "label": "mv1_dc" }, "dc_vlan_domain": true, "has_fex": true }, "created": "2018-10-01", "last_updated": "2018-11-05T07:01:52.213888Z" },{ "id": 10, "name": "3750_OOB_EXT_SW", "display_name": "3750_OOB_EXT_SW", "device_type": { "id": 15, "url": "http://url/api/dcim/device-types/15/", "manufacturer": { "id": 2, "url": "http://url/api/dcim/manufacturers/2/", "name": "Cisco", "slug": "cisco" }, "model": "WS-C3750-24TS", "slug": "ws-c3750-24ts" }, "device_role": { "id": 10, "url": "http://url/api/dcim/device-roles/10/", "name": "OOB Switch", "slug": "oob-switch" }, "tenant": { "id": 2, "url": "http://url/api/tenancy/tenants/2/", "name": "MZA-Infra", "slug": "mza-infra" }, "platform": { "id": 1, "url": "http://url/api/dcim/platforms/1/", "name": "Cisco IOS", "slug": "cisco_ios" }, "serial": "", "asset_tag": null, "site": { "id": 2, "url": "http://url/api/dcim/sites/2/", "name": "MV1", "slug": "mv1" }, "rack": { "id": 2, "url": "http://url/api/dcim/racks/2/", "name": "DCC1", "display_name": "DCC1" }, "position": 24, "face": { "label": "Front", "value": 0 }, "parent_device": null, "status": { "label": "Active", "value": 1 }, "primary_ip": null, "primary_ip4": null, "primary_ip6": null, "cluster": null, "virtual_chassis": null, "vc_position": null, "vc_priority": null, "comments": "", "tags": [], "custom_fields": { "vlandomain": null, "dc_vlan_domain": null, "has_fex": false }, "created": "2018-10-08", "last_updated": "2018-11-04T06:40:22.275963Z" } please let me know if you need more information.

dbarrosop commented 5 years ago

The json blob you pasted is not valid, would you mind double checking the parts you removed to make sure you didn't leave sections half-cut?

dbarrosop commented 5 years ago

You can use this to validate your json and inspect it looks as you expect:

https://jsonlint.com/?json=

Thanks!

dbarrosop commented 5 years ago

Ok, no need for that. Taking a quick look at the current code it was easy to spot the issue and was able to reproduce.

Can you test this PR?

https://github.com/nornir-automation/nornir/pull/274

The commit fixing your issue is: https://github.com/nornir-automation/nornir/pull/274/commits/8b9641578d691a3208b9d1a38ff68e2c83d7cfd7

walidamer711 commented 5 years ago

Wow. I have tested with branch "inv_plugins_data" and retrieving the hosts from netbox with custom filed is working perfectly fine as well as the filtering with F filter with nested option.

Thanks

dbarrosop commented 5 years ago

Second beta with support for this has been released.