nahun commented 3 years ago

This PR is a little big (sorry about that), but I'll summarize the changes.

Uses OPTIONS to lookup choices. Requires https://github.com/scaleway/python-netboxapi/pull/9 to be merged.
Ignores IPv6 link local address which is anything in fe80::/10.
Will only send a PUT request for interfaces and devices when something is going to change.
Get all interfaces on a device first and create a map to lookup later. This allows just one bigger request vs many small ones.
Does not change the interface type to Other of an existing interface unless overwrite is specified.
Fixes IP assignment to the newer model with later NetBox versions (assigned_object_id and assigned_object_type).
QSFP interface type fixes.
Fixes VLAN tagged list processing.
Adds Loopback interface as a Virtual type.
Adds detailed LLDP neighbors method to Cisco IOS.

I haven't really looked at the testing. Mostly tested on newer Cisco Cat9Ks.

Fixes #39

nahun commented 3 years ago

I'm not sure if my QSFP changes are correct, but it seems good for Cisco IOS at least. I know the testing is failing for Cisco Nexus though. Probably need to add some testing for QSFP28 100G interfaces.

aruhier commented 3 years ago

Hey, Thanks a lot for this PR! I'm not working at Scaleway anymore (that's why I transferred them the ownership of these 2 repos), but I can still do some review.

It'll be however complicated for me to test it, but on the other hand, this tool doesn't work with the recent versions of Netbox, and that for a while, so I won't be picky. If anyone at @scaleway wants to test it, it would help, but otherwise I'll merge if things look ok.

Anyway, I already had a quick look, I'm going to do a review tomorrow. Overall it looks good! There's one thing I thought about the Options. Maybe it can be integrated into the mappers a bit better (in netbox-api) so they handle the caching, and you wouldn't have to bother caring about it here. But your current solution is good enough and it's something that can be added later if there's some interests on it.

i4networks commented 3 years ago

Hi,

ERROR: netbox_importer: Error when polling device xx-xx--xx: Unexpected format in the OPTIONS response at dcim/

How can we help out?

Best,

I4Networks

aruhier commented 3 years ago

@i4networks: thanks for the testing. Can you run it again in debug mode so you could paste the OPTIONS request?

i4networks commented 3 years ago

@aruhier what parameter do u use for debug?

aruhier commented 3 years ago

--verbose debug should do the trick. It'll be very verbose but normally requests should print you every http requests it's doing.

i4networks commented 3 years ago

@aruhier ...... switch1# DEBUG: netbox_importer: Interface VRF_0_vlan1015 has IP but was not listed DEBUG: urllib3.connectionpool: Starting new HTTP connection (1): localhost:8001 DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/devices/?name=switch1&limit=50 HTTP/1.1" 200 1772 DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/interfaces/?device_id=31&limit=50 HTTP/1.1" 200 43113 DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/interfaces/?device_id=31&limit=50&offset=50 HTTP/1.1" 200 42974 DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/interfaces/?device_id=31&limit=50&offset=100 HTTP/1.1" 200 42927 DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/interfaces/?device_id=31&limit=50&offset=150 HTTP/1.1" 200 41119 DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/interfaces/?device_id=31&limit=50&offset=200 HTTP/1.1" 200 40691 DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/interfaces/?device_id=31&limit=50&offset=250 HTTP/1.1" 200 40749 DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost DEBUG: urllib3.connectionpool: http://localhost:8001 "GET /api/dcim/interfaces/?device_id=31&limit=50&offset=300 HTTP/1.1" 200 8231 DEBUG: urllib3.connectionpool: Resetting dropped connection: localhost DEBUG: urllib3.connectionpool: http://localhost:8001 "OPTIONS /api/dcim/ HTTP/1.1" 200 181 DEBUG: netmiko: write_channel: b'\n' DEBUG: netmiko: read_channel: i4n-cr1-ahm-dp# DEBUG: netmiko: read_channel: DEBUG: netmiko: read_channel: DEBUG: netmiko: write_channel: b'exit\n' ERROR: netbox_importer: Error when polling device switch1: Unexpected format in the OPTIONS response at dcim/ 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:57<00:00, 57.62s/it] [root@nms netbox-netprod-importer]#

aruhier commented 3 years ago

I don't get why it tries to do a query on api/dcim. @nahun can you investigate this issue? I will also try to do the same later on my side.

nahun commented 3 years ago

Interesting, I can't find any reason that would be happening.

@i4networks, can you post more information like your config.yml and devices.yml?

I thought maybe it was due to not using the latest changes in netboxapi, but it would throw a straight python exception about no options() method in the mapper if that was the case.

The only call to options() is in the _NetboxPusher.search_value_in_choices() method and it seems fine. Doesn't seem possible that it wouldn't use the model correctly for the route in netboxapi.

aruhier commented 3 years ago

@i4networks: Any news about the bug you encountered?

i4networks commented 3 years ago

Hi @aruhier ,

Sorry for the late response. I tried both Cisco and Juniper MX with a fairly simple yaml.

test.yml netboxhostname: driver: ios

config.yml cat config.yml ########################

Global options

########################

Be more verbose

verbose: None

Disable ssl warnings in urllib3

disable_ssl_warnings: False

################

Netbox

################

netbox:

Netbox API URL

url: "http://localhost:8001/api"

username: "user"

password: "password"

or to use a token instead

token: "letsnotleakthetokenongithub"

##########################

Interconnections

##########################

On some devices, LLDP will expose the host FQDN. If devices are stored on

Netbox only by their hostname, the interconnection process will not be able

to find them. Fill this list to strip the domain name from exposed names.

remove_domains:

"foo.tld"
"bar.tld"

vim: set ts=2 sw=2:

netbox version v2.11.4 Python 2.7.5

nahun commented 3 years ago

@i4networks, it happens on both Cisco and Juniper devices if you try one at a time? I don't have a Juniper device to test with unfortunately.

i4networks commented 3 years ago

@nahun i tested with only one device at the time in the test.yml

nahun commented 3 years ago

I can't reproduce that error. You'll need to be very specific on how you're able to reproduce.

Versions of everything like Python & Python packages (use pip freeze). Confirm using the latest netboxapi from the master branch.

Config and device files

Model of each device and what version they're running.

Provide anything else that might be useful.

aruhier commented 3 years ago

Lets not block this PR for too long, now this project is not working anymore for a lot of people anyway. I'm doing the review right now.

VictorPavlushin commented 2 years ago

Hi,

ERROR: netbox_importer: Error when polling device xx-xx--xx: Unexpected format in the OPTIONS response at dcim/

How can we help out?

Best,

I4Networks

This is due to the fact that the latest changes to the library https://github.com/scaleway/python-netboxapi are not published in the pip repository. Delete the system one and install it from github.

i4networks commented 2 years ago

Hi, ERROR: netbox_importer: Error when polling device xx-xx--xx: Unexpected format in the OPTIONS response at dcim/ How can we help out? Best, I4Networks

This is due to the fact that the latest changes to the library https://github.com/scaleway/python-netboxapi are not published in the pip repository. Delete the system one and install it from github.

pip uninstall netboxapi python -m pip install git+https://github.com/scaleway/python-netboxapi.git

fixed the issue.

brianpr commented 2 years ago

I have the same problem

pip install git+https://github.com/scaleway/python-netboxapi.git

it didn't work for me, now it tells me:

ERROR: netbox_importer: Error when polling device xxx.xxx.xxx: 404 Client Error: Not Found for url: https://test-domain.com/api/dcim/_choices/?limit=50 -

any ideas?

VictorPavlushin commented 2 years ago

I have the same problem

pip install git+https://github.com/scaleway/python-netboxapi.git

it didn't work for me, now it tells me:

ERROR: netbox_importer: Error when polling device xxx.xxx.xxx: 404 Client Error: Not Found for url: https://test-domain.com/api/dcim/_choices/?limit=50 -

any ideas?

use this branch with commits: https://github.com/nahun/netbox-netprod-importer/pull/1

appleoddity commented 1 year ago

So, what happened here? Why did these fixes not get merged? This project is broken and dead. Worthless to anyone.

@nahun How do I pull this fixed code base, it looks like you deleted it?

scaleway / netbox-netprod-importer

Various Fixes and Updates for newer NetBox Versions #56

Global options

Be more verbose

Disable ssl warnings in urllib3

Netbox

Netbox API URL

username: "user"

password: "password"

or to use a token instead

Interconnections

On some devices, LLDP will expose the host FQDN. If devices are stored on

Netbox only by their hostname, the interconnection process will not be able

to find them. Fill this list to strip the domain name from exposed names.

vim: set ts=2 sw=2: