search

nautobot / nautobot-app-device-onboarding

Device Onboarding Plugin for Nautobot
https://docs.nautobot.com/projects/device-onboarding/en/latest/
Other
44 stars 31 forks source link

"Related Field got invalid lookup: address" error during device onboarding #9

Closed tim-fiola closed 3 years ago

tim-fiola commented 3 years ago

Environment

Steps to Reproduce

  1. Spin up instance of dev/test server
  2. Install Nautobot 1.0.0b4
  3. Import database or manually create a site
  4. Install device onboarding plugin 1.0.0
  5. Plugins --> Onboarding Tasks --> Add (+)
  6. Fill out form
  7. Click on Create button

Expected Behavior

Device to be successfully onboarded

Observed Behavior

Failure due to Related Field got invalid lookup: address

Log snips

==> /var/log/nautobot/nautobot.log <==
2021-04-20 22:27:44 DEBUG Form validation was successful

==> /var/log/nautobot/nautobot_worker.log <==
2021-04-20 22:27:44 INFO default: nautobot_device_onboarding.worker.onboard_device(UUID('a37e00aa-dc61-4a33-8ba6-19f77aab6659'), *Credentials argument hidden*) (69ddd347-3452-4ae2-b502-c3551b314764)

==> /var/log/nautobot/nautobot.log <==
2021-04-20 22:27:44 INFO Created onboarding task local : csr2 (PK: a37e00aa-dc61-4a33-8ba6-19f77aab6659)

==> /var/log/nautobot/nautobot_worker.log <==
2021-04-20 22:27:45 INFO START: onboard device
2021-04-20 22:27:45 ERROR Onboarding Error - Exception
2021-04-20 22:27:45 ERROR Related Field got invalid lookup: address
2021-04-20 22:27:45 INFO default: Job OK (69ddd347-3452-4ae2-b502-c3551b314764)
2021-04-20 22:27:45 INFO Result is kept for 500 seconds
bsmeding commented 3 years ago

same issue here, hopefully #8 will solve this

bsmeding commented 3 years ago

Tryed new version pip install git+https://github.com/nautobot/nautobot-plugin-device-onboarding.git@develop

But still have the same error

mzbroch commented 3 years ago

@bsmeding Have you restarted your worker?

If you did, can you provide more information? Is it re-onboarding, first onboarding, what kind of objects already exist in your database etc?

bsmeding commented 3 years ago

hi @mzbroch yes i restarted both services

It happen with both, existing and non-existing devices. Also directly after starting the task. database is rather empty, only one device manual added

Kani999 commented 3 years ago

@mzbroch - I can confirm that onboarding devices end up with the same error for me. Maybe the issue should be reopened?

mzbroch commented 3 years ago

@Kani999 @bsmeding

Please install plugin release 1.0.1 via PyPi: 

pip install pip install 'nautobot-device-onboarding==1.0.1'
Kani999 commented 3 years ago

@mzbroch - It works now.

mzbroch commented 3 years ago

@Kani999 We see this error from time to time - it's related to underlying library and the device type itself. You might need to tweak SSH settings on XR, plus see of following plugin settings in configuration.py help :

NAPALM_ARGS = {
    "conn_timeout": 30,
    "session_timeout": 60,
    "auth_timeout": 40,
    "banner_timeout": 40,
}
Kani999 commented 3 years ago

@mzbroch I tried this args and even others with different timeouts, but I always end up with the error

==> nautobot-worker.log <==
09:23:45 default: nautobot_device_onboarding.worker.onboard_device(UUID('a2ab8fd1-9952-4bf6-a5ea-ea2ed2843bfd'), *Credentials argument hidden*) (e9cfb421-a677-4554-b377-e0f339bfbe66)
09:23:45 START: onboard device
09:23:45 Getting device with IP lookup failed: Device matching query does not exist.
09:23:45 CHECK: IP 195.xxx.xxx.xxx:22
09:23:45 COLLECT: device information 195.xxx.xxx.xxx

==> nautobot-worker-error.log <==
09:23:45 OnboardException: fail-general: Error reading SSH protocol banner
mzbroch commented 3 years ago

Please first try to create a new platform object in netbox for IOSXR (define napalm driver), and select this platform manually during onboarding. If this doesn't help, within this environment's context, spawn-up napalm session manually to this device and check connectivity.

Kani999 commented 3 years ago

root@faadb8a8a6f8:/opt/nautobot# ./bin/napalm --user 'kani999'--password 'password' --vendor iosxr --debug IPADDRESS call get_facts

2021-04-28 05:31:46,863 - napalm - DEBUG - Starting napalm's debugging tool
2021-04-28 05:31:46,863 - napalm - DEBUG - Gathering napalm packages
2021-04-28 05:31:46,863 - napalm - DEBUG - napalm==3.2.0
2021-04-28 05:31:46,863 - napalm - DEBUG - get_network_driver - Calling with args: ('iosxr',), {}
2021-04-28 05:31:46,863 - napalm - DEBUG - get_network_driver - Successful
2021-04-28 05:31:46,864 - napalm.iosxr.iosxr - INFO - {}
2021-04-28 05:31:46,864 - napalm - DEBUG - __init__ - Successful
2021-04-28 05:31:46,864 - napalm - DEBUG - pre_connection_tests - Calling with args: (<napalm.iosxr.iosxr.IOSXRDriver object at 0x7f8df51333d0>,), {}
2021-04-28 05:31:46,864 - napalm - DEBUG - open - Calling with args: (<napalm.iosxr.iosxr.IOSXRDriver object at 0x7f8df51333d0>,), {}
2021-04-28 05:31:58,191 - napalm - DEBUG - open - Successful
2021-04-28 05:31:58,191 - napalm - DEBUG - connection_tests - Calling with args: (<napalm.iosxr.iosxr.IOSXRDriver object at 0x7f8df51333d0>,), {}
2021-04-28 05:31:58,192 - napalm - DEBUG - get_facts - Calling with args: (<napalm.iosxr.iosxr.IOSXRDriver object at 0x7f8df51333d0>,), {}
2021-04-28 05:32:03,052 - napalm.pyIOSXR.iosxr - DEBUG - <Element Response at 0x7f8df5203c80>
2021-04-28 05:32:04,888 - napalm.pyIOSXR.iosxr - DEBUG - <Element Response at 0x7f8df52359c0>
2021-04-28 05:32:04,895 - napalm - DEBUG - get_facts - Successful
2021-04-28 05:32:04,895 - napalm - DEBUG - method - Calling with args: (<napalm.iosxr.iosxr.IOSXRDriver object at 0x7f8df51333d0>, 'get_facts'), {}
2021-04-28 05:32:04,895 - napalm - DEBUG - get_facts - Attempting to resolve method
2021-04-28 05:32:04,895 - napalm - DEBUG - get_facts - Attempting to call method with kwargs: {}
2021-04-28 05:32:09,959 - napalm.pyIOSXR.iosxr - DEBUG - <Element Response at 0x7f8df52353c0>
2021-04-28 05:32:11,405 - napalm.pyIOSXR.iosxr - DEBUG - <Element Response at 0x7f8df5259940>
2021-04-28 05:32:11,411 - napalm - DEBUG - get_facts - Response
{
    "vendor": "Cisco",
    "os_version": "6.6.3",
    "hostname": "R-lab",
    "serial_number": "SMA18040SW7",
    "model": "CRS-4-CH",
    "interface_list": [
        "GigabitEthernet0/1/0/0",
    ]
}
2021-04-28 05:32:11,412 - napalm - DEBUG - method - Successful
2021-04-28 05:32:11,412 - napalm - DEBUG - close - Calling with args: (<napalm.iosxr.iosxr.IOSXRDriver object at 0x7f8df51333d0>,), {}
2021-04-28 05:32:11,412 - napalm.iosxr.iosxr - DEBUG - Closed connection with device 
2021-04-28 05:32:11,412 - napalm - DEBUG - close - Successful
2021-04-28 05:32:11,412 - napalm - DEBUG - post_connection_tests - Calling with args: (<napalm.iosxr.iosxr.IOSXRDriver object at 0x7f8df51333d0>,), {}
2021-04-28 05:32:11,412 - napalm.iosxr.iosxr - DEBUG - Closed connection with device

==> nautobot-worker-error.log <== 10:36:15 ERROR: Error reading SSH protocol banner 10:36:15 OnboardException: fail-general: OnboardException: fail-connect: ERROR: Error reading SSH protocol banner

==> nautobot-worker.log <== 10:36:15 START: onboard device 10:36:15 Getting device with IP lookup failed: Device matching query does not exist. 10:36:15 CHECK: IP IPADDR:22 10:36:15 COLLECT: device information IPADDR 10:36:15 INFO OPTIONAL ARGS: {'conn_timeout': 3000, 'session_timeout': 3000, 'auth_timeout': 3000, 'banner_timeout': 3000} 10:36:15 INFO guessing device type: IPADDR 10:36:15 default: Job OK (b0229a46-c6d1-43d6-b74d-f4cf4f1069ce) 10:36:15 Result is kept for 500 seconds


- and it does not matter if I create the platform or not (and assign it when onboarding)

![image](https://user-images.githubusercontent.com/17337078/116391001-ec303680-a81e-11eb-94aa-a1084cd8d9d1.png)

--
Kani999 commented 3 years ago

I figure out, that there is a point, where my code crashes to exception napalm_device.open()

or with SSHDetect(**remote_device) when the platform is not set manually

bsmeding commented 3 years ago

pip install pip install 'nautobot-device-onboarding==1.0.1'

@mzbroch installed this version instead of development and onboarding still works! Thanks

bsmeding commented 3 years ago

@Kani999 can you add a device manually and check if the NAPALM is working from within the device page? Get the current config.

Kani999 commented 3 years ago

@bsmeding 

==> nautobot-app-error.log <==
[29/Apr/2021 09:49:50] "GET /dcim/devices/d2ad7d2c-9c16-4db1-97c6-97a429b54fbe/config/ HTTP/1.1" 200 55477
{'conn_timeout': 3000, 'auth_timeout': 3000, 'banner_timeout': 3000, 'session_timeout': 3000}
Local version/idstring: SSH-2.0-paramiko_2.7.2
Remote version/idstring: SSH-2.0-Cisco-2.0
Connected (version 2.0, client Cisco-2.0)
.
.
.
Adding ecdsa-sha2-nistp256 host key for XXX.XXX.XXX.XX:
userauth is OK
Authentication (password) successful!

XXX:R199-lab#
Thu Apr 29 11:50:01.325 METDST
XML>
read_channel:
write_channel: b'<?xml version="1.0" encoding="UTF-8"?><Request MajorVersion="1" MinorVersion="0"><CLI><Configuration>show running-config</Configuration></CLI></Request>\n'
read_channel: <?xml version="1.0" encoding="UTF-8"?>
<Response MajorVersion="1" MinorVersion="0"><CLI><Configuration>
Building configuration...
!! IOS XR Configuration 6.6.3
!! Last configuration change at Tue Apr 27 11:31:59 2021 by username
!
hostname R199-lab
.
.
.
</Configuration></CLI><ResultSummary ErrorCount="0"/></Response>
XML>
Closed connection with device XXX.XXX.XXX.XX
[chan 0] EOF sent (0)
Closed connection with device XXX.XXX.XXX.XX
[29/Apr/2021 09:50:05] "GET /api/dcim/devices/d2ad7d2c-9c16-4db1-97c6-97a429b54fbe/napalm/?method=get_config HTTP/1.1" 200 75707
[chan 0] EOF received (0)
EOF in transport thread
Kani999 commented 3 years ago

Increasing rate-limit on the device solved the problem. Yet another problem occurs: