Unable to onboard devices with 4.0.0rc1

[mdeng10]

Environment

• Python version: 3.11.9
• Nautobot version: 2.2.5
• nautobot-device-onboarding version: 4.0.0rc1

This is when running the "Sync Devices From Network" workflow I expected the device to be onboarded (created as dcim | device) with its primary IP, interface/device type, etc filled out

Instead i received this error message (seems to hint at serial number not existing?) - i logged into the device and can confirm that show version does include the serial number

Steps to Reproduce

[scetron]

Thanks for opening the issue. Would you share the device os and version? And perhaps the version string from the command?

The jpath for your particular version may be off... Here's a bit more info on that: https://github.com/nautobot/nautobot-app-device-onboarding/blob/v4.0.0-rc.1/docs/user/app_yaml_overrides.md

[mdeng10]

this is the output of show version with specifics redacted/sanitised

Cisco IOS XE Software, Version 16.12.02
Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.12.02, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Tue 19-Nov-19 10:04 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2019 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.  For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 16.12.2r, RELEASE SOFTWARE (P)

device_name uptime is 2 years, 9 weeks, 2 days, 1 hour, 0 minutes
Uptime for this control processor is 2 years, 9 weeks, 2 days, 1 hour, 1 minute
System returned to ROM by PowerOn at 15:15:39 AEDST Fri Feb 7 2020
System restarted at 11:32:52 AEST Mon Apr 18 2022
System image file is "flash:packages.conf"
Last reload reason: PowerOn

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information: 

------------------------------------------------------------------------------
Technology-package                                     Technology-package
Current                        Type                       Next reboot  
------------------------------------------------------------------------------
network-advantage       Smart License                    network-advantage   
dna-advantage           Subscription Smart License       dna-advantage                 
AIR License Level: AIR DNA Advantage
Next reload AIR license Level: AIR DNA Advantage

Smart Licensing Status: UNREGISTERED/EVAL EXPIRED

cisco C9300-48T (X86) processor with 1343980K/6147K bytes of memory.
Processor board ID FOC23400000
11 Virtual Ethernet interfaces
52 Gigabit Ethernet interfaces
8 Ten Gigabit Ethernet interfaces
2 TwentyFive Gigabit Ethernet interfaces
2 Forty Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
8388608K bytes of physical memory.
1638400K bytes of Crash Files at crashinfo:.
11264000K bytes of Flash at flash:.
0K bytes of WebUI ODM Files at webui:.

Base Ethernet MAC Address          : AA:BB:CC:DD:EE:FF
Motherboard Assembly Number        : 73-00000-05
Motherboard Serial Number          : FOC23400000
Model Revision Number              : A0
Motherboard Revision Number        : A0
Model Number                       : C9300-48T
System Serial Number               : FOC23400000

Switch Ports Model              SW Version        SW Image              Mode   
------ ----- -----              ----------        ----------            ----   
*    1 65    C9300-48T          16.12.02          CAT9K_IOSXE           INSTALL

Configuration register is 0x102

i will try update jpath and see if it fixes anything but might not get to it until next week

[mdeng10]

I noticed i'm also unable to use the other job Sync Network Data From Network

Schema validation failed for test_device_a Error: [] is not of type 'string'

Failed validating 'type' in schema['properties']['serial']: {'description': 'Serial number of the network device', 'minItems': 1, 'type': 'string'}

On instance['serial']: []

test_device_a : Connection or data error, this device will not be synced. Schema validation failed.

the serial number is set in nautobot - so i'm guessing it's not reading the SN properly from the device

[scetron]

I think that makes sense as they are the same schema/pattern. The Sync Network Data From Network will compare what's in Nautobot to what's collected from the device, and this is failing where it's getting collected from the device.

[scetron]

To help troubleshoot this issue, we have a hidden job that collects the data the helps us with testing. Would you be able go in and enable the hidden job (and unhide, if you like) "Runs Commands on a Device to simulate SSoT Command Getter", sanitize the job output and add it here or a gist?

[susanhooks]

Would you also be able to share which version of ntc-templates is installed?

[mdeng10]

Would you also be able to share which version of ntc-templates is installed?

ntc-templates                       5.0.0                       

I will try update to 5.1.0 and then if still not working i'll test the hidden job mentioned above

[mdeng10]

So updating ntc-templates didn't do anything - i then ran Runs Commands on a Device to simulate SSoT Command Getter and it generated a file with the following info

{"show version": [], "show interfaces": []}

which was a bit weird, so i changed the platform manually (the device is IOSXE, and the cisco_xe driver was being automatically used) to use cisco_ios, and then it seems to be working, i guess the issue is in the cisco_xe driver?

does this job only work for cisco ios devices? most if not all nxos/iosxe devices are failing

[jeffkala]

Did you get this figured out @mdeng10 ? Most likely you just need to force your xe platform to use the ntc-templates of ios. This can be done in the admin panel by overloading the defaults. Something like this.... specifically the ntc_templates key.

[mdeng10]

i didn't figure it out - but will try your suggestion today and update you

[mdeng10]

looks like that worked for iosxe devices - i'm still unable to run the Job Result: Sync Network Data From Network job though

i get the same error as mentioned in https://github.com/nautobot/nautobot-app-device-onboarding/issues/190#issuecomment-2190855028

[scetron]

Hello @mdeng10 , if you can try again with the latest release and still if you are having the issue? We have fixed similar issues for others.

[mdeng10]

it's looking much better - it failed once on a pair of devices, but for some reason running it again had no issues - will chalk it up to user error

thanks for the update

[scetron]

Hello @mdeng10 I saw one more message, was that still an issue on 4.0?

[mdeng10]

i think the error was due to connectivity with the device - i'm hitting another error which i'll open in another issue