Netmiko Cisco NXOS Empty Passwords / Default MOTD Issue

[fisherbe]

Description of Issue/Question

We are attempting to authenticate to a Cisco NXOS device (among others). Our Arista connections are working (default passwords set) but the Cisco comes to us with no password set. We tried null string '' which does not work. We tried None. We tried with and without secret and with and without using .enable() and running a command that doesn't require enable. Everything fails on the authentication side though. We initially thought the issue was netmiko being unable to handle a no/null/empty password situation, but now I'm wondering if it's a motd banner issue I need to customize a check for somehow. But, it's a pretty basic default motd banner so I would be surprised if that was the issue.

The bottom few lines from netmiko log we generated:

DEBUG:paramiko.transport:userauth is OK
INFO:paramiko.transport:Auth banner: b'User Access Verification\n'
INFO:paramiko.transport:Authentication (password) failed.

Connecting manually:

$ ssh admin@<redacted>
User Access Verification
Password:

Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (C) 2002-2019, Cisco and/or its affiliates.
All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under their own
licenses, such as open source.  This software is provided "as is," and unless
otherwise stated, there is no warranty, express or implied, including but not
limited to warranties of merchantability and fitness for a particular purpose.
Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or
GNU General Public License (GPL) version 3.0  or the GNU
Lesser General Public License (LGPL) Version 2.1 or
Lesser General Public License (LGPL) Version 2.0.
A copy of each such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://opensource.org/licenses/gpl-3.0.html and
http://www.opensource.org/licenses/lgpl-2.1.php and
http://www.gnu.org/licenses/old-licenses/library.txt.
switch#

Seems it never makes it past 'User Access Verification' which is a default NXOS motd banner AFAIK.

Setup

Netmiko version

netmiko==4.3.0

Netmiko device_type (if relevant to the issue)

cisco_nxos

Steps to Reproduce the Issue

Error Traceback

Traceback (most recent call last):
  File "/path/to/venv/lib/python3.12/site-packages/netmiko/base_connection.py", line 1137, in establish_connection
    self.remote_conn_pre.connect(**ssh_connect_params)
  File "/path/to/venv/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect
    self._auth(
  File "/path/to/venv/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth
    raise saved_exception
  File "/path/to/venv/lib/python3.12/site-packages/paramiko/client.py", line 805, in _auth
    self._transport.auth_password(username, password)
  File "/path/to/venv/lib/python3.12/site-packages/paramiko/transport.py", line 1603, in auth_password
    return self.auth_handler.wait_for_response(my_event)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/venv/lib/python3.12/site-packages/paramiko/auth_handler.py", line 263, in wait_for_response
    raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path/to/project/src/litebrite_backend/command_line_support/test.py", line 16, in <module>
    with ConnectHandler(**kwargs) as conn:
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/venv/lib/python3.12/site-packages/netmiko/ssh_dispatcher.py", line 399, in ConnectHandler
    return ConnectionClass(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/venv/lib/python3.12/site-packages/netmiko/base_connection.py", line 488, in __init__
    self._open()
  File "/path/to/venv/lib/python3.12/site-packages/netmiko/base_connection.py", line 493, in _open
    self.establish_connection()
  File "/path/to/venv/lib/python3.12/site-packages/netmiko/base_connection.py", line 1174, in establish_connection
    raise NetmikoAuthenticationException(msg)
netmiko.exceptions.NetmikoAuthenticationException: Authentication to device failed.

Common causes of this problem are:
1. Invalid username and password
2. Incorrect SSH-key file
3. Connecting to the wrong device

Relevant Python code

Test code used:

import logging
from netmiko import ConnectHandler  # pyright: ignore[reportMissingTypeStubs]

logging.basicConfig(filename='test.log', level=logging.DEBUG)

kwargs = {
    'device_type': 'cisco_nxos',
    'host': '<redacted>',
    'username': 'admin',
    'password': '',
    'secret': ''
}

with ConnectHandler(**kwargs) as conn:
    conn.enable()
    conn.send_command('show version')

I probably don't need secret / .enable() here for 'show version' but will need it for some other stuff in the future. Just testing.

[fisherbe]

This might be the same issue as #3424 which looks identical but is for Cisco IOS.

[FriendlyGecko]

You can try the fix made in #3425 for the meantime.

[fisherbe]

You can try the fix made in #3425 for the meantime.

@FriendlyGecko

We have to use SSH, not sure if your telnet focused solution will help or not?

[FriendlyGecko]

If you look through the code, the telnet function is fed into SSH functions. I only use SSH for networking, but since it inherits they inherit the telnet functions, I placed it there where it can benefit both. Fun fact: serial connections also inherit from telnet, so the name telnet is a little misleading.

[ktbyers]

@FriendlyGecko Your statement is not correct--telnet_login() is not used by SSH. It is shared for "serial" logins (as both telnet and serial have an interactive login). Your earlier problem was using "serial" which is why you saw that "telnet_login" code get called.

telnet_login is irrelevant/not used for any of the SSH code.

[ktbyers]

@fisherbe Can you show what your motd looks like? You can obscure any company name or other company identifying information.

[FriendlyGecko]

It has been a minute since I messed with the code for that merge request, but I don't believe that I changed anything except for the telnet code. I use that code now and it works for SSH connections for me.

[ktbyers]

@FriendlyGecko Just double checked Netmiko source code. telnet_login is never used with SSH; it is only used for telnet and also for serial_login. Your original issue with stack trace was a failed serial login.

[FriendlyGecko]

@ktbyers Oh ok, I don't recall how I fixed it for SSH off the top of my head. So sorry I couldn't help more.

[ktbyers]

@FriendlyGecko No worries at all...I just didn't see how the telnet_login fix would help with an SSH issue.

[FriendlyGecko]

I honestly probably just pasted that same code wherever the SSH authentication section is, but I don't have time to check/try it out right now. Something I can do in a couple weeks when I get back from vacation.

[ktbyers]

@FriendlyGecko SSH generally doesn't do an interactive authentication (so there is no equivalent code for SSH). There might be some special case exceptions for some platforms that do non-standard things for logging in.

[fisherbe]

@ktbyers ,

Default motd: "User Access Verification"

[fisherbe]

@ktbyers ,

If it perhaps helps, I know I mentioned above we have an empty password set for login. Also, no secret password set at all. But if we don't use .enable(), don't pass secret, and use a basic command not requiring enable, it still fails.

DEBUG:paramiko.transport:userauth is OK
INFO:paramiko.transport:Auth banner: b'User Access Verification\n'
INFO:paramiko.transport:Authentication (password) failed.

Maybe it's not the banner as the issue? Maybe it just doesn't correctly handle empty passwords? Or if no secret is set it somehow fails even though not using .enable()?

Not entirely sure, though I can log into the device via ssh manually just fine.

[ktbyers]

Yeah, it pretty clearly is failing authentication.

Can you show what it looks like when you manually SSH to the device? Just post the CLI interaction here?

[ktbyers]

@fisherbe Never mind...you posted it above:

$ ssh admin@<redacted>
User Access Verification
Password:

So you just hit here and you login?

[fisherbe]

@fisherbe Never mind...you posted it above:

$ ssh admin@<redacted>
User Access Verification
Password:

So you just hit here and you login?

Yes, just hit enter there and you login with prompt of <hostname>#

There's some extra text that comes back from the switch after you authenticate and before you get the prompt. Just the basic info about the switch that every NXOS gives you afaik.

[fisherbe]

@ktbyers ,

I've confirmed that setting a password for the user makes the issue go away. Unfortunately we have some requirements to allow blank passwords in the environment since it's bare metal pre-deploy automation stuff. But it works for now. Hopefully the issue can be resolved. I will attempt to figure it out also here and there as I have time.

[ktbyers]

@fisherbe Issue is (probably) this:

https://stackoverflow.com/questions/71749222/paramiko-authentication-to-server-with-no-password-fails

[fisherbe]

@fisherbe Issue is (probably) this:

https://stackoverflow.com/questions/71749222/paramiko-authentication-to-server-with-no-password-fails

@ktbyers

So what I'm reading this as is netmiko doesn't support no password methods directly, you have to use paramiko on your own essentially, etc?

[ktbyers]

@fisherbe You could probably make a custom driver and add support in. If you look at HP ProCurve you can see the code pattern for supporting the noauth.

See here:

https://github.com/ktbyers/netmiko/blob/develop/netmiko/hp/hp_procurve.py#L189-L190

So you would basically create a new driver and a new class, the class would inherit from the standard Cisco NXOS class and then have that _build_ssh_client() code. You would also probably need a special session_preparation at least to send a null password and to verify you were logged in (once again you can look at that HP ProCurve driver as a reference).

The issue basically boils down to that no-password is its own special case to Paramiko and Netmiko doesn't have easy ways to switch over to this (especially in cases where it is an atypical pattern for given platform).

[fisherbe]

@ktbyers thanks for the responses!