Authentication failed despite correct password

ntaken commented 3 years ago

Happens to different HP switches seemingly at random. In this specific case a old 2915-8G-PoE with HP ProCurve A.15.16.0023. Always works fine with different ssh clients. Autentication uses tacacs but I think I remember having the same trouble with local accounts.

Connection settings:

    net_connect = ConnectHandler(device_type=device_type, host=ip_address, username=username, password=password, conn_timeout=60, auth_timeout=60)

Error message:

172.20.100.111 = hp_procurve

Authentication to device failed.

Common causes of this problem are:

Invalid username and password

Incorrect SSH-key file

Connecting to the wrong device

Device settings: hp_procurve 172.20.100.111:22

Authentication failed.

Execution time: 0:00:06.068738

Debug:

DEBUG:paramiko.transport:starting thread (client mode): 0x76dde070 DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.7.2 DEBUG:paramiko.transport:Remote version/idstring: SSH-2.0-Mocana SSH 5.8 INFO:paramiko.transport:Connected (version 2.0, client Mocana) DEBUG:paramiko.transport:kex algos:['diffie-hellman-group14-sha1'] server key:['ssh-rsa'] client encrypt:['aes256-ctr', 'aes256-cbc', 'rijndael-cbc@lysator.liu.se', 'aes192-ctr', 'aes192-cbc', 'aes128-ctr', '>DEBUG:paramiko.transport:Kex agreed: diffie-hellman-group14-sha1 DEBUG:paramiko.transport:HostKey agreed: ssh-rsa DEBUG:paramiko.transport:Cipher agreed: aes128-ctr DEBUG:paramiko.transport:MAC agreed: hmac-sha1 DEBUG:paramiko.transport:Compression agreed: none DEBUG:paramiko.transport:kex engine KexGroup14 specified hash_algo DEBUG:paramiko.transport:Switch to new keys ... DEBUG:paramiko.transport:Adding ssh-rsa host key for 172.20.100.111: b'e06d255e7b1cb191c2e09e7e23651654' DEBUG:paramiko.transport:userauth is OK INFO:paramiko.transport:Auth banner: b"We'd like to keep you up to date about:\n Software feature updates\n New product announcements\n * Special events\nPlease register your products now at: www.hp.c>INFO:paramiko.transport:Authentication (password) failed.

ktbyers commented 3 years ago

@ntaken Can you try the HP ProCurve code in Netmiko develop branch. I recently did a lot of work improving the ProCurve driver.

Regards, Kirk

ntaken commented 3 years ago

@ntaken Can you try the HP ProCurve code in Netmiko develop branch. I recently did a lot of work improving the ProCurve driver.

Regards, Kirk

I removed the old and installed 'pipenv install -e git+https://github.com/ktbyers/netmiko.git@develop#egg=netmiko' to my virtalenv.

Much faster execution time 2-4 sec instead of 10 s on successful retrival. I did however have a new failure to authenticate.

172.20.100.144 = hp_procurve

Authentication to device failed.

Common causes of this problem are:

172.20.100.144 = hp_procurve

Authentication to device failed.

Common causes of this problem are:

Invalid username and password

Incorrect SSH-key file

Connecting to the wrong device

Device settings: hp_procurve 172.20.100.144:22

Authentication failed. Execution time: 0:00:09.917785

Debug:

DEBUG:paramiko.transport:starting thread (client mode): 0xca72c640 DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.8.0 DEBUG:paramiko.transport:Remote version/idstring: SSH-2.0-Mocana SSH 6.3 INFO:paramiko.transport:Connected (version 2.0, client Mocana) DEBUG:paramiko.transport:kex algos:['diffie-hellman-group14-sha1'] server key:['ssh-rsa'] client encrypt:['aes256-ctr', 'aes256-cbc', 'rijndael-cbc@lysator.liu.se', 'aes192-ctr', 'aes192-cbc', 'aes128-ctr', 'aes128-cbc', '3des-cbc'] server encrypt:['aes256-ctr', 'aes256-cbc', 'rijndael-cbc@lysator.liu.se', 'aes192-ctr', 'aes192-cbc', 'aes128-ctr', 'aes128-cbc', '3des-cbc'] client mac:['hmac-sha1-96', 'hmac-md5', 'hmac-sha1', 'hmac-md5-96'] server mac:['hmac-sha1-96', 'hmac-md5', 'hmac-sha1', 'hmac-md5-96'] client compress:['none'] server compress:['none'] client lang:[''] server lang:[''] kex follows?False DEBUG:paramiko.transport:Kex agreed: diffie-hellman-group14-sha1 DEBUG:paramiko.transport:HostKey agreed: ssh-rsa DEBUG:paramiko.transport:Cipher agreed: aes128-ctr DEBUG:paramiko.transport:MAC agreed: hmac-sha1 DEBUG:paramiko.transport:Compression agreed: none DEBUG:paramiko.transport:EOF in transport thread DEBUG:paramiko.transport:kex engine KexGroup14 specified hash_algo DEBUG:paramiko.transport:Switch to new keys ... DEBUG:paramiko.transport:Adding ssh-rsa host key for 172.20.100.144: b'8fdd152cc1bf885354a3799cecd62529' DEBUG:paramiko.transport:userauth is OK INFO:paramiko.transport:Auth banner: b"We'd like to keep you up to date about:\n Software feature updates\n New product announcements\n * Special events\nPlease register your products now at: www.hpe.com/networking/register\n\n\n" INFO:paramiko.transport:Authentication (password) failed.

This time the failing switch was a 2920-24G-PoE+, HP ProCurve WB.16.04.0009.
Note i do my testing with a dict of dicts of our 120 HP switches. One at a time, always the same order. Always a few, as far as I can see randomly fails.
I look at the messages and when the script reports a failure I immediately test with a ssh client. I also recently moved script execution to a server with very little use of available resurces
The ISE (tacacs) interestingly show a delay from granting authentication to granting authorization. About 6-7 sec. Ordinary for a success the difference is about 0.2 sec. Is it here the problem lies? I might misremember about earlier attempts with local accounts. It might also mean that the problem might not be limited to procurve.

ktbyers commented 3 years ago

I suspect it is ISE (tacacs) that is causing the failure. 6 to 7 second delay on authentication/authorization is really slow.

You could try increasing the auth_timeout argument to 20 or 30 seconds and see if the problem goes away.

ntaken commented 3 years ago

I suspect it is ISE (tacacs) that is causing the failure. 6 to 7 second delay on authentication/authorization is really slow.

You could try increasing the auth_timeout argument to 20 or 30 seconds and see if the problem goes away.

But do I not already have the auth_timeout at 60s? Should I put it somewhere else?

net_connect = ConnectHandler(device_type=device_type, host=ip_address, username=username, password=password, conn_timeout=60, auth_timeout=60)

ktbyers commented 3 years ago

Missed that...

Can you test with a local account and see if it works in that case?

AAA is one of the most frequent things that breaks automation.

Kirk

ntaken commented 3 years ago

Tried removing the switches temporary from TACACS and yes it worked great. Don't think I will be allowed too keep it that way though or necessarily want to myself.

It is to bad that none of the timeouts seems to truly catch the AAA delay.

ktbyers commented 3 years ago

@ntaken Can you show me the full exception stack trace on a failure (using the develop branch)?

ntaken commented 3 years ago

Not sure what a full stack trace is. Is it more than the traceback immediately before? Do I use a debugger?

BTW I am using visual studio code when scripting. I will look into it and collect what I can find before the end of the week. Very much appreciated you taking your time looking into this.

Den ons 20 okt. 2021 kl 18:05 skrev Kirk Byers @.***>:

@ntaken https://github.com/ntaken Can you show me the full stack trace on a failure?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ktbyers/netmiko/issues/2532#issuecomment-947815147, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARARCNSJ5TDU63T4JTSWCWDUH3SGBANCNFSM5FZ7AKAA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

ktbyers commented 3 years ago

No just the full exception stack trace when you run your code.

Make sure you remove any try/except statements from your code so you don't hide the error, however.

ntaken commented 3 years ago

Traceback (most recent call last):
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/netmiko/base_connection.py", line 935, in establish_connection
    self.remote_conn_pre.connect(**ssh_connect_params)
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/paramiko/client.py", line 435, in connect
    self._auth(
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/paramiko/client.py", line 764, in _auth
    raise saved_exception
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/paramiko/client.py", line 751, in _auth
    self._transport.auth_password(username, password)
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/paramiko/transport.py", line 1509, in auth_password
    return self.auth_handler.wait_for_response(my_event)
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/paramiko/auth_handler.py", line 250, in wait_for_response
    raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/netautoscripts/telnetprocurve.py", line 1437, in <module>
    net_connect = ConnectHandler(device_type=device_type, host=ip_address,
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/netmiko/ssh_dispatcher.py", line 326, in ConnectHandler
    return ConnectionClass(*args, **kwargs)
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/netmiko/base_connection.py", line 350, in __init__
    self._open()
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/netmiko/base_connection.py", line 355, in _open
    self.establish_connection()
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/netmiko/base_connection.py", line 972, in establish_connection
    raise NetmikoAuthenticationException(msg)
netmiko.ssh_exception.NetmikoAuthenticationException: Authentication to device failed.

Common causes of this problem are:

Invalid username and password
Incorrect SSH-key file
Connecting to the wrong device

Device settings: hp_procurve 172.20.100.144:22

ktbyers commented 3 years ago

Can you show me what this looks like manually if you SSH into the device? You can obscure the username/password if needed...i.e. just use something fictional here.

Is there anything odd about this login--is it really slow or anything else?

Regards, Kirk

ntaken commented 3 years ago

Not sure why because it do not seem to matter which model or brand. Added html file anyhow. recording-172.20.100.144.html.zip I have the same random problem with cisco switches.

Below is from a Nornir run. Though the problem seems the same. Three of 23 cisco switches failed. (20workers, tried 5w still one failed)

On this one when I logged in manually, it was very responsive and everything seemed great.

WS-C3560CX-12PD-S v15.2(7)E3

2021-10-22 09:38:44,412 - nornir.core.task -    ERROR -      start() - Host 'TTT-TTT-TTT-SW28': task 'netmiko_send_command' failed with traceback:
Traceback (most recent call last):
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/netmiko/base_connection.py", line 935, in establish_connection
    self.remote_conn_pre.connect(**ssh_connect_params)
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/paramiko/client.py", line 435, in connect
    self._auth(
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/paramiko/client.py", line 764, in _auth
    raise saved_exception
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/paramiko/client.py", line 751, in _auth
    self._transport.auth_password(username, password)
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/paramiko/transport.py", line 1509, in auth_password
    return self.auth_handler.wait_for_response(my_event)
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/paramiko/auth_handler.py", line 250, in wait_for_response
    raise e
paramiko.ssh_exception.AuthenticationException: Authentication failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/nornir/core/task.py", line 99, in start
    r = self.task(self, **self.params)
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/nornir_netmiko/tasks/netmiko_send_command.py", line 26, in netmiko_send_command
    net_connect = task.host.get_connection(CONNECTION_NAME, task.nornir.config)
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/nornir/core/inventory.py", line 494, in get_connection
    self.open_connection(
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/nornir/core/inventory.py", line 546, in open_connection
    conn_obj.open(
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/nornir_netmiko/connections/netmiko.py", line 59, in open
    connection = ConnectHandler(**parameters)
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/netmiko/ssh_dispatcher.py", line 326, in ConnectHandler
    return ConnectionClass(*args, **kwargs)
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/netmiko/cisco/cisco_ios.py", line 17, in __init__
    return super().__init__(*args, **kwargs)
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/netmiko/base_connection.py", line 350, in __init__
    self._open()
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/netmiko/base_connection.py", line 355, in _open
    self.establish_connection()
  File "/home/itservice/.local/share/virtualenvs/netautoscripts-Ad84lSuf/lib/python3.8/site-packages/netmiko/base_connection.py", line 972, in establish_connection
    raise NetmikoAuthenticationException(msg)
netmiko.ssh_exception.NetmikoAuthenticationException: Authentication to device failed.

Common causes of this problem are:
1. Invalid username and password
2. Incorrect SSH-key file
3. Connecting to the wrong device

Device settings: cisco_ios 10.82.224.28:22

ktbyers commented 3 years ago

Are you using AAA for authentication?

It is always the same device(s) that fail or does the failing device move around?

ntaken commented 3 years ago

Yes, I run AAA. Tacacs

Tried removing AAA and everything worked great, repeatedly.

Different devices, it seems not to be limited to old devices or slow connection and when the failure happens it is much faster than the auth_timeout settings. Like in 6 sec but the timeout is 60/120 sec. Never had trouble to login to the device that had the auth error, immediately after failure.

Den fre 22 okt. 2021 kl 18:19 skrev Kirk Byers @.***>:

Are you using AAA for authentication?

It is always the same device(s) that fail or does the failing device move around?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ktbyers/netmiko/issues/2532#issuecomment-949776096, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARARCNQIDUX7XJ3JQQZFEHTUIGFIHANCNFSM5FZ7AKAA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

ktbyers commented 3 years ago

Which versions of Netmiko and Paramiko are you using?

ntaken commented 3 years ago

Main. netmiko 3.4.0, paramiko 2.8.0

Tried also in separate environment. netmiko 4.0.0a4, paramiko 2.8.0

ntaken commented 3 years ago

I did. No rejection.

There is that delay though between authentication and authorization on the decision, on the switch that failed.

Never been something we notice with our ssh clients.

Tried adding another tacacs server but have not seen any improvement.

Den tis 26 okt. 2021 22:35Kirk Byers @.***> skrev:

Did you look at the logs in your TACACS server and see if it is actually rejecting the authentications?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ktbyers/netmiko/issues/2532#issuecomment-952303601, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARARCNVOBFDXTDMBRZISLGTUI4GJ7ANCNFSM5FZ7AKAA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

pete-north commented 2 years ago

Did you guys ever get to the bottom of this. I'm having a very similar issue with cisco NXOS switches. I have a list of switches to apply configuration to. I will get an authentication failure from netmiko for say switch 5, I run the script again, it works for switch 5 but fails on 6. run again and it makes it one more down the list to 7. every time it reports an authentication problem:

line 972, in establish_connection raise NetmikoAuthenticationException(msg) netmiko.ssh_exception.NetmikoAuthenticationException: Authentication to device failed.

Common causes of this problem are:

Invalid username and password
Incorrect SSH-key file
Connecting to the wrong device

Device settings: cisco_nxos 10.50.1.9:22

Authentication failed.

very strange, feels more like an SSH key problem than authentication.

ntaken commented 2 years ago

The problem is not reproducible with a new ISE server set up.

jscooksey commented 2 years ago

I'm getting the same issue.

  File "C:\Python\Python39\lib\site-packages\netmiko\ssh_dispatcher.py", line 344, in ConnectHandler
    return ConnectionClass(*args, **kwargs)
  File "C:\Python\Python39\lib\site-packages\netmiko\base_connection.py", line 434, in __init__
    self._open()
  File "C:\Python\Python39\lib\site-packages\netmiko\base_connection.py", line 439, in _open
    self.establish_connection()
  File "C:\Python\Python39\lib\site-packages\netmiko\base_connection.py", line 1058, in establish_connection
    raise NetmikoAuthenticationException(msg)
netmiko.exceptions.NetmikoAuthenticationException: Authentication to device failed.

Common causes of this problem are:
1. Invalid username and password
2. Incorrect SSH-key file
3. Connecting to the wrong device

Device settings: cisco_xe 172.20.204.1:22

There is no delay, the error is pretty instantaneous.

If I try gitbash and ssh I get, and Im not sure if it's related. (ugg Win10 PC)

$ ssh user@172.20.204.1
Unable to negotiate with 172.20.204.1 port 22: no matching key exchange method found. Their offer: diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1

If I try Windows/Powershell command prompt with ssh it continually asks for my password

Yet I can access this fine using putty or SecureCRT. OpenSSH (for Windows) is installed and at version 8.1.0.1

ktbyers commented 2 years ago

Try adding:

disabled_algorithms = {'pubkeys': ['rsa-sha2-256', 'rsa-sha2-512']}

As an argument to ConnectHandler.

Anirudh-Suri commented 2 years ago

For me it was the same error. But then I saw it was due to the wrong IP given and also the reason might be that your SSH from the putty is not active.

AasthaNarula commented 2 years ago

I am getting the same error when I login with putty there is no error but when I login using python authentication failure occurs

roys10 commented 1 year ago

Did you guys ever get to the bottom of this. I'm having a very similar issue with cisco NXOS switches. I have a list of switches to apply configuration to. I will get an authentication failure from netmiko for say switch 5, I run the script again, it works for switch 5 but fails on 6. run again and it makes it one more down the list to 7. every time it reports an authentication problem:

line 972, in establish_connection raise NetmikoAuthenticationException(msg) netmiko.ssh_exception.NetmikoAuthenticationException: Authentication to device failed.

Common causes of this problem are:

Invalid username and password

Incorrect SSH-key file

Connecting to the wrong device

Device settings: cisco_nxos 10.50.1.9:22

Authentication failed.

very strange, feels more like an SSH key problem than authentication.

Hey I know it was a while ago but I am currently having the same problem, did you manage to fix it?

ktbyers commented 1 year ago

@roys10 You should create a new issue. This issue was closed as not reproducible.

pete-north commented 1 year ago

Did you guys ever get to the bottom of this. I'm having a very similar issue with cisco NXOS switches. I have a list of switches to apply configuration to. I will get an authentication failure from netmiko for say switch 5, I run the script again, it works for switch 5 but fails on 6. run again and it makes it one more down the list to 7. every time it reports an authentication problem: line 972, in establish_connection raise NetmikoAuthenticationException(msg) netmiko.ssh_exception.NetmikoAuthenticationException: Authentication to device failed. Common causes of this problem are:

Invalid username and password

Incorrect SSH-key file

Connecting to the wrong device

Device settings: cisco_nxos 10.50.1.9:22 Authentication failed. very strange, feels more like an SSH key problem than authentication.

Hey I know it was a while ago but I am currently having the same problem, did you manage to fix it?

I put it down to some sort of issue with inconsistent TACACS+ response times from our ISE server, likely related to processing load. I implemented a workaround via a loop that just made up to 3 attempts to connect to a device before declaring it unreachable. I only encountered the issue with NXOS network devices at one specific site.

roys10 commented 1 year ago

Did you guys ever get to the bottom of this. I'm having a very similar issue with cisco NXOS switches. I have a list of switches to apply configuration to. I will get an authentication failure from netmiko for say switch 5, I run the script again, it works for switch 5 but fails on 6. run again and it makes it one more down the list to 7. every time it reports an authentication problem: line 972, in establish_connection raise NetmikoAuthenticationException(msg) netmiko.ssh_exception.NetmikoAuthenticationException: Authentication to device failed. Common causes of this problem are:

Invalid username and password

Incorrect SSH-key file

Connecting to the wrong device

Device settings: cisco_nxos 10.50.1.9:22 Authentication failed. very strange, feels more like an SSH key problem than authentication.

Hey I know it was a while ago but I am currently having the same problem, did you manage to fix it?

I put it down to some sort of issue with inconsistent TACACS+ response times from our ISE server, likely related to processing load. I implemented a workaround via a loop that just made up to 3 attempts to connect to a device before declaring it unreachable. I only encountered the issue with NXOS network devices at one specific site.

Thanks :) For me this problem only occurs when I use threads to connect to the switches, do you remember if you used something similar?

ktbyers commented 1 year ago

@roys10 Yes, it is reasonably common for threads to overload the TACACS+ authentication/authorization system (i.e. the TACACS+ system becomes the bottleneck). I.E. the TACACS+ system can't handle the load and responds too slowly or with too much variation.

ktbyers / netmiko

Authentication failed despite correct password #2532