Weird issue re: failed logins

[paulcalabro]

I've seen this error occur a few times so I think it might be a bug:

$ ssh -i id_rsa <some_host> -l <some_user>

Account locked due to 9 failed logins

Account locked due to 10 failed logins

Account locked due to 11 failed logins

It's odd b/c the user is logging in with a key. I think it should either work or not. And for the most part it works, however, this happens on occasion. Any idea what's causing this?

[paulcalabro]

Also, just to clarify, we're not using a built-in user account.

[paulcalabro]

If you miss a phone call, for example when using Duo, you see this:

Calling your phone...
Error during call: No keys pressed.

Account locked due to 12 failed logins

Account locked due to 13 failed logins

Why is it showing multiple failed logins for one failed login attempt?

[osterman]

Are you using a keychain with multiple keys?

If so, you might want to adjust: https://github.com/cloudposse/bastion/blob/master/rootfs/etc/ssh/sshd_config#L27

(or some property like it)

[paulcalabro]

Thanks @osterman. I'll give that a try and post back.

[jfgudk]

Nice work on this by the way! I started using this solution yesterday and I have had the same thing happen a couple times using Google Auth too. Must admit I am using the root account, but the numbers appear only to increment on my connection attempts so pretty sure its not a hacking attempt. Restarting the container resolves so I suspect it is not environmental. Will provide more info when I can capture properly but wanted to confirm the issue.

[stephan-henningsen]

For what it's worth... I've seen something very similar, and had the feeling that it (at least in my case) was because the RTC of the host machine would skew. I think this would cause trouble if the OTPs are time-based. I gave up since I wasn't able to properly install a NTP client to confirm my assumption (I run a weird boot2docker setup on my server).

[jfgudk]

Latest update:

It looks like this: [jfgudk@centos1 bastion]$ ssh root@....us-east-1.elb.amazonaws.com Verification code: Account locked due to 6 failed logins Account locked due to 7 failed logins Permission denied (keyboard-interactive).

[jfgudk@centos1 bastion]$ ssh root@....us-east-1.elb.amazonaws.com Account locked due to 8 failed logins Account locked due to 9 failed logins Account locked due to 10 failed logins Permission denied (keyboard-interactive).

Occurred while someone else was successfully logged in and I tried to connect. Occurred before anyone else knew it was running - i.e. nobody else connected. Lockout count seemed to only be from me - +3 per attempt. Suspect EKS host time sync - need bastion to validate. Applied rebooted EKS pod, logged in to bastion and validated there was no significant time sync on any EKS hosts.

Note bastion access restricted to our 'trusted' networks.

Has anyone having the issue tried potential fix noted above https://github.com/cloudposse/bastion/blob/master/rootfs/etc/ssh/sshd_config#L27?

[paulcalabro]

@jfgudk I haven't tried it yet, however, I don't suspect it will address the source of the issue based on the docs:

MaxAuthTries

Specifies the maximum number of authentication attempts permitted per connection. Once the number of failures reaches half this value, additional failures are logged. The default is 6.

If I'm interpreting that correctly, it will close the TCP socket after 6 failed attempts. If you have multiple methods of authenticating, each one counts as an attempt. In my case, however, my SSH config is specifying the key to use so it should just be one failed attempt from Duo (e.g. if you don't ack the call). I think the problem is perhaps related to the Duo component.

[osterman]

If you run “ssh -vvv the-bastion-hostname” do you see it offer up exactly one key and no more?

[jfgudk]

This can be easily reproduced by attempting to connect and just hitting enter at the verification code prompt.

Looking into options to automatically reset after certain period.

FYI - I am using Google Authenticator

ssh is offering up 2 keys

[tecknicaltom]

I believe this is due to misconfiguration of pam_tally2. It does not appear to be clearing the failed login count upon a successful login. According to https://www.tecmint.com/use-pam_tally2-to-lock-and-unlock-ssh-failed-login-attempts/ there is a second line that should be added to Pam config to be run in the account phase which resets the failure count. I haven't verified this yet but you can run pam_tally2on the bastion and observe how the failed login count increases for a user even on a successful login

Edit: try running pam_tally2 -r to reset the failure count for all users

[jfgudk]

I can confirm that the counter increments even on a successful login. I changed my ssh client config to ensure only one cert is offered up and still the same issue. I haven't had time to dig into this more but if you can tell me what config changes to make I will test them out.

[jfgudk]

Finally back to fixing this, this patch fixed the issue (also disabling mfa):

--- /etc/pam.d/sshd.old
+++ /etc/pam.d/sshd
@@ -2,6 +2,6 @@
 auth       include     rate-limit
 auth       required    pam_env.so
+account    required    pam_tally2.so
 session    optional    pam_umask.so umask=0066
-auth       include     mfa
 session    include     sudosh
 session    include     enforcer

Thanks @tecknicaltom

[marji]

Hello @osterman, FYI, I believe that PR #53 fixes this issue of being blocked after multiple successful logins within a short period. Thanks @jfgudk and @tecknicaltom