Weird user lockout problem/behaviour: "Login has been temporarily blocked for this User"

[nooblag]

Description:

I'm an administrator on a Rocket Chat instance and got a user report that they were locked out of their account because they forgot their password and had failed login attempts. I was able to reset the password as admin some time later (hours later after they reported it) but even the next day (more than 24hrs later) and after a reboot of the Rocket Chat server the user reported being still locked out of their account. They were also on another IP address by that time (and after trying on multiple devices). I checked the settings in Administration -> Accounts -> Failed Login Attempts and they are as follows:

So I then tried clicking on the user account in Administration -> Users and clicked "Reset TOTP" and also clicked "Deactivate" and waited a moment, then "Activate" to re-enable the account. Both didn't work.

The only way the user was able to get back into their account was for me to toggle the "verify" button next to their email address so it became unverified. The user then was able to login successfully, change their password, and reverify their email address.

Clearly something is broken here... Also, forum post about this issue too: https://forums.rocket.chat/t/how-to-unblock-blocked-user/9800 Though they said they are having the same problem regardless of 2FA.

Our 2FA settings are these:

Steps to reproduce:

1. User account is locked due to failed login attempts

Expected behavior:

1. User IP is unblocked after specified time and/or
2. User account is hard reactivated after an admin does a password reset and/or going through "Deactivate" and "Activate" process (which should be unnecessary anyway but an absolute failsafe?)

Actual behavior:

User account remained locked until email address was switched to "unverified".

Server Setup Information:

• Version of Rocket.Chat Server: 3.12.0
• Operating System: Ubuntu 18.04.5 LTS
• Deployment Method: TAR (manual)
• Number of Running Instances: 1
• DB Replicaset Oplog: True
• NodeJS Version: 12.18.4
• MongoDB Version: 4.0.23

Client Setup Information

• Desktop App or Browser Version: Firefox 86.0.1
• Operating System: Mac OS Big Sur and Windows 10

Additional context

Forum post about this issue: https://forums.rocket.chat/t/how-to-unblock-blocked-user/9800

Relevant logs:

These logs after the server has been reset and more than 24hrs after first report of account locking:

I20210317-16:37:07.350(11) Failed login detected - Username[unknown] ClientAddress[xxx.xxx.xxx.xxx] ForwardedFor[xxx.xxx.xxx.xxx] XRealIp[xxx.xxx.xxx.xxx] UserAgent[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15] 
I20210317-16:37:07.355(11) Exception while invoking method login Error: Login has been temporarily blocked For User [error-login-blocked-for-user]     at app/authentication/server/startup/index.js:308:9     at packages/callback-hook/hook.js:131:22     at packages/accounts-base/accounts_server.js:167:15     at Hook.each (packages/callback-hook/hook.js:109:15)     at AccountsServer._validateLogin (packages/accounts-base/accounts_server.js:164:29)     at AccountsServer._attemptLogin (packages/accounts-base/accounts_server.js:353:10)     at MethodInvocation.methods.login (packages/accounts-base/accounts_server.js:535:23)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:263:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:263:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9
I20210317-16:41:40.431(11) Exception while invoking method login Error: Login has been temporarily blocked For User [error-login-blocked-for-user]     at app/authentication/server/startup/index.js:308:9     at packages/callback-hook/hook.js:131:22     at packages/accounts-base/accounts_server.js:167:15     at Hook.each (packages/callback-hook/hook.js:109:15)     at AccountsServer._validateLogin (packages/accounts-base/accounts_server.js:164:29)     at AccountsServer._attemptLogin (packages/accounts-base/accounts_server.js:353:10)     at MethodInvocation.methods.login (packages/accounts-base/accounts_server.js:535:23)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:263:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:263:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9
I20210317-17:18:54.501(11) Exception while invoking method verifyEmail Error: TOTP Required [totp-required]     at checkCodeForUser (app/2fa/server/code/index.ts:186:9)     at app/2fa/server/loginHandler.js:30:2     at callbacks.runItem (app/callbacks/lib/callbacks.js:111:70)     at Object.callbacks.runItem (app/metrics/server/callbacksMetrics.js:24:20)     at app/callbacks/lib/callbacks.js:39:35     at app/callbacks/lib/callbacks.js:45:45     at callbacks.run (app/callbacks/lib/callbacks.js:127:9)     at Object.callbacks.run (app/metrics/server/callbacksMetrics.js:14:17)     at app/authentication/server/startup/index.js:346:20     at packages/callback-hook/hook.js:131:22     at packages/accounts-base/accounts_server.js:167:15     at Hook.each (packages/callback-hook/hook.js:109:15)     at AccountsServer._validateLogin (packages/accounts-base/accounts_server.js:164:29)     at AccountsServer._attemptLogin (packages/accounts-base/accounts_server.js:353:10)     at AccountsServer._loginMethod (packages/accounts-base/accounts_server.js:385:17)     at MethodInvocation.verifyEmail (packages/accounts-password/password_server.js:928:19)     at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)     at packages/ddp-server/livedata_server.js:1689:15     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at packages/ddp-server/livedata_server.js:1687:36     at new Promise (<anonymous>)     at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)     at Server.apply (packages/ddp-server/livedata_server.js:1625:26)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:263:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9  => awaited here:     at Promise.await (/opt/Rocket.Chat/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)     at Server.apply (packages/ddp-server/livedata_server.js:1638:22)     at Server.call (packages/ddp-server/livedata_server.js:1607:17)     at Object.post (app/api/server/v1/misc.js:263:26)     at app/api/server/api.js:394:82     at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)     at Object._internalRouteActionHandler [as action] (app/api/server/api.js:394:39)     at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)     at packages/nimble_restivus/lib/route.coffee:59:33     at packages/simple_json-routes.js:98:9 

[Sarimuko]

I am trying to work on this issue, any suggestions on where to begin with?

[nooblag]

I have no idea sorry, I'm not a developer, just reporting the bug

[renancleyson-dev]

After reproducing(without 2FA), this is what I figured out, let's use the enumeration to refer to these checks:

1. The server counts how many failed attempts to log in the user did since his last session. Then checks if it's less than the limit count https://github.com/RocketChat/Rocket.Chat/blob/f01b25554bb5eaf08ac4aada9c6c2c9ff48f44dc/app/authentication/server/lib/restrictLoginAttempts.ts#L62-L66
2. then checks if the current moment is after the blocked time + the time of the previous failed attempt to login. It will create a new failed attempt if false and the next attempt will do the same check, but with the current attempt moment https://github.com/RocketChat/Rocket.Chat/blob/f01b25554bb5eaf08ac4aada9c6c2c9ff48f44dc/app/authentication/server/lib/restrictLoginAttempts.ts#L80-L83

If both of the cases are false, then the server thinks that the user is blocked.

Thus we have some problems:

• If someone tries to log in while is blocked, the blocked time will "reset"(caused by 2). So if you wait 9 minutes and you are set to be blocked for 10 minutes, then when you try to log in, you will have to wait more than 10 minutes
• The server will still consider the previous failed attempts count after the block period, which means that if your login fails only once after being blocked and the attempt fails, you will be blocked again(caused by 1 that will calculate the block only before the last session).

Clearly something is broken here... Also, forum post about this issue too: https://forums.rocket.chat/t/how-to-unblock-blocked-user/9800 Though they said they are having the same problem regardless of 2FA.

I think some of the issues I explained was the problem faced on the forum.

Now, reproducing with TOTP:

1. You write the username and password, then submit.
2. The server sees it as a failed attempt to log in because TOTP is required, this is what triggers the TOTP dialog to show, but it will make the block time resets too(caused by 2 again)!
3. You write the TOTP code, but you are blocked.

I think that the problem it's simply the block moment not being stored anywhere on the database.

[nooblag]

Has this been fixed I wonder?

[benji1000]

Hello,

we are experiencing the issue on our instance of Rocket.chat version 3.14.4. The scenario was exactly the same as @nooblag described: user had 2FA enabled, failed to authenticate 5 times in a row, and got locked up. However, the resolution was different, because disabling 2FA-TOTP and the "Email verified" switch did not solve the issue, no more than disabling 2FA-email (with a MongoDB query, since it is inexplicably not possible within the UI).

For the user to be able to log in again, we had to disable the three switches in the Accounts > Failed Login Attempts section (effectively disabling the anti-bruteforce mechanism for our whole instance), asking the user to authenticate, and then re-enabling the three switches. Since 2FA was disabled for the user during our previous tests, it may have had a role in the resolution. However, we already had re-enabled the "Email verified" switch, so the user was able to authenticate even though his email address was marked as verified: it really was disabling the anti-bruteforce mechanism that worked for us.

Additionally, when monitoring the user attempts using the console available in the Admin UI to see which errors were logged, the user login information was not reported, as well as the IP address. Here is what we saw:

Failed login detected - Username[unknown] ClientAddress[null]

The user was using his email address to authenticate. We began seeing detailed data in the logs when the user was instructed to input his username instead. It doesn't really make sense to me that whatever the user submitted as login is not displayed within the logs (to some extent... with security mechanisms to prevent XSS or anything like that), but it probably is another issue.

Anyway, this blocking issue was first reported 6 months ago by another user on the Rocket.chat forum, and @renancleyson-dev has done a great job of looking through the source code and potentiaIly identifying the root cause. I think it may be time that the devs look into the issue and fix it, or at least officially acknowledge the issue and add it to the next development phase? Pinging at random (well, based on the top contributors): @rodrigok, @sampaiodiego, @engelgabriel. Thank you guys, and good luck.

While you're at it, would it be possible to:

• add the possibility within the admin UI to manually unblock a user,
• add the possibility within the admin UI to disable 2FA-email (not just 2FA-TOTP).

[nooblag]

Closing, as gave up on Rocket Chat many years ago.