Lock down accounts by IP after N failed attemps at logging

[mattab]

Our security policy aims to make security a principal design behind Piwik. One aspect that bugs me currently is that good old brute force attacks could be vector of penetration in Piwik (if eg. attacker knows the login).

We should provide a core mechanism that would lock out, for 30min for example, a user after N failed attemps. Settings could be changed by the Super User and feature would be enabled by default, lock 30 min out after 5 failed attempts.

Implementation proposal:

• Record, using Piwik_SetOption, count of lockdown for each IP that fails to enter valid login / pwd combination
• After N failures, lock IP down and refuse authentication (even if the combination is actually valid!).
• Document as FAQ, linked from UI, the sql to delete all locked out IPs in case the SU was actually locked out and can't wait.

[sgiehl]

I would suggest to handle that the way like windows and many other software does. After 3 failed attemnds, lock the account and let the user wait a few minutes until he can retry. With every following failure raise the time to wait. I would do that global and not for each IP as it is too easy to change/switch the IP. Maybe we could implement an option to unlock the account with an token send by mail or something like that.

[robocoder]

If there's a lockdown, it should be by ip or /24.

The piwik_option table is not an appropriate place for this, imho, given the other scenarios I listed in #2794. We need.to keep track of the type of attack, ip, number of attempts, and timestamp of last attempt.

There should be some flexibility in the implementation to accomodate different responses to an attack. Can this implemented as a plugin?

[mattab]

The counter increase for a given IP should take place for any request which authenticates:

• failed login attempts (e.g., brute force)
• failed lost password requests / username/email check
• password reset with invalid (e.g., expired) reset token (e.g., replay)

For these, we should automatically blacklist the IP for X seconds, after N failed attempts within M seconds.

For an extended security (possible for a Version 2 of this feature since it complicates it)

• API request with invalid token Here maybe we shouldn't blacklist as there could be an error in a code calling the API which would blacklist possibly other functions calling API with a proper token. For a user calling the API with a wrong token, we should simply alert at first, and/or have an opt-in black list limit ?

[mainboarder]

I would like an extra subdirectory for administration (like ./admin) So the login could be restricted to ip ranges or a single ip via .htaccess or protected with basic auth But I think it would be a huge change in the code :/

Maybe a fail2ban lockdown could be as usefull as the .htaccess feature.

[mattab]

• We should also use this mechanism to protect against brute forcing the SMS authorization mechanism (since code is 5 chars, could be brute forced to send unwanted texts)

[anonymous-matomo-user]

I would also like this feature to be implemented and suggest also having an "immediately lock out IP when trying invalid/non-existent usernames" feature.

Also, email reports of when login attempts happen would be useful so you have a feel for how often you were being targeted.

I find both these features useful when using Wordfence for my WordPress sites.

[robocoder]

I suggest adding new event hooks and a plugin that leverages the PHPIDS or Expose libraries.

• https://github.com/PHPIDS/PHPIDS
• https://github.com/enygma/expose

[mainboarder]

Replying to ham12343:

I would also like this feature to be implemented and suggest also having an "immediately lock out IP when trying invalid/non-existent usernames" feature.

I think lockdown if a wrong username is used is useless or even a risk:

• what if you have a typo? like "hsm1243" instead of "ham12343"
• attackers could try to find out a correct username. it is found if the lockdown doesn`t happen immediately. (as long as there is a feedback like "your logins are now ignored")

[mattab]

Moving to short term as we'd like to be pro-active with security and this issue is an important protection layer.

[gaumondp]

Some ideas, could be one or all...

1. Lockdown IP after X attempts. Some kind of blacklist management will be need.
2. On bad credential wait X seconds before showing login screen (minimize web brute force).
3. Send an email to admin after X unsuccessful attempts.

The first one is the more complex, 2 and 3 seems quite easy to implement.

[d19dotca]

I'd also like to see the attempts logged somewhere too, mainly so that users can implement fail2ban with Piwik with ease.

[mattab]

@dustindauncey sure we will log them in Piwik application log

[jkraemer]

Actually just adding the logging (including timestamp and remote ip) might solve the whole issue for a lot of people. Fail2ban is made exactly for this purpose, there's no need to re-invent the wheel here imho.

[tsteur]

I think this is quite an important issue for security of Piwik. It is not too difficult to brute force installations otherwise.

Lockdown IP after X attempts. Some kind of blacklist management will be need.

After eg 50 attempts within 12 hours I would lock down IP (we'd need config for this if all users come from same IP or reuse trust_cookies setting which is used in Intranets and depending on this disable it etc).

On bad credential wait X seconds before showing login screen (minimize web brute force).

After say 5 wrong login attempts for same user, I would make login slower (also on API level but won't be trivial) each time. Eg in the beginning wait 1 second, next try wait 3 seconds, next try wait 6 seconds ... This needs to be implemented wisely since one could "shut down" a Piwik under circumstances by doing wrong login requests on purpose etc. Won't be trivial to implement I reckon but I'm sure for such things there are good solutions available on the internet

One could otherwise just crawl for several Piwik instances, try most common usernames with some most common passwords and I'm sure it's possible to get access to some installations

[mattab]

@tsteur moved to Short term and added Major tag - how much effort do you think it would take to seriously mitigate this security risk?

[gaumondp]

In TYPO3 we got Backend access with a sleep(5) on bad login for 15 years. There was a discussion about brute force that could be interesting to read :

https://forum.typo3.org/index.php/t/196184/

[tsteur]

From https://forum.typo3.org/index.php/t/196184/ :

You will get a warning email (if you set up an address in the install tool) after four unsuccessful attempts anyway.

This could be interesting in general and useful, I'll create an issue for this: https://github.com/piwik/piwik/issues/9140

Also maybe a good read: https://www.owasp.org/index.php/Blocking_Brute_Force_Attacks

I think a first version would be something between 1-2 days. It's really not easy I think since there are so many possibilities to workaround this and it would be possible to lock out real users.

[mattab]

FYI here is the UI for the "Limit login attempt" wordpress plugin which we use on piwik.org:

[ghub2015]

I have to agree with the comments that suggest just logging Piwik login attempts will be very useful.

That way we can use fail2ban, or other preferred method for ratelimiting or blocking attackers.

I am still very concerned about protecting Piwik without this.

[criwe]

+1

[tsteur]

FYI: There is https://plugins.piwik.org/ActivityLog which logs any login attempts etc and shows them in the UI. It is planned to show Country + IP .

For logging the requests a PR would be appreciated 👍 It may not be too hard. @ghub2015 what kind of format would be needed for this in the logs? Or is it format independent? Asking in case someone wants to add this feature

[ghub2015]

@tsteur, being able to parse standard Apache/Nginx log format (or any other webserver), is key for fail2ban. (In the case of Apache, it may be defined in the LogFormat stanza in /etc/apache2/apache2.conf)

So for example, logging an HTTP status code 401 on failed login attempt would be amazing!

These links may be of interest:

Fail2ban and Apache: http://www.fail2ban.org/wiki/index.php/Apache

HTTP Status code reference: https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.4.2

[mannp]

Wondered if there had been any progress on this one at all :)

Thanks in advance

[patrickbr]

I added a simple plugin called LoginFailLog to the marketplace. Once activated, every failed login attempt is written to the Piwik log file like this:

WARNING LoginFailLog[2017-07-22 23:35:20] [b215d] Failed login from 172.217.22.227 'patrick'.
WARNING LoginFailLog[2017-07-22 23:35:20] [b215d] Failed login from 172.217.22.227 with username 'patrick'.

You can use this plugin to protect Piwik with fail2ban or similar tools. See the README for a fail2ban example filter.

There is one problem though: Piwik logs everything in UTC time, for whatever reason. This is not configurable and may cause problems. For fail2ban, a workaround is described in the README.

[ghub2015]

@patrickbr that is awesome, thanks for this!

I will test and give feedback.

Let me just recommend (before even testing it) that you consider adding "Piwik" to what is logged so that it becomes:

WARNING Piwik LoginFailLog[2017-07-22 23:35:20] [b215d] Failed login from 172.217.22.227 'patrick'.
WARNING Piwik LoginFailLog[2017-07-22 23:35:20] [b215d] Failed login from 172.217.22.227 with username 'patrick'.

Just good for housekeeping/visual searching/grep'ing

[patrickbr]

@ghub2015 Everything before the "Failed login..." message is written by Piwik. The plugin uses the built-in logging system. Adding "Piwik" after "WARNING" would be redundant, as the piwik.log file only contains Piwik logs :)

[mattab]

Nice trick @patrickbr :+1:

There is one problem though: Piwik logs everything in UTC time, for whatever reason.

it seems there's an issue with our logger, as it does not log the timezone. LoginFailLog[2017-07-22 23:35:20] should include the timezone maybe LoginFailLog[2017-07-22 23:35:20 UTC] (would need to check whether this is the valid way). Please feel free to create a pull request for this :+1:

[Findus23]

@patrickbr

It seems like Fail2Ban does support UTC:https://github.com/fail2ban/fail2ban/pull/1583, https://github.com/fail2ban/fail2ban/issues/1575

But for that to work the log line needs to include the timezone as @mattab mentioned.

[patrickbr]

@mattab I created a pull request for this: https://github.com/piwik/piwik/pull/11893

[flaviusturc24]

I have a problem, I tested and it shows that IP is banned, but when I enter correct credentials, it is loggin in. What to do in this case?

[patrickbr]

@flaviusturc24 fail2ban works IP based on the connection level, it has nothing to to with your credentials. nor does it prevent you from logging in. It prevents you from even reaching the server/login page. If your fail2ban log shows that your IP is banned, but you can still reach the login page, then something is either misconfigured in fail2ban, or you have some rule in your firewall that allows traffic from IP to come through, regardless of any other rule.

[tsteur]

How do we handle whitelisted IPs? Should they be blocked as well or are they always trusted?

[mattab]

Blocked as well

[tsteur]

Of course we have thousands of options for this feature and can go crazy. But was thinking something simple in the beginning like this could do:

Implemented some base logic in https://github.com/matomo-org/matomo/compare/2888?expand=1 but haven't tested a single line.

Basically you define how many login attempts per time range are OK. Any further login attempt within that time range will be blocked. We could also lock the IP for a specified time range after this occurred but maybe not needed in V1.

Will need to see how to make sure this check is performed independent on the auth / session adapter... Also need to perform check for API.UsersManager.getTokenAuth. FYI: This will also need to be executed in tracking mode and authenticated requests will result in queries. We can look into caching the results but this would result in duplicating the logic as we would need to cache the raw data more or less which is not really useful (can also count IP => attempts etc but it is not 100% the same).

These are system settings. I will need to add a feature to clear / remove some blocked IPs... because not everyone can access the server or database and delete entries (eg on a cloud hosted Matomo). Ideally the settings would be also on that page but that means we'd need to create the form to configure these settings manually, add specific APIs for them, add tests, etc.

On this page we also might show some basic stats about currently locked IPs. Won't be sending out any email notifications for now as otherwise need to keep track for which locked IPs we have sent out notifications, need to offer to receive only daily or weekly emails etc. I would add this info page under "Diagnostics".

There is also a scheduled task to keep the table size smallish...

any thoughts?

[tsteur]

@mattab the basic logic is implemented but untested. Next step be to add tests and to ensure to run the check on all authentications independent of auth adapter.

The diagnostic page looks like this to keep things simple:

If there are any objections how things work be good to know before spending any time on adding tests.

FYI: I've put all that logic into the Login plugin.

[mattab]

Both screenshots look good! 👍

Regarding tracker api, I think maybe we do not want to activate this, because an attacker wouldn't be able to verify a token is valid or not using the tracking API (because the tracking API returns a valid 200 status code, even when token is invalid.)

[tsteur]

You can find out if token is valid eg by looking at time it takes to track a request (when token is invalid it would respond faster cause an exception is triggered and no DB involved etc)... or by using bulk request maybe (not tested but I think it might report it as invalid or something).

[mattab]

There is a good reason it's not wanted to activate "Block IP for N minutes" feature in tracking API:

• it would end up most of the time blocking IP addresses of "trusted servers" using HTTPS tracking API but having for some reason an invalid token for even a temporary time, or for other reasons.
• An invalid token would result in the whole server IP being blocked, potentially blocking loads of other customers / websites / etc. being tracked in the same Matomo instance, and imported from the same server (using logs, SDKs, etc.), but having valid tokens. It would also block tracking api requests that don't have a token.
• An invalid token is an acceptable state and partial / wrong data is tracked (when token is wrong, data is tracked with the server IP address instead of visitor IP, for example) so there is no direct logic in blocking these requests

At the same time we can't let people brute force and find a working token.

-> So maybe the solution is 1) disable this feature in Tracking API, and 2) not return errors in the Tracking API (even in bulk) so that Tracking API never leaks whether a token is valid? Then attackers have to use the Reporting API where the block IP would work.

(For Tracking API point of view, thinking the concern is also around preventing DDos/ loads of fake data. So maybe we could create another issue to introduce/reuse this IP blocking mechanism, but instead of blocking users with "too many failed token authentication", we could block IPs that send too much data over 1 hour, or 1 day, etc. This could be a nice tool to block robots, scripts, etc. storing wrong data. Currently, users have to define IP address range to block, but it's not automatic, and not retroactive, so painful if bad data has been tracked already)

[tsteur]

Currently, even a regular tracking request returns an HTTP 400 when not using correct token. This is expected so users can see something went wrong. It would be actually kind of breaking API changing behaviour there.

There would be also always still time attacks possible to find out the token.

For Tracking API point of view, thinking the concern is also around preventing DDos/ loads of fake data

It cannot result in fake data when keeping existing tracking behaviour plus apply brute force prevention.

DDos can happen same whether it is any API or UI request just like with the tracking reuest.

The brute force prevention wouldn't have much effect when not applied to all authentications as an attacker naturally picks the easiest way.

it would end up most of the time blocking IP addresses of "trusted servers" using HTTPS tracking API but having for some reason an invalid token for even a temporary time, or for other reasons.

The same can happen to regular API requests etc. That's not much of an issue and if minutes are kept low like 30 or 60 then they won't be locked out for long. Also if needed you can always whitelist specific IPs etc.

An invalid token would result in the whole server IP being blocked, potentially blocking loads of other customers / websites / etc. being tracked in the same Matomo instance, and imported from the same server (using logs, SDKs, etc.), but having valid tokens. It would also block tracking api requests that don't have a token.

Same applies to any API or UI request. And it would NOT block any api requests without a token.

An invalid token is an acceptable state and partial / wrong data is tracked (when token is wrong, data is tracked with the server IP address instead of visitor IP, for example) so there is no direct logic in blocking these requests

We currently don't track data when wrong token is provided and we shouldn't. Because users wouldn't notice they are tracking wrong data.

[mattab]

And it would NOT block any api requests without a token.

:ok: that's good, then it is fine to enable IP blocking in Tracking API

We currently don't track data when wrong token is provided and we shouldn't. Because users wouldn't notice they are tracking wrong data.

FYI I just tested with a wrong token and the tracking request was inserted correctly... I've created a separate issue to discuss this: https://github.com/matomo-org/matomo/issues/13471

[mattab]

New FAQ: How do I disable brute force authentication security checks for specific IP addresses?