The goal of 'wforce' is to detect brute forcing of passwords across many servers, services and instances. In order to support the real world, brute force detection policy can be tailored to deal with "bulk, but legitimate" users of your service, as well as botnet-wide slowscans of passwords.
The aim is to support the largest of installations, providing services to hundreds of millions of users. The current version of weakforced is not quite there yet, although it certainly scales to support up to ten million users, if not more. The limiting factor is number of logins per second at peak.
Wforce is a project by Dovecot, PowerDNS and Open-Xchange. For historical reasons, it lives in the PowerDNS github organization. If you have any questions, email neil.cook@open-xchange.com.
For detailed technical documentation, please go to https://powerdns.github.io/weakforced/.
Here is how it works:
wforce is aimed to receive message from services like:
By gathering failed and successful login attempts from as many services as possible, brute forcing attacks as well as other suspicious behaviour can be detected and prevented more effectively.
Docker:
There is a docker image hosted on docker hub, see https://powerdns.github.io/weakforced/ for more details.
From GitHub:
The easy way:
$ git clone https://github.com/PowerDNS/weakforced.git
$ cd weakforced
$ git submodule init
$ git submodule update
$ builder/build.sh debian-bullseye | debian-bookworm | el-7 | el-8 | el-9 | amazon-2This will build packages (wforce,wforce-trackalert and wforce-debuginfo) for the appropriate OS. You will need docker for the builder to work.
Note that since the 2.12 release, the built packages include an openresty luajit fork (wforce-lua-dist); this is because that fork
fixes issues in the main luajit library that wforce runs into under conditions of high load. The wforce-lua-dist package
also contains some lua modules that have proved useful to wforce deployments over the years as well as luarocks to install
new modules.
The hard way:
$ git clone https://github.com/PowerDNS/weakforced.git
$ cd weakforced
$ autoreconf -i
$ ./configure
$ makeThis requires recent versions of libtool, automake and autoconf to be installed.
It also requires:
To build on OS X, brew install readline and use
./configure PKG_CONFIG_PATH=<path to your openssl installation> LDFLAGS=-L/usr/local/opt/readline/lib CPPFLAGS=-I/usr/local/opt/readline/include
Add --with-luajit to the end of the configure line if you want to use LuaJIT.
There is a sensible, if very simple, default policy in wforce.conf (running without this means no policy), and extensive support for crafting your own policies using the insanely great Lua scripting language.
Note that although there is a single Lua configuration file, the canonicalize, reset, report and allow functions run in different lua states from the rest of the configuration. This mostly "just works", but may lead to unexpected behaviour such as running Lua commands at the server Lua prompt, and getting multiple answers (because Lua commands are passed to all Lua states).
Sample:
-- set up the things we want to track
field_map = {}
-- use hyperloglog to track cardinality of (failed) password attempts
field_map["diffFailedPasswords"] = "hll"
-- track those things over 6x10 minute windows
newStringStatsDB("OneHourDB", 600, 6, field_map)
-- this function counts interesting things when "report" is invoked
function twreport(lt)
sdb = getStringStatsDB("OneHourDB")
if (not lt.success)
then
sdb:twAdd(lt.remote, "diffFailedPasswords", lt.pwhash)
addrlogin = lt.remote:tostring() .. lt.login
sdb:twAdd(addrlogin, "diffFailedPasswords", lt.pwhash)
end
end
function allow(lt)
sdb = getStringStatsDB("OneHourDB")
if(sdb:twGet(lt.remote, "diffFailedPasswords") > 50)
then
return -1, "", "", {} -- BLOCK!
end
// concatenate the IP address and login string
addrlogin = lt.remote:tostring() .. lt.login
if(sdb:twGet(addrlogin, "diffFailedPasswords") > 3)
then
return 3, "tarpitted", "diffFailedPasswords", {} -- must wait for 3 seconds
end
return 0, "", "", {} -- OK!
endMany more metrics are available to base decisions on. Some example code is in wforce.conf, and more extensive examples are in wforce.conf.example. For full documentation, use "man wforce.conf".
To report (if you configured with 'webserver("127.0.0.1:8084", "secret")'):
$ for a in {1..101}
do
curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1", "pwhash":"1234'$a'", "success":"false"}' \
http://127.0.0.1:8084/?command=report -u wforce:secret
doneThis reports 101 failed logins for one user, but with different password hashes.
Now to look up if we're still allowed in:
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1", "pwhash":"1234"}' \
http://127.0.0.1:8084/?command=allow -u wforce:super
{"status": -1, "msg": "diffFailedPasswords"}It appears we are not!
You can also provide additional information for use by weakforce using the optional "attrs" object. An example:
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1",
"pwhash":"1234", "attrs":{"attr1":"val1", "attr2":"val2"}}' \
http://127.0.0.1:8084/?command=allow -u wforce:super
{"status": 0, "msg": ""}An example using the optional attrs object using multi-valued attributes:
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "remote": "127.0.0.1",
"pwhash":"1234", "attrs":{"attr1":"val1", "attr2":["val2","val3"]}}' \
http://127.0.0.1:8084/?command=allow -u wforce:super
{"status": 0, "msg": ""}There is also a command to reset the stats for a given login and/or IP Address, using the 'reset' command, the logic for which is also implemented in Lua. The default configuration for reset is as follows:
function reset(type, login, ip)
sdb = getStringStatsDB("OneHourDB")
if (string.find(type, "ip"))
then
sdb:twReset(ip)
end
if (string.find(type, "login"))
then
sdb:twReset(login)
end
if (string.find(type, "ip") and string.find(type, "login"))
then
iplogin = ip:tostring() .. login
sdb:twReset(iplogin)
end
return true
endTo test it out, try the following to reset the login 'ahu':
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu"}'\
http://127.0.0.1:8084/?command=reset -u wforce:super
{"status": "ok"}You can reset IP addresses also:
$ curl -X POST -H "Content-Type: application/json" --data '{"ip":"128.243.21.16"}'\
http://127.0.0.1:8084/?command=reset -u wforce:super
{"status": "ok"}Or both in the same command (this helps if you are tracking stats using compound keys combining both IP address and login):
$ curl -X POST -H "Content-Type: application/json" --data '{"login":"ahu", "ip":"FE80::0202:B3FF:FE1E:8329"}'\
http://127.0.0.1:8084/?command=reset -u wforce:super
{"status": "ok"}Finally there is a "ping" command, to check the server is up and answering requests:
$ curl -X GET http://127.0.0.1:8084/?command=ping -u wforce:super
{"status": "ok"}Available over TCP/IP, like this:
setKey("Ay9KXgU3g4ygK+qWT0Ut4gH8PPz02gbtPeXWPdjD0HE=")
controlSocket("0.0.0.0:4004")Launch wforce as a daemon (wforce --daemon), to connect, run wforce -c.
Comes with autocomplete and command history. If you put an actual IP address
in place of 0.0.0.0, you can use the same config to listen and connect
remotely.
To get some stats, try:
> stats()
40 reports, 8 allow-queries, 40 entries in databaseThe wforce manpage describes the command-line options and all the possible console commands in more detail.
Wforce accepts reports with 4 mandatory fields plus multiple optional fields.
Mandatory:
Optional:
The entire HTTP API is documented using the excellent OpenAPI (swagger) specification.
The pwhash field deserves some clarification. In order to distinguish actual brute forcing of a password, and repeated incorrect but identical login attempts, we need some marker that tells us if passwords are different.
Naively, we could hash the password, but this would spread knowledge of secret credentials beyond where it should reasonably be. Even if we salt and iterate the hash, or use a specific 'slow' hash, we're still spreading knowledge.
However, if we take any kind of hash and truncate it severely, for example to 12 bits, the hash tells us very little about the password itself - since one in 4096 random strings will match it anyhow. But for detecting multiple identical logins, it is good enough.
For additional security, hash the login name together with the password - this prevents detecting different logins that might have the same password.
NOTE: wforce does not require any specific kind of hashing scheme, but it is important that all services reporting successful/failed logins use the same scheme!
When in doubt, try:
TRUNCATE(SHA256(SECRET + LOGIN + '\x00' + PASSWORD), 12)Which denotes to take the first 12 bits of the hash of the concatenation of a secret, the login, a 0 byte and the password. Prepend 4 0 bits to get something that can be expressed as two bytes.
We can call 'report', and 'allow' commands. The optional 'attrs' field enables the client to send additional data to weakforced.
To report, POST to /?command=report a JSON object with fields from the LoginTuple as described above.
To request if a login should be allowed, POST to /?command=allow, again with the LoginTuple. The result is a JSON object with a "status" field. If this is -1, do not perform login validation (i.e. provide no clue to the client if the password was correct or not, or even if the account exists).
If 0, allow login validation to proceed. If a positive number, sleep this many seconds until allowing login validation to proceed.
You can create custom API commands (REST Endpoints) using the following configuration:
setCustomEndpoint("custom", customfunc)which will create a new API command "custom", which calls the Lua function "customfunc" whenever that command is invoked. Parameters to custom commands are always in the same form, which is key-value pairs wrapped in an 'attrs' object. For example, the following parameters sents as json in the message body would be valid:
{ "attrs" : { "key" : "value" }}Custom functions return values are also key-value pairs, this time wrapped in an 'r_attrs' object, along with a boolean success field, for example:
{ "r_attrs" : { "key" : "value" }, "success" : true}An example configuration for a custom API endpoint would look like:
function custom(args)
for k,v in pairs(args.attrs) do
infoLog("custom func argument attrs", { key=k, value=v });
end
-- return consists of a boolean, followed by { key-value pairs }
return true, { key=value }
end
setCustomEndpoint("custom", custom)An example curl command would be:
% curl -v -X POST -H "Content-Type: application/json" --data
'{"attrs":{"login1":"ahu", "remote": "127.0.0.1", "pwhash":"1234"}}'
http://127.0.0.1:8084/?command=custom -u wforce:super
{"r_attrs": {}, "success": true}It is possible to configure webhooks, which get called whenever specific events occur. To do this, use the "addWebHook" configuration command. For example:
config_keys={}
config_keys["url"] = "http://webhooks.example.com:8080/webhook/"
config_keys["secret"] = "verysecretcode"
events = { "report", "allow" }
addWebHook(events, config_keys)The above will call the webhook at the specified url, for every report and allow command received, with the body of the POST containing the original json data sent to wforce. For more information use "man wforce.conf" and "man wforce_webhook".
Custom webhooks can also be defined, which are not invoked based on specific events, but instead from Lua. Configuration is similar to normal webhooks:
config_keys={}
config_keys["url"] = "http://webhooks.example.com:8080/webhook/"
config_keys["secret"] = "verysecretcode"
config_keys["content-type"] = "application/json"
addCustomWebHook("mycustomhook", config_keys)However, the webhook will only be invoked via the Lua "runCustomWebHook" command, for example:
runCustomWebHook(mycustomhook", "{ \"foo\":\"bar\" }")The above command will invoke the custom webhook "mycustomhook" with the data contained in the second argument, which is simply a Lua string. No parsing of the data is performed, however the Content-Type of the webhook, which defaults to application/json can be customized as shown above.
Blacklisting capability is provided via either REST endpoints or Lua commands, to add/delete IP addresses, logins or IP:login tuples from the Blacklist. Blacklist information can be replicated (see below), and also optionally persisted in a Redis DB. Use "man wforce.conf" to learn more about the blacklist commands.
For high-availability or performance reasons it may be desireable to run multiple instances of wforce. To present a unified view of status however, these instances then need to share data. To do so, wforce implements a simple knowledge-sharing system.
The original version of wforce simply broadcast all received report tuples (best effort, UDP) to all siblings. However the latest version only broadcasts incremental changes to the underlying state databases, namely the stats dbs and the blacklist.
The sibling list is parsed such that we don't broadcast messages to ourselves accidentally, and can thus be identical across all servers.
Even if you configure siblings, stats db data is not replicated by default. To do this, use the "twEnableReplication()" command on each stats db for which you wish to enable replication. Blacklist information is automatically replicated if you have configured siblings.
To define siblings, use:
setKey("Ay9KXgU3g4ygK+qWT0Ut4gH8PPz02gbtPeXWPdjD0HE=")
addSibling("192.168.1.79")
addSibling("192.168.1.30")
addSibling("192.168.1.54")
siblingListener("0.0.0.0")The first line sets the authentication and encryption key for our sibling
communications. To make your own key (recommended), run makeKey() on the
console and paste the output in all your configuration files.
This last line configures that we also listen to our other siblings (which is nice). The default port is 4001, the protocol is UDP.
To view sibling stats:
> siblings()
Address Send Successes Send Failures Rcv Successes Rcv Failures Note
192.168.1.79:4001 18 7 0 0
192.168.1.30:4001 25 0 0 0
192.168.1.54:4001 0 0 0 0 SelfWith this setup, several wforces are all kept in sync, and can be load balanced behind (for example) haproxy, which incidentally can also offer SSL.
GeoIP support is provided using the GeoIP2 Maxmind APIs DBs (i.e. DBs ending in .mmdb). This is the preferred integration to use, as support for GeoIP Legacy DBs will be discontinued by Maxmind in 2019.
GeoIP2 DBs are represented by a Lua object that is created with the following call:
newGeoIP2DB("Name", "/path/to/file.mmdb")The Lua object is retrieved with the following call:
local mygeodb = getGeoIP2DB("Name")You can then lookup information using the following calls:
For example:
local geoip_data = mygeodp:lookupCity(newCA("128.243.21.16"))
print(geoip_data.city)
print(geoip_data.longitude)
print(geoip_data.latitude)Support for legacy GeoIP databases (i.e. ending in .dat) is deprecated, since Maxmind will be discontinuing support for them in 2019.
Three types of GeoIP lookup are supported:
The Country and ISP lookups return a string, while lookupCity() returns a Lua map consisting of the following keys:
For example:
local geoip_data = lookupCity(newCA("128.243.21.16"))
print(geoip_data.city)When a DB is initialized, wforce attempts to open both v4 and v6 versions of the database. If either is not found an error is thrown, so make sure both ipv4 and v6 versions of each DB are installed.
Additionally, when using the free/lite versions of the databases, you may see errors such as "initGeoIPCityDB(): Error initialising GeoIP (No geoip v6 city db available)". This is usually because the filenames for the "lite" DBs are not the same as the expected filenames for the full DBs, specifically all files must start with GeoIP rather than GeoLite. Creating symbolic links to the expected filenames will fix this problem, for example:
ln -s GeoLiteCityv6.dat GeoIPCityv6.dat