fail2ban / fail2ban

Daemon to ban hosts that cause multiple authentication errors
http://www.fail2ban.org
Other
12.32k stars 1.26k forks source link

F2B v0.10.2 fails resolving the "matches" variable properly when passing to actions #2039

Closed ghost closed 6 years ago

ghost commented 6 years ago

Environment:

The issue:

After updating to the latest version an error occured within the resolve of variables (e.g: \<matches>) which get passed to specific filters (blocklist_de in this case). I already checked everything and didn't find the root cause which leads to that behaviour. As a result all generated reports get denied by blocklist.de since they miss correct log files and only pass a $f2bV_matches.

Another little addition to the checkbox about customizations above: I added my blocklist-API key in the [DEFAULT] section of my jail.config as described in the commented area from the corresponding jail.

Steps to reproduce

Activate the 'blocklist_de' action in the jail.config / jail.local file and wait till F2B gets 3 failed attempts in its authentification log. Check the fail2ban log after that and inspect the same error I described above.

Expected behavior

F2B should send a curl request containing sender adress, api key, bantime, service category and the corresponding logs in a url-encoded text format.

Observed behavior

F2B returns an error stating that the request got denied, if I remove the --fail parameter from the action file I get the following text back from blocklist.de: logs:

Please insert the Logfiles, and more than 20 Signs.
status: error

Any additional information

It seems like the variable $f2bV_matches gets passed at it is instead of resolving it into the log excerpt. Other webmasters are experiencing those issues too, as you can see right here: https://forum.blocklist.de/viewtopic.php?f=4&t=678 (it's in German though).

Configuration, dump and another helpful excerpts

jail.config

...
[sshd]

# To use more aggressive sshd modes set filter parameter "mode" in jail.local:
# normal (default), ddos, extra or aggressive (combines all).
# See "tests/files/logs/sshd" or "filter.d/sshd.conf" for usage example and details.
enable  = true
mode    = aggressive
port    = ssh
logpath = %(sshd_log)s
backend = %(sshd_backend)s
...

Any customizations done to /etc/fail2ban/ configuration

Relevant parts of /var/log/fail2ban.log file:

preferably obtained while running fail2ban with loglevel = 4

fail2ban.log

2018-02-03 18:29:50,735 fail2ban.utils          [12244]: Level 39 7f55f81ae8c8 -- exec: ['f2bV_matches=$0 \ncurl --fail --data-urlencode \'server=xxx@xxx.xx\' --data \'apikey=xxxxxxxxxx\' --data \'service=<service>\' --data \'ip=176.188.xxx.xxx\' --data-urlencode \'logs=$f2bV_matches\' --data \'format=text\' --user-agent "Fail2Ban/0.10.2" "https://www.blocklist.de/en/httpreports.html"', 'Feb  3 18:28:54 censored sshd[14382]: Failed password for invalid user pi from 176.188.xxx.xxx port 39980 ssh2']
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '                                 Dload  Upload   Total   Spent    Left  Speed'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: ''
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0     42  0:00:02  0:00:02 --:--:--    42'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0     30  0:00:03  0:00:03 --:--:--    30'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0     24  0:00:04  0:00:04 --:--:--    24'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0     20  0:00:05  0:00:05 --:--:--    22'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0     17  0:00:06  0:00:06 --:--:--    22'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0     14  0:00:08  0:00:07  0:00:01     0'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0     13  0:00:08  0:00:08 --:--:--     0'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0     11  0:00:10  0:00:09  0:00:01     0'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0     10  0:00:11  0:00:10  0:00:01     0'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      9  0:00:12  0:00:11  0:00:01     0'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      9  0:00:12  0:00:12 --:--:--     0'
2018-02-03 18:29:50,735 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      8  0:00:14  0:00:13  0:00:01     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      7  0:00:16  0:00:14  0:00:02     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      7  0:00:16  0:00:15  0:00:01     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      6  0:00:19  0:00:16  0:00:03     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      6  0:00:19  0:00:17  0:00:02     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      6  0:00:19  0:00:18  0:00:01     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      5  0:00:23  0:00:19  0:00:04     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      5  0:00:23  0:00:20  0:00:03     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      5  0:00:23  0:00:21  0:00:02     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      5  0:00:23  0:00:22  0:00:01     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      4  0:00:29  0:00:23  0:00:06     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      4  0:00:29  0:00:24  0:00:05     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      4  0:00:29  0:00:25  0:00:04     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      4  0:00:29  0:00:26  0:00:03     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      4  0:00:29  0:00:27  0:00:02     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      4  0:00:29  0:00:28  0:00:01     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      3  0:00:38  0:00:29  0:00:09     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      3  0:00:38  0:00:30  0:00:08     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      3  0:00:38  0:00:31  0:00:07     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      3  0:00:38  0:00:32  0:00:06     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      3  0:00:38  0:00:33  0:00:05     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      3  0:00:38  0:00:34  0:00:04     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      3  0:00:38  0:00:35  0:00:03     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      3  0:00:38  0:00:36  0:00:02     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      3  0:00:38  0:00:37  0:00:01     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:38  0:00:20     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:39  0:00:19     0'
2018-02-03 18:29:50,736 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:40  0:00:18     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:41  0:00:17     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:42  0:00:16     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:43  0:00:15     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:44  0:00:14     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:45  0:00:13     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:46  0:00:12     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:47  0:00:11     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:48  0:00:10     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:49  0:00:09     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:50  0:00:08     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:51  0:00:07     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:52  0:00:06     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:53  0:00:05     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:54  0:00:04     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:55  0:00:03     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: '100   116    0     0  100   116      0      2  0:00:58  0:00:56  0:00:02     0'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- stderr: 'curl: (22) The requested URL returned error: 400 Bad Request'
2018-02-03 18:29:50,737 fail2ban.utils          [12244]: ERROR   7f55f81ae8c8 -- returned 22
2018-02-03 18:29:50,737 fail2ban.actions        [12244]: ERROR   Failed to execute ban jail 'sshd' action 'blocklist_de' info 'ActionInfo({'ip': '176.188.xxx.xxx', 'family': 'inet4', 'ip-rev': '152.200.xxx.xxx.', 'ip-host': 'lcs07-xxx', 'fid': '176.188.xxx.xxx', 'failures': 3, 'time': 1517678934.0, 'matches': 'Feb  3 18:28:54 censored sshd[14382]: Failed password for invalid user pi from 176.188.xxx.xxx port 39980 ssh2', 'restored': 0, 'F-*': {'matches': ['Feb  3 18:28:54 censored sshd[14382]: Failed password for invalid user pi from 176.188.xxx.xxx port 39980 ssh2'], 'failures': 3, 'mlfid': ' censored sshd[14382]: ', 'user': 'pi', 'ip4': '176.188.xxx.xxx'}, 'ipmatches': 'Feb  3 18:28:54 censored sshd[14382]: Failed password for invalid user pi from 176.188.xxx.xxx port 39980 ssh2', 'ipjailmatches': 'Feb  3 18:28:54 censored sshd[14382]: Failed password for invalid user pi from 176.188.xxx.xxx port 39980 ssh2', 'ipfailures': 3, 'ipjailfailures': 3})': Error banning 176.188.xxx.xxx

Relevant lines from monitored log files in question:

auth.log

Feb  3 18:28:54 censored sshd[14382]: Invalid user pi from 176.188.xxx.xxx port 39980
Feb  3 18:28:54 censored sshd[14382]: error: Could not get shadow information for NOUSER
Feb  3 18:28:54 censored sshd[14382]: Failed password for invalid user pi from 176.188.xxx.xxx port 39980 ssh2
Feb  3 18:28:54 censored sshd[14382]: Connection closed by invalid user pi 176.188.xxx.xxx port 39980 [preauth]
Feb  3 18:28:54 censored sshd[14384]: Invalid user pi from 176.188.xxx.xxx port 39984
Feb  3 18:28:54 censored sshd[14384]: error: Could not get shadow information for NOUSER
Feb  3 18:28:54 censored sshd[14384]: Failed password for invalid user pi from 176.188.xxx.xxx port 39984 ssh2
Feb  3 18:28:54 censored sshd[14384]: Connection closed by invalid user pi 176.188.xxx.xxx port 39984 [preauth]
ghost commented 6 years ago

I just tested around and that's the response I get when trying it manually with the provided info out of my fail2ban logfiles:

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 2a00:xxxx:x:xxxx::x...
* TCP_NODELAY set
*   Trying xxx.xxx.xxx.xxx...
* TCP_NODELAY set
* Connected to www.blocklist.de (xxx.xxx.xxx.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [107 bytes data]
* NPN, negotiated HTTP1.1
{ [5 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2871 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Unknown (67):
} [36 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: OU=Domain Control Validated; CN=*.blocklist.de
*  start date: Dec xxx.xxx.xxx.xxx 2017 GMT
*  expire date: Jan xxx.xxx.xxx.xxx 2019 GMT
*  subjectAltName: host "www.blocklist.de" matched cert's "*.blocklist.de"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=AlphaSSL CA - SHA256 - G2
*  SSL certificate verify ok.
} [5 bytes data]
> POST /en/httpreports.html HTTP/1.1
> Host: www.blocklist.de
> User-Agent: Fail2Ban/0.10.2
> Accept: */*
> Content-Length: 110
> Content-Type: application/x-www-form-urlencoded
> 
} [110 bytes data]
* upload completely sent off: 110 out of 110 bytes
{ [5 bytes data]
< HTTP/xxx.xxx.xxx.xxx Bad Request
< Server: nginx/1.12.2
< Date: Sat, 03 Feb xxx.xxx.xxx.xxx:46 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Keep-Alive: timeout=20
< X-Frame-Options: sameorigin
< 
{ [93 bytes data]

100   192    0    82  100   110    178    239 --:--:-- --:--:-- --:--:--   418
* Connection #0 to host www.blocklist.de left intact
logs: Please insert the Logfiles, and more than 20 Signs.<br />status: error<br />
sebres commented 6 years ago

Duplicate of #2028. Fixed in #2034. See changes for blocklist_de.conf - single-quotes replaced with double-quotes.