search

YunoHost-Apps / synapse_ynh

Matrix server (synapse) package for YunoHost
https://matrix.org/
GNU General Public License v3.0
79 stars 42 forks source link

Fail2ban support #88

Closed anmol26s closed 5 years ago

anmol26s commented 5 years ago

Hi Is it possible to have Fail2ban support for Synapse?

Josue-T commented 5 years ago

Hello,

I don't know if it's possible, but probably. It in my todo list, but don't have time know.

Anyway if you have time to do this it could be also good...

Le 18 novembre 2018 19:43:04 GMT+01:00, anmol26s notifications@github.com a écrit :

Hi Is it possible to have Fail2ban support for Synapse?

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/YunoHost-Apps/synapse_ynh/issues/88

-- Envoyé de mon appareil Android avec K-9 Mail. Veuillez excuser ma brièveté.

anmol26s commented 5 years ago

If you can tell me where are the failed login attempts in logs I can try to implement it.

Josue-T commented 5 years ago

Normally the log file is in /var/log/matrix-synapse/homeserver.log. And for the regex if a login fail you need to do some test. But I think you should find something in this file because actually it's already really verbose.

anmol26s commented 5 years ago

It only shows info log can warning and error log be show in it too ?

Josue-T commented 5 years ago

I think the warning and error should be in too. The config file for the log is here https://github.com/YunoHost-Apps/synapse_ynh/blob/master/conf/log.yaml

anmol26s commented 5 years ago

Its actually hard to create regular expression for both a username which exits and password is wrong and when username don't exit because logs are so different and complicated. :(

Josue-T commented 5 years ago

Mybe you can create 2 regex.

Do you have an example of log for theses both cases ?

anmol26s commented 5 years ago

Its little difficult for me.

Here is the user not exit log:

2018-11-19 13:56:18,214 - synapse.access.http.8008 - 233 - INFO - POST-1128042- 2a02:XXXX:XXX:XXX::X - 8008 - Received request: POST /_matrix/client/r0/login?
2018-11-19 13:56:18,214 - synapse.rest.client.v1.login - 173 - INFO - POST-1128042- Got login request with identifier: {u'type': u'm.id.user', u'user': u'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'}, medium: None, address: None, user: u'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'
2018-11-19 13:56:18,221 - synapse.storage._base - 318 - WARNING - - Starting db txn 'get_users_by_id_case_insensitive' from sentinel context
2018-11-19 13:56:18,221 - synapse.storage._base - 354 - WARNING - - Starting db connection from sentinel context: metrics will be lost
2018-11-19 13:56:18,222 - synapse.metrics - 374 - INFO - - Collecting gc 0
2018-11-19 13:56:18,224 - synapse.handlers.auth - 570 - WARNING - - Attempted to login as @aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa:domain.tld but they do not exist
2018-11-19 13:56:18,224 - synapse.http.server - 85 - INFO - - <XForwardedForRequest at 0xXXXXXX method=u'POST' uri=u'/_matrix/client/r0/login?' clientproto=u'HTTP/1.0' site=8008> SynapseError: 403 - Invalid password

Here is invalid password log:

2018-11-19 13:59:03,964 - synapse.access.http.8008 - 233 - INFO - POST-1128163- 2a02:7b40:c3b5:f4f8::1 - 8008 - Received request: POST /_matrix/client/r0/login?
2018-11-19 13:59:03,964 - synapse.rest.client.v1.login - 173 - INFO - POST-1128163- Got login request with identifier: {u'type': u'm.id.user', u'user': u'user'}, medium: None, address: None, user: u'user'
2018-11-19 13:59:03,981 - synapse.storage._base - 318 - WARNING - - Starting db txn 'get_users_by_id_case_insensitive' from sentinel context
2018-11-19 13:59:03,981 - synapse.storage._base - 354 - WARNING - - Starting db connection from sentinel context: metrics will be lost
2018-11-19 13:59:03,983 - synapse.handlers.auth - 724 - WARNING - - Failed password login for user @user:domain.tld
2018-11-19 13:59:03,984 - synapse.http.server - 85 - INFO - - <XForwardedForRequest at 0xXXXXXXX method=u'POST' uri=u'/_matrix/client/r0/login?' clientproto=u'HTTP/1.0' site=8008> SynapseError: 403 - Invalid password
2018-11-19 13:59:03,985 - synapse.access.http.8008 - 302 - INFO - POST-1128163- 2a02:7b40:c3b5:f4f8::1 - 8008 - {None} Processed request: 0.021sec/0.000sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 68B 403 "POST /_matrix/client/r0/login? HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0" [0 dbevts]
Josue-T commented 5 years ago

Ok well I understand the problem, I could try to find a regex. For the password fail can you give me a log with an other username user because I don't know if it's the username or the word "user". USERNAME might be more explicit.

anmol26s commented 5 years ago

Sure, here is the log with USERNAME

synapse.access.http.8008 - 233 - INFO - POST-1131096- 2a02:XXX:XXX:XXX::1 - 8008 - Received request: POST /_matrix/client/r0/login?
2018-11-19 15:00:55,562 - synapse.rest.client.v1.login - 173 - INFO - POST-1131096- Got login request with identifier: {u'type': u'm.id.user', u'user': u'USERNAME'}, medium: None, address: None, user: u'USERNAME'
2018-11-19 15:00:55,581 - synapse.storage._base - 318 - WARNING - - Starting db txn 'get_users_by_id_case_insensitive' from sentinel context
2018-11-19 15:00:55,581 - synapse.storage._base - 354 - WARNING - - Starting db connection from sentinel context: metrics will be lost
2018-11-19 15:00:55,583 - synapse.handlers.auth - 724 - WARNING - - Failed password login for user @USERNAME:domain.tld
2018-11-19 15:00:55,584 - synapse.http.server - 85 - INFO - - <XForwardedForRequest at 0x7f51845fdfc8 method=u'POST' uri=u'/_matrix/client/r0/login?' clientproto=u'HTTP/1.0' site=8008> SynapseError: 403 - Invalid password
2018-11-19 15:00:55,585 - synapse.access.http.8008 - 302 - INFO - POST-1131096- 2a02:XXX:XXX:XXXX:1 - 8008 - {None} Processed request: 0.023sec/0.000sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 68B 403 "POST /_matrix/client/r0/login? HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0" [0 dbevts]
2018-11-19 15:00:55,694 - synapse.util.logcontext - 435 - WARNING - update_client_ips-272510- Unexpected logging context: POST-1131096@7f519178cd70 is not sentinel
Josue-T commented 5 years ago

Hello,

I found this configuration which should work : 

[Definition]

failregex = ^.+ \- INFO \- POST\-(\d+)\- <HOST> \- \d+ \- Received request\: POST /_matrix/client/r0/login\?<SKIPLINES>+.+ \- INFO \- POST\-\1\- Got login request with identifier: \{u'type': u'm.id.user', u'user'\: u'(.+?)'\}, medium\: None, address: None, user\: u'\5'<SKIPLINES>+.+WARNING \- \- (Attempted to login as @\5\:.+ but they do not exist|Failed password login for user @\5\:.+)$

[Init]

maxlines = 20

Note that it work on my side only with IPv4 but it's probably because I have an old fail2ban version.

You can also validate with this command : fail2ban-regex -v YOUR_LOG.LOG /PATH/TO/YOUR/CONFIG/FILE

anmol26s commented 5 years ago

This is not working for me for both IPv4 and IPv6. :(

Josue-T commented 5 years ago

How did you try ?

Did you add the "maxlines" option ?

Josue-T commented 5 years ago

On my side I have this : 

# fail2ban-regex -v synapse.log /root/f2b-synapse.conf 

Running tests
=============

Use   failregex file : /root/newtest.conf
Use         maxlines : 20
Use         log file : test_f2b_synapse.log
Use         encoding : UTF-8

Results
=======

Failregex: 3 total
|-  #) [# of hits] regular expression
|   1) [3] ^.+ \- INFO \- POST\-(\d+)\- <HOST> \- \d+ \- Received request\: POST /_matrix/client/r0/login\?<SKIPLINES>+.+ \- INFO \- POST\-\1\- Got login request with identifier: \{u'type': u'm.id.user', u'user'\: u'(.+?)'\}, medium\: None, address: None, user\: u'\5'<SKIPLINES>+.+WARNING \- \- (Attempted to login as @\5\:.+ but they do not exist|Failed password login for user @\5\:.+)$
|      192.168.1.2  Mon Nov 19 13:56:18 2018
|      192.168.1.2  Mon Nov 19 15:00:55 2018
|      192.168.1.3  Mon Nov 19 13:59:03 2018
`-

Ignoreregex: 0 total

Date template hits:
|- [# of hits] date format
|  [21] Year(?P<_sep>[-/.])Month(?P=_sep)Day 24hour:Minute:Second(?:,Microseconds)?
|  [0] (?:DAY )?MON Day 24hour:Minute:Second(?:\.Microseconds)?(?: Year)?
|  [0] (?:DAY )?MON Day Year 24hour:Minute:Second(?:\.Microseconds)?
|  [0] Day(?P<_sep>[-/])Month(?P=_sep)(?:Year|Year2) 24hour:Minute:Second
|  [0] Day(?P<_sep>[-/])MON(?P=_sep)Year[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone offset)?
|  [0] Month/Day/Year:24hour:Minute:Second
|  [0] Month-Day-Year 24hour:Minute:Second\.Microseconds
|  [0] TAI64N
|  [0] Epoch
|  [0] Year-Month-Day[T ]24hour:Minute:Second(?:\.Microseconds)?(?:Zone offset)?
|  [0] ^24hour:Minute:Second
|  [0] ^<Month/Day/Year2@24hour:Minute:Second>
|  [0] ^Year2MonthDay  ?24hour:Minute:Second
|  [0] MON Day, Year 12hour:Minute:Second AMPM
|  [0] ^MON-Day-Year2 24hour:Minute:Second
`-

Lines: 22 lines, 0 ignored, 16 matched, 6 missed
[processed in 0.04 sec]

|- Missed line(s):
|  2018-11-19 13:56:18,224 - synapse.http.server - 85 - INFO - - <XForwardedForRequest at 0xXXXXXX method=u'POST' uri=u'/_matrix/client/r0/login?' clientproto=u'HTTP/1.0' site=8008> SynapseError: 403 - Invalid password
|  2018-11-19 15:00:55,584 - synapse.http.server - 85 - INFO - - <XForwardedForRequest at 0x7f51845fdfc8 method=u'POST' uri=u'/_matrix/client/r0/login?' clientproto=u'HTTP/1.0' site=8008> SynapseError: 403 - Invalid password
|  2018-11-19 15:00:55,585 - synapse.access.http.8008 - 302 - INFO - POST-1131096- 192.168.1.2 - 8008 - {None} Processed request: 0.023sec/0.000sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 68B 403 "POST /_matrix/client/r0/login? HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0" [0 dbevts]
|  2018-11-19 15:00:55,694 - synapse.util.logcontext - 435 - WARNING - update_client_ips-272510- Unexpected logging context: POST-1131096@7f519178cd70 is not sentinel
|  2018-11-19 13:59:03,984 - synapse.http.server - 85 - INFO - - <XForwardedForRequest at 0xXXXXXXX method=u'POST' uri=u'/_matrix/client/r0/login?' clientproto=u'HTTP/1.0' site=8008> SynapseError: 403 - Invalid password
|  2018-11-19 13:59:03,985 - synapse.access.http.8008 - 302 - INFO - POST-1128163- 192.168.1.5 - 8008 - {None} Processed request: 0.021sec/0.000sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 68B 403 "POST /_matrix/client/r0/login? HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0" [0 dbevts]
`-
anmol26s commented 5 years ago

Yes I added 'maxlines'. Nothing found.

Running tests
=============

Use   failregex filter file : synapse, basedir: /etc/fail2ban
Use         maxlines : 20
Use         log file : /var/log/matrix-synapse/homeserver.log
Use         encoding : UTF-8

Results
=======

Failregex: 0 total
|-  #) [# of hits] regular expression
|   1) [0] ^.+ \- INFO \- POST\-(\d+)\- <HOST> \- \d+ \- Received request\: POST /_matrix/client/r0/login\?<SKIPLINES>+.+ \- INFO \- POST\-\1\- Got login request with identifier: \{u'type': u'm.id.user', u'user'\: u'(.+?)'\}, medium\: None, address: None, user\: u'\5'<SKIPLINES>+.+WARNING \- \- (Attempted to login as @\5\:.+ but they do not exist|Failed password login for user @\5\:.+)$
`-

Ignoreregex: 0 total

Date template hits:
Josue-T commented 5 years ago

Should be ok with that : 

[Definition]

__synapse_start_line = .? \- synapse\..+ \-

failregex = ^%(__synapse_start_line)s INFO \- POST\-(\d+)\- <HOST> \- \d+ \- Received request\: POST /_matrix/client/r0/login\??<SKIPLINES>%(__synapse_start_line)s INFO \- POST\-\1\- Got login request with identifier: \{u'type': u'm.id.user', u'user'\: u'(.+?)'\}, medium\: None, address: None, user\: u'\5'<SKIPLINES>%(__synapse_start_line)s WARNING \- \- (Attempted to login as @\5\:.+ but they do not exist|Failed password login for user @\5\:.+)$

[Init]

maxlines = 20
Josue-T commented 5 years ago

If it don't work can you give me a new time your log which don't match.

anmol26s commented 5 years ago

It did not worked this time too.

Logs for valid user with wrong password:

2018-11-23 05:29:06,883 - synapse.access.http.8008 - 302 - INFO - OPTIONS-28287- 2a02:XXX:XXX:XXX::1 - 8008 - {None} Processed request: 0.001sec/0.002sec (0.004sec, 0.000sec) (0.000sec/0.000sec/0) 22B 200 "OPTIONS /_matrix/client/r0/login? HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:63.0) Gecko/20100101 Firefox/63.0" [0 dbevts]
2018-11-23 05:29:07,105 - synapse.access.http.8008 - 233 - INFO - POST-28288- 2a02:XXX:XXX:XXX::1 - 8008 - Received request: POST /_matrix/client/r0/login?
2018-11-23 05:29:07,106 - synapse.rest.client.v1.login - 173 - INFO - POST-28288- Got login request with identifier: {u'type': u'm.id.user', u'user': u'URSERNAME'}, medium: None, address: None, user: u'USERNAME'
2018-11-23 05:29:07,125 - synapse.storage._base - 318 - WARNING - - Starting db txn 'get_users_by_id_case_insensitive' from sentinel context
2018-11-23 05:29:07,125 - synapse.storage._base - 354 - WARNING - - Starting db connection from sentinel context: metrics will be lost
2018-11-23 05:29:07,128 - synapse.handlers.auth - 745 - WARNING - - Failed password login for user @USERNAME:domain.tld
2018-11-23 05:29:07,129 - synapse.http.server - 85 - INFO - - <XForwardedForRequest at 0x7fcfa0100170 method=u'POST' uri=u'/_matrix/client/r0/login?' clientproto=u'HTTP/1.0' site=8008> SynapseError: 403 - Invalid password

Log for invalid user : 


2018-11-23 05:35:03,442 - synapse.access.http.8008 - 302 - INFO - OPTIONS-28329- 185.xx.xxx.xx - 8008 - {None} Processed request: 0.001sec/0.001sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 22B 200 "OPTIONS /_matrix/client/r0/login? HTTP/1.0" "Mozilla/5.0 (X11; Linux x86_64; rv:60.9) Gecko/20100101 Goanna/4.1 Firefox/60.9 PaleMoon/28.2.0" [0 dbevts]
2018-11-23 05:35:03,607 - synapse.access.http.8008 - 233 - INFO - POST-28330- 185.xx.xxx.xx - 8008 - Received request: POST /_matrix/client/r0/login?
2018-11-23 05:35:03,608 - synapse.rest.client.v1.login - 173 - INFO - POST-28330- Got login request with identifier: {u'type': u'm.id.user', u'user': u'aaaaaaaaaaaa'}, medium: None, address: None, user: u'aaaaaaaaaaaa'
2018-11-23 05:35:03,615 - synapse.metrics - 374 - INFO - - Collecting gc 0
2018-11-23 05:35:03,627 - synapse.storage._base - 318 - WARNING - - Starting db txn 'get_users_by_id_case_insensitive' from sentinel context
2018-11-23 05:35:03,628 - synapse.storage._base - 354 - WARNING - - Starting db connection from sentinel context: metrics will be lost
2018-11-23 05:35:03,631 - synapse.handlers.auth - 591 - WARNING - - Attempted to login as @aaaaaaaaaaaa:domain.tld but they do not exist
2018-11-23 05:35:03,632 - synapse.http.server - 85 - INFO - - <XForwardedForRequest at 0x7fcfa05a03f8 method=u'POST' uri=u'/_matrix/client/r0/login?' clientproto=u'HTTP/1.0' site=8008> SynapseError: 403 - Invalid password
anmol26s commented 5 years ago

This works for me:

[Definition]

failregex = ^.+ - INFO - POST-.- - 8008 - Received request: POST /_matrix/client/r0/login\?+.+ - INFO - POST-.- Got login request with identifier: {u'type': u'm.id.user', u'user': u'(.+?)'}, medium: None, address: None, user: u'.'+.+WARNING - - (Attempted to login as @. but they do not exist|Failed password login for user @.*)$

[Init]

maxlines = 20

JOduMonT commented 5 years ago

in some one fail2ban don't work

2019-04-02 03:25:14,749: DEBUG - + replace_string=synapse 2019-04-02 03:25:14,749: DEBUG - + sudo sed --in-place s@APP@synapse@g /etc/fail2ban/filter.d/synapse.conf 2019-04-02 03:25:14,749: DEBUG - + ynh_store_file_checksum /etc/fail2ban/jail.d/synapse.conf 2019-04-02 03:25:14,749: DEBUG - + local checksum_setting_name=checksumetc_fail2ban_jail.d_synapse.conf 2019-04-02 03:25:14,749: DEBUG - ++ sudo md5sum /etc/fail2ban/jail.d/synapse.conf 2019-04-02 03:25:14,749: DEBUG - ++ cut '-d ' -f1 2019-04-02 03:25:14,850: DEBUG - + ynh_app_setting_set synapse checksumetc_fail2ban_jail.d_synapse.conf 02ae324ceb193ab34e131e1a3b38cb86 2019-04-02 03:25:14,850: DEBUG - + sudo yunohost app setting synapse checksumetc_fail2ban_jail.d_synapse.conf --value=02ae324ceb193ab34e131e1a3b38cb86 --quiet 2019-04-02 03:25:14,950: DEBUG - + ynh_store_file_checksum /etc/fail2ban/filter.d/synapse.conf 2019-04-02 03:25:14,950: DEBUG - + local checksum_setting_name=checksumetc_fail2ban_filter.d_synapse.conf 2019-04-02 03:25:14,950: DEBUG - ++ sudo md5sum /etc/fail2ban/filter.d/synapse.conf 2019-04-02 03:25:14,951: DEBUG - ++ cut '-d ' -f1 2019-04-02 03:25:14,951: DEBUG - + ynh_app_setting_set synapse checksum__etc_fail2ban_filter.d_synapse.conf 1c55d648327a502ddddaa64c4430feba 2019-04-02 03:25:14,951: DEBUG - + sudo yunohost app setting synapse checksum__etc_fail2ban_filter.d_synapse.conf --value=1c55d648327a502ddddaa64c4430feba --quiet 2019-04-02 03:25:15,151: DEBUG - + systemctl try-reload-or-restart fail2ban 2019-04-02 03:25:20,470: DEBUG - + ynh_exit_properly 2019-04-02 03:25:20,470: WARNING - Job for fail2ban.service failed because the control process exited with error code. 2019-04-02 03:25:20,470: DEBUG - + local exit_code=1 2019-04-02 03:25:20,470: WARNING - See "systemctl status fail2ban.service" and "journalctl -xe" for details. 2019-04-02 03:25:20,471: DEBUG - + '[' 1 -eq 0 ']' 2019-04-02 03:25:20,471: WARNING - !! 2019-04-02 03:25:20,471: DEBUG - + trap '' EXIT 2019-04-02 03:25:20,471: WARNING - synapse's script has encountered an error. Its execution was cancelled. 2019-04-02 03:25:20,471: DEBUG - + set +eu 2019-04-02 03:25:20,471: WARNING - !! 2019-04-02 03:25:20,472: DEBUG - + echo -e '!!\n synapse'\''s script has encountered an error. Its execution was canc

Josue-T commented 5 years ago

Well, Yes but I think it's more related to the fail2ban helper than to this specific app.

Maybe can you create an issue here : https://github.com/YunoHost/issues/issues

Josue-T commented 5 years ago

Should be probably fixed by https://github.com/YunoHost/yunohost/pull/711/files

anmol26s commented 5 years ago

It there temp fix for this? I am not able to install the app.

Josue-T commented 5 years ago

Closing this issue because I think it should be fixed now. Please reopen or create a new issue if you sill have an issue about that.