Fail2ban - Raspberry Pi5 64bit Bookworm - not working as expected, not reading systemd logs?

[redsector72]

I run many Pi, but on the latest one, the pi 5, with the 64 bit Bookworm version, it does not work.

I had to manually tweak the "systemd" in /etc/fail2ban/paths-common.conf to replace

- default_backend = %(default/backend)s
+ default_backend = systemd

and avoid the error "Failed during configuration: Have not found any log file for sshd jail"

But after this, even if fail2ban is working, it does not see anything coming from journal regarding sshd. It simply stay put.

I run the latest version from Debian repository:

apt policy fail2ban
fail2ban:
  Installed: 1.0.2-2
  Candidate: 1.0.2-2
  Version table:
 *** 1.0.2-2 500
        500 http://deb.debian.org/debian bookworm/main arm64 Packages
        500 http://deb.debian.org/debian bookworm/main armhf Packages
        100 /var/lib/dpkg/status

On another PI with Buster 64 bit I have version Installed: 0.10.2-2.1 that works with systemd (maybe this one on bookwork is not the latest?)

Here is the OS version: PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian

I have tried googling for solutions, but seems like everyone fixes this with systemd modification (which worked for me on the others older pi).

thanks for pointing me in the right direction

[sebres]

I had to manually tweak the "systemd" in /etc/fail2ban/paths-common.conf to replace

Well, it depends what paths-common.conf does contain in your case, e. g. whether it was (incorrectly) patched by maintainers etc... However I guess some configs are changed previously by hand (and not upgraded later) or you have some mix now. Don't modify *.conf files... Instead, just use *.local to write local modifications only (not a copy of *.conf, but empty file with modified parameters only).

You can compare your version with the stock paths-common.conf and paths-debian.conf (or whatever is included in your jail.conf).

Anyway one doesn't need to set default_backend, because normally in jail.conf sshd jail uses https://github.com/fail2ban/fail2ban/blob/65e9c411ef7a78b302e974724bfce782df4eb57e/config/jail.conf#L282 and it is included in https://github.com/fail2ban/fail2ban/blob/65e9c411ef7a78b302e974724bfce782df4eb57e/config/paths-debian.conf#L15

So I'm unsure setting of default_backend would set it to the sshd jail.

Instead, you have either to set sshd_backend or to specify it directly in jail sshd in your jail.local (where you enable the jail):

[sshd]
backend = systemd
enabled = true

But after this, even if fail2ban is working, it does not see anything coming from journal regarding sshd.

Firstly ensure that sshd jail really uses backend systemd (inspect fail2ban.log or output of fail2ban-client -d | grep sshd).

Also you've to check whether you can see the sshd-messages with journalctl. If so there may be two reasons:

• either incorrect journalflags (see #2444 for details)
• or incorrect journalmatch... by default sshd filter uses https://github.com/fail2ban/fail2ban/blob/65e9c411ef7a78b302e974724bfce782df4eb57e/config/filter.d/sshd.conf#L129 You can find correct value if you inspect JSON output of journalctl for some sshd-message that must match.

You could then check whether you'd see matches with other values using: fail2ban-regex --journalmatch="..." systemd-journal[journalflags=1] sshd.

If there are no sshd messages with journalctl too, you have to find logpath where sshd really logs, set it to the jail and then set backend to auto (for file related monitoring).

[redsector72]

Thank @sebres, Here is my report after your suggestions 1) Tried to make a /etc/fail2ban/paths-common.local and revert the .conf to default, but then fail2ban won't start. Maybe I've not understood the suggestion

2)with fail2ban-client -d | grep sshd I see ['add', 'sshd', 'systemd'] so I suppose it's correct

3) in journalctl is see sshd events correctly in real-time May 19 04:47:13 Raspi sshd[351506]: Failed password for XxX from 192.168.1.7 port 54837 ssh2

4) I've tried to decode the rest of your message and I figure out that raspi journalctl will produce json entries of sshd with journalctl -t sshd -o json-pretty like this one: { "MESSAGE" : "Received signal 15; terminating.", "_SYSTEMD_UNIT" : "ssh.service", "_SYSTEMD_CGROUP" : "/system.slice/ssh.service", "SYSLOG_IDENTIFIER" : "sshd", "_RUNTIME_SCOPE" : "system", "_UID" : "0", "_TRANSPORT" : "syslog", "_GID" : "0", "_BOOT_ID" : "fd170a20f4014d829c87630ba9b99c59", "SYSLOG_FACILITY" : "4", "_CAP_EFFECTIVE" : "1ffffffffff", "PRIORITY" : "6", "_SYSTEMD_SLICE" : "system.slice", "__MONOTONIC_TIMESTAMP" : "1236398097895", "_EXE" : "/usr/sbin/sshd", "SYSLOG_TIMESTAMP" : "May 11 01:38:23 ", "_PID" : "3026454", "_HOSTNAME" : "raspi", "CURSOR" : "s=70a9a355a57644179aad5c7dc55898cf;i=ffeb;b=fd170a20f4014d829c87630ba9b99c59;m=11fdf11c5e7;t=618270f8a1c65;x=cb45d48fd36426df", "_COMM" : "sshd", "_CMDLINE" : "\"sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups\"", "_SYSTEMD_INVOCATION_ID" : "6151415db05e43ce8ebdb42aebfd384e", "_SOURCE_REALTIME_TIMESTAMP" : "1715405903764563", "REALTIME_TIMESTAMP" : "1715405903764581", "SYSLOG_PID" : "3026454", "_MACHINE_ID" : "0605a94ee130427f8d1f8d8fa39b4082" }

I have to note that to het this output I am not able to user the -u option as the service name is ssh and not sshd, but looking into this json log I cannot see the failed elements

So I dug further and looked at the full journal in json. when a password is wrong I see: {"_HOSTNAME":"Raspi","MESSAGE":"Failed password for pi from 192.168.1.7 port 56310 ssh2","SYSLOG_IDENTIFIER":"sshd","_SYSTEMD_INVOCATION_ID":"f89307b2e15d42c5a30aa75db319c80f","SYSLOG_PID":"352486","_RUNTIME_SC>

I cannot figure our any further. thanks for any assistance

[redsector72]

I've found the solution (at least it seems like so). I paste the solution found elsewhere for the people. in my previous implementation I was missing the "init" part.

Seems There is a bug with the Debian implementation of fail2ban, where the debian backend is now all systemd but fail2ban expects the old way for logging. Here's how you fix it.

First, activate systemd as the backend by going to the defaults-debian.conf using the command: sudo nano /etc/fail2ban/defaults-debian.conf

and adding backend = systemd under the [DEFAULTS] tag in the file. So it looks like:

[DEFAULT] backend = systemd Next, go to /etc/fail2ban/filter.d/sshd.conf and find the line: journalmatch = _SYSTEMD_UNIT=ssh.service + _COMM=sshd

add the word [Init] above it. So now it should look as:

[Init] journalmatch = _SYSTEMD_UNIT=ssh.service + _COMM=sshd Done.

Now restart the service with: sudo systemctl restart fail2ban, and then check with sudo systemctl status fail2ban.

[sebres]

Seems There is a bug with the Debian implementation of fail2ban, where the debian backend is now all systemd but fail2ban expects the old way for logging

I don't see the issue in stock debian 12, the unit is indeed ssh, but it'd anyway find all entries by _COMM=sshd and therefore journalmatch still works as expected on my debian boxes.

@fail2ban/maintainers, @sylvestre what shall we do here?

Anyway for the people having same issue for some reason, the simple configuration can be this (directly in jail.local for the sshd jail, where you enable it):

[sshd]
backend = systemd
journalmatch = _SYSTEMD_UNIT=ssh.service + _COMM=sshd
enabled = true

[sylvestre]

@sebres sure, let me fix that. Where do you recommend I added this ? (ssh by default) here ? https://salsa.debian.org/python-team/packages/fail2ban/-/blob/master/debian/debian-files/jail.d_defaults-debian.conf Thanks

[sebres]

Yep, it looks like good place for that... Just, the question is how it looks with other distros... If this is default name for the unit now, we could then adjust it here, directly in stock filter in https://github.com/fail2ban/fail2ban/blob/65e9c411ef7a78b302e974724bfce782df4eb57e/config/filter.d/sshd.conf#L129

[lodriguez]

Isn't it just called ssh.service on debian/ubuntu? Looks like all™ other distros are still using sshd.service. Had to revert it on arch.

[sebres]

Had to revert it on arch.

Hmm... why? There is still (unchanged) _COMM=sshd match what should additionally find sshd journal entries, doesn't it? Can you, please, provide a JSON output of message that matches _SYSTEMD_UNIT=sshd.service only.

[lodriguez]

Oh, I found the problem: "_COMM" : "sshd-session",

{
        "_SYSTEMD_UNIT" : "sshd.service",
        "_UID" : "0",
        "__MONOTONIC_TIMESTAMP" : "1086050885160",
        "_PID" : "998915",
        "__CURSOR" : "s=aa69a95780f646daa60b21cee16b4ab9;i=2f4f8c1;b=1cd45aacf0744b6b87bb5e45296d3e1b;m=fcddad5e28;t=61cf972ea41e1;x=ec5962cfc5c1b8ca",
        "_BOOT_ID" : "1cd45aacf0744b6b87bb5e45296d3e1b",
        "_CAP_EFFECTIVE" : "1ffffffffff",
        "_SOURCE_REALTIME_TIMESTAMP" : "1720707560653211",
        "SYSLOG_PID" : "998915",
        "PRIORITY" : "6",
        "SYSLOG_IDENTIFIER" : "sshd-session",
        "_MACHINE_ID" : "c22961e82cf84500bf312d2a6181ad7c",
        "_RUNTIME_SCOPE" : "system",
        "_COMM" : "sshd-session",
        "_EXE" : "/usr/lib/ssh/sshd-session",
        "_HOSTNAME" : "cyber",
        "_CMDLINE" : "\"sshd-session: [accepted]\"",
        "__SEQNUM" : "49608897",
        "_SYSTEMD_CGROUP" : "/system.slice/sshd.service",
        "__SEQNUM_ID" : "aa69a95780f646daa60b21cee16b4ab9",
        "_GID" : "0",
        "SYSLOG_TIMESTAMP" : "Jul 11 16:19:20 ",
        "_SYSTEMD_INVOCATION_ID" : "89248af0b8184cb1b13b5bf55fbda51d",
        "__REALTIME_TIMESTAMP" : "1720707560653281",
        "_TRANSPORT" : "syslog",
        "MESSAGE" : "User root from 46.43.9.189 not allowed because not listed in AllowUsers",
        "SYSLOG_FACILITY" : "4",
        "_SYSTEMD_SLICE" : "system.slice"
}

[lodriguez]

Should I open a new issue for this?

[sebres]

No, it belong surely to this issue.

[sebres]

However (and probably especially important due to #3782 / #3785), currently I don't know better option than:

journalmatch = _SYSTEMD_UNIT=sshd.service + _SYSTEMD_UNIT=ssh.service + _COMM=sshd + _COMM=sshd-session

The maintainer (or the users) could patch them to shorter value targeting the distribution and/or sshd version more precise.

[sebres]

Reverted back in master branch (c769046a1f729880cc53efdff4b52ac96010752f) and fixed for debian now (8902a2b884c4912d24221f3eda272c5de7bb0978). Thus close.