search

crowdsecurity / cs-firewall-bouncer

Crowdsec bouncer written in golang for firewalls
MIT License
119 stars 43 forks source link

mismatch between IPs from CrowdSec servers and those seen in nftables, reverse octet order (endianness) #368

Closed bughunter2 closed 5 months ago

bughunter2 commented 6 months ago

What happened?

2024-05-13

Problem:
--------

These symptoms seem to hint at one or more bugs.

There's a mismatch between the octet order in both the IPv4 and IPv6 addresses
added to nftables and the IP addresses returned by the CrowdSec servers and those
shown in the crowdsec-firewall-bouncer.log file (when log_level is set to debug).

At the very least, this can be a source of confusion.
At worst, it means CrowdSec is banning the wrong IP addresses.

And because there's a mismatch between what the CrowdSec servers return and
the IP addresses that get added to nftables, it might actually be
that CrowdSec is banning the wrong IPs. I surely hope this is not the case?

Also, you can't easily 'grep' your nftables for an IP address from
a CrowdSec log file, and instead you have to reverse the octets before
looking for that address in your firewall.

Questions:
----------

  - It seems to be an endianness bug, but which of the following, if any, is true?

  - Does CrowdSec ban the wrong IP addresses (reverse octet order)?

  - Do the CrowdSec servers return the wrong IP addresses (reverse octet order) through their HTTP API?

  - Does the crowdsec-firewall-bouncer.log file show IP addresses in the wrong octet order?

  - Are IP addresses added in the wrong octet order to nftables?

Troubleshooting details:
------------------------

1. Stop the crowdsec services.
2. Set crowdsec-firewall-bouncer's log_level to 'debug'.
3. Start the crowdsec service.
4. Start the crowdsec-firewall-bouncer service.
5. Observe the following:

The CrowdSec servers return something like this (output shown only partially here):

/var/log/crowdsec-firewall-bouncer.log contains a line like this (contains our example: 101.42.161.99):

time="13-05-2024 22:31:12" level=debug msg="Response: HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n
Content-Type: application/json; charset=utf-8\r\nDate: Mon, 13 May 2024 20:31:12 GMT\r\n\r\n231e5a\r\n{\"deleted\":null,
"new\":[{\"duration\":\"155h56m43.973575713s\",\"id\":14973,\"origin\":\"CAPI\",\"scenario\":\"crowdsecurity/http-bad-user-
agent\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":\"34.219.212.12\"},{\"duration\":\"155h56m43.973571691s\",\"id\":
14974,\"origin\":\"CAPI\",\"scenario\":\"crowdsecurity/http-probing\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":
\"4.227.114.13\"},{\"duration\":\"155h56m43.97356988s\",\"id\":14975,\"origin\":\"CAPI\",\"scenario\"...
...
...
r-agent\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":\"35.227.24.137\"},{\"duration\":\"155h56m43.97353777s\",\"id\":
14995,\"origin\":\"CAPI\",\"scenario\":\"crowdsecurity/ssh-slow-bf\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":
\"101.42.161.99\"},{\"duration\":\"155h56m43.973536851s\",\"id\":14996,\"origin\":\"CAPI\",\"scenario\":\"crowdsecurity/
http-bad-user-agent\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":\"54.185.132.207\"},{\"duration\":
\"155h56m43.973535581s\",\"id\":14997,\"origin\":\"CAPI\",\"scenario\":\"crowdsecurity/http-bad-user-agent\",\"...
...

/var/log/crowdsec-firewall-bouncer.log contains a line like this:

time="13-05-2024 22:31:12" level=debug msg="Adding '101.42.161.99' for '155h56m43.97353777s'"

... but the 'nft list ruleset' output does NOT contain IPv4 address 101.42.161.99.
However, it does contain the IPv4 address 99.161.42.101 ...:

# nft list ruleset | grep -in "101.42.161.99" # <== No matches.
# nft list ruleset | grep -in "99.161.42.101" # <== Does match:

3173:                99.161.42.101 timeout 6d11h56m43s972ms expires 6d11h55m29s16ms, 99.166.70.54 timeout 6d18h56m43s968ms expires 6d18h55m31s604ms,

The same happens with IPv6 addresses:
-------------------------------------

/var/log/crowdsec-firewall-bouncer.log contains a line like this:

time="13-05-2024 22:31:19" level=debug msg="adding 2a03:4000:0:39c:e40c:92ff:fe2d:8e38 to buffer "

While 'nft list ruleset' contains:

# nft list ruleset | grep -in "ff92"

7596:                388e:2dfe:ff92:ce4:9c03:0:40:32a timeout 6d18h56m43s968ms expires 6d18h55m35s444ms,

So which is correct?
  - 2a03:4000:0:39c:e40c:92ff:fe2d:8e38 # Returned by CrowdSec servers and shown in crowdsec-firewall-bouncer.log
  - 388e:2dfe:ff92:ce4:9c03:0:40:32a    # Shown in nft list ruleset.

Again, we can see that the octet order is reversed:

It seems to be "double reversed", in the sense that within each octet, the order is reversed as well.
For example, the beginning was 2a03, but it moves to the end and becomes 32a.

2a03 => 32a
4000 => 40
0    => 0
39c  => 9c03
e40c => ce4
92ff => ff92
fe2d => 2dfe
8e38 => 388e

What did you expect to happen?

That there's no mismatch between the IPv4 and IPv6 addresses returned by the CrowdSec servers, what the crowdsec-firewall-bouncer.log file shows, and what nftables shows (nft list ruleset). They should all be in agreement.

How can we reproduce it (as minimally and precisely as possible)?

See above.

Anything else we need to know?

No response

Crowdsec version

```console $ cscli version 2024/05/14 00:33:26 version: v1.4.6-6~deb12u1-debian 2024/05/14 00:33:26 Codename: alphaga 2024/05/14 00:33:26 BuildDate: 2023-07-15_09:29:33 2024/05/14 00:33:26 GoVersion: 1.19.8 2024/05/14 00:33:26 Platform: linux 2024/05/14 00:33:26 Constraint_parser: >= 1.0, <= 2.0 2024/05/14 00:33:26 Constraint_scenario: >= 1.0, < 3.0 2024/05/14 00:33:26 Constraint_api: v1 2024/05/14 00:33:26 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

```console $ cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" NAME="Debian GNU/Linux" VERSION_ID="12" VERSION="12 (bookworm)" VERSION_CODENAME=bookworm ID=debian HOME_URL="https://www.debian.org/" SUPPORT_URL="https://www.debian.org/support" BUG_REPORT_URL="https://bugs.debian.org/" $ uname -a 6.1.0-20-amd64 ```

Enabled collections and parsers

```console $ cscli hub list -o raw crowdsecurity/apache2,enabled,0.1,apache2 support : parser and generic http scenarios ,collections crowdsecurity/base-http-scenarios,enabled,0.6,http common : scanners detection,collections crowdsecurity/http-cve,enabled,1.9,,collections crowdsecurity/linux,enabled,0.2,core linux support : syslog+ssh,collections crowdsecurity/nginx,enabled,0.2,nginx support : parser and generic http scenarios,collections crowdsecurity/sshd,enabled,0.2,sshd support : parser and brute-force detection,collections crowdsecurity/apache2-logs,enabled,1.3,Parse Apache2 access and error logs,parsers crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/http-logs,enabled,1.1,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers crowdsecurity/nginx-logs,enabled,1.3,Parse nginx access and error logs,parsers crowdsecurity/sshd-logs,enabled,2.0,Parse openSSH logs,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers crowdsecurity/CVE-2022-26134,enabled,0.1,Detect CVE-2022-26134 exploits,scenarios crowdsecurity/CVE-2022-35914,enabled,0.1,Detect CVE-2022-35914 exploits,scenarios crowdsecurity/CVE-2022-37042,enabled,0.1,Detect CVE-2022-37042 exploits,scenarios crowdsecurity/CVE-2022-40684,enabled,0.2,Detect cve-2022-40684 exploitation attempts,scenarios crowdsecurity/CVE-2022-41082,enabled,0.3,Detect CVE-2022-41082 exploits,scenarios crowdsecurity/CVE-2022-41697,enabled,0.1,Detect CVE-2022-41697 enumeration,scenarios crowdsecurity/CVE-2022-42889,enabled,0.2,Detect CVE-2022-42889 exploits (Text4Shell),scenarios crowdsecurity/CVE-2022-44877,enabled,0.2,Detect CVE-2022-44877 exploits,scenarios crowdsecurity/CVE-2022-46169,enabled,0.1,Detect CVE-2022-46169 brute forcing,scenarios crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.4,Detect cve-2021-44228 exploitation attemps,scenarios crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.1,Detect cve-2020-5902 exploitation attemps,scenarios crowdsecurity/fortinet-cve-2018-13379,enabled,0.2,Detect cve-2018-13379 exploitation attemps,scenarios crowdsecurity/grafana-cve-2021-43798,enabled,0.1,Detect cve-2021-43798 exploitation attemps,scenarios crowdsecurity/http-backdoors-attempts,enabled,0.3,Detect attempt to common backdoors,scenarios crowdsecurity/http-bad-user-agent,enabled,0.7,Detect bad user-agents,scenarios crowdsecurity/http-crawl-non_statics,enabled,0.3,Detect aggressive crawl from single ip,scenarios crowdsecurity/http-cve-2021-41773,enabled,0.1,cve-2021-41773,scenarios crowdsecurity/http-cve-2021-42013,enabled,0.1,cve-2021-42013,scenarios crowdsecurity/http-generic-bf,enabled,0.4,Detect generic http brute force,scenarios crowdsecurity/http-open-proxy,enabled,0.3,Detect scan for open proxy,scenarios crowdsecurity/http-path-traversal-probing,enabled,0.2,Detect path traversal attempt,scenarios crowdsecurity/http-probing,enabled,0.2,Detect site scanning/probing from a single ip,scenarios crowdsecurity/http-sensitive-files,enabled,0.2,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios crowdsecurity/http-sqli-probing,enabled,0.2,A scenario that detects SQL injection probing with minimal false positives,scenarios crowdsecurity/http-xss-probing,enabled,0.2,A scenario that detects XSS probing with minimal false positives,scenarios crowdsecurity/jira_cve-2021-26086,enabled,0.1,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios crowdsecurity/nginx-req-limit-exceeded,enabled,0.1,Detects IPs which violate nginx's user set request limit.,scenarios crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.2,Detect cve-2019-11510 exploitation attemps,scenarios crowdsecurity/spring4shell_cve-2022-22965,enabled,0.2,Detect cve-2022-22965 probing,scenarios crowdsecurity/ssh-bf,enabled,0.1,Detect ssh bruteforce,scenarios crowdsecurity/ssh-slow-bf,enabled,0.2,Detect slow ssh bruteforce,scenarios crowdsecurity/thinkphp-cve-2018-20062,enabled,0.3,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios crowdsecurity/vmware-cve-2022-22954,enabled,0.2,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.1,Detect VMSA-2021-0027 exploitation attemps,scenarios ltsich/http-w00tw00t,enabled,0.1,detect w00tw00t,scenarios ```

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* # paste output here # On Windows: C:\> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml # paste output here

Config show

```console $ cscli config show Global: - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /var/lib/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log/ - Log level : info - Log Media : file Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /etc/crowdsec/acquis.d cscli: - Output : human - Hub Branch : - Hub Folder : /var/lib/crowdsec/hub Local API Server: - Listen URL : 127.0.0.1:8080 - Profile File : /etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : sqlite - Path : /var/lib/crowdsec/data/crowdsec.db - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console $ cscli metrics # paste output here ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

github-actions[bot] commented 6 months ago

@bughunter2: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.
blotus commented 6 months ago

Hello,

Looking at the version number of crowdsec, it seems that you are using the packages available in your distro repository. I cannot reproduce the issue with either the latest crowdsec + firewall bouncer version installed from our own repositories OR with a manual build on a debian 12 of crowdsec 1.4.6 and firewall bouncer 0.0.25 (which are the version shipped in debian), which would indicate the issue is coming specifically from the debian build.

The easiest fix for you would be to switch to our repositories (https://docs.crowdsec.net/docs/next/getting_started/install_crowdsec), that way you will be able to install much more recent and "known to work" versions.

Now, for the issue itself, it seems like it's coming from the firewall bouncer: when I manually add a decision in crowdsec and query LAPI directly, I see the proper IP in the stream, so the only place for the IP to get reversed is inside the bouncer itself. I'll try to contact the debian package maintainer to see if he has any idea where this could come from.

(I'm also moving the issue to the firewall bouncer repository, and I'll leave it open for now just for tracking purposes)

github-actions[bot] commented 6 months ago

@bughunter2: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Documentation to see if your issue can be self resolved.
  2. You can also join our Discord
Details I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/cs-firewall-bouncer/blob/main/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.
github-actions[bot] commented 6 months ago

@bughunter2: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

Details I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/cs-firewall-bouncer/blob/main/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.
CyrilBrulebois commented 6 months ago

Thanks for filing/forwarding the report, I'm able to reproduce this on my bookworm/amd64 system and I'm equally surprised and ashamed to have missed it until today…

That's likely an issue in the google/nftables module, and there's an issue + associated commit that looks promising:

I'll check what happens when I cherry-pick that in the Debian package, and rebuild the bouncer against it. If that helps, I'll upload the package to unstable and request a rebuild of the bouncer (or perform an upload if there are other changes to include along the way). Then I'll file a request to get that updated in stable as well (plus a rebuild).

Sorry about that…

CyrilBrulebois commented 6 months ago

I've just opened https://bugs.debian.org/1071247 for tracking purposes.

TL;DR: Applying the aforementioned patch and rebuilding the bouncer against it fixes the issue on LE systems, and doesn't regress on BE systems. I'll coordinate remediation with the security team (if that warrants a DSA) or with the release team (if it doesn't).

CyrilBrulebois commented 6 months ago

I haven't heard back from the security team yet, but I've just uploaded both packages (golang-github-google-nftables and crowdsec-firewall-bouncer) to unstable, which urgency=high. Hopefully they'll reach testing soon.

As for bookworm, I've published rebuilds of both packages with version numbers that match what would be used for bookworm-security or bookworm-proposed-updates in a custom repository, which can be configured with:

deb [trusted=yes] https://crowdsec.apt.debamax.com/debian bookworm main

(The signing key is available at that URL if you wish to configure it.)

CyrilBrulebois commented 5 months ago

The Debian security team decided this didn't warrant a security announcement, and that we should get this fixed via the upcoming point release instead: following a green light from the release team, I've just uploaded both packages. They should be available shortly in the bookworm-proposed-updates repository, and be merged into the bookworm one during the 12.6 point release, scheduled at the end of the month.

CyrilBrulebois commented 5 months ago

Packages have been accepted, built, and published in bookworm-proposed-updates:

kibi@tokyo:~$ apt-cache policy crowdsec-firewall-bouncer 
crowdsec-firewall-bouncer:
  Installed: (none)
  Candidate: 0.0.25-4~deb12u1
  Version table:
→    0.0.25-4~deb12u1 500
→       500 http://deb.debian.org/debian bookworm-proposed-updates/main amd64 Packages
     0.0.25-3 500
        500 http://deb.debian.org/debian bookworm/main amd64 Packages

(The fixed version is highlighted with arrows.)

LaurenceJJones commented 5 months ago

Thank you @CyrilBrulebois for all the updates on the matter, as this seems to be backported and fixed can we class the issue as resolved. Feel free @bughunter2 to unresolved the issue if this doesn't satisfy as a resolution.