Open lingfish opened 1 year ago
github-actions[bot]
commented
1 year ago @lingfish: Thanks for opening an issue, it is currently awaiting triage.
In the meantime, you can:
LaurenceJJones
commented
1 year ago So I prewarn calling the api using the same key as the bouncer will cause issue as the way the system works is it calculates the time different between calls and that how it knows what is new and old.
I will try to see to reproduce but there is not much to go on here. Is the bouncer actually working? Have you tested by triggering an alert then seeing if you can still gain access?
lingfish
commented
1 year ago Hi. I only started poking into the API using curl after the issue started, so I'm pretty sure what I did hasn't caused it; I understand your warning though going forward.
I get it's a strange one, but if there's any logs etc I can turn up to trace, let me know. I know there's not much to go on, need your guidance.
The bouncer is otherwise working just fine; bans are installed into a nftables set etc... it's just this one single IP. I don't think the issue is the bouncer, as it's never being told about this weird IP, only others. An example of a addition:
time="22-01-2023 15:04:00" level=debug msg="req-api: GET http://<LAPI host>:8080/v1/decisions/stream?origins=cscli%2Ccrowdsec"
time="22-01-2023 15:04:00" level=trace msg="auth-api request: GET /v1/decisions/stream?origins=cscli%2Ccrowdsec HTTP/1.1\r\nHost: <LAPI host>:8080\r\nUser-Agent: crowdsec-firewall-bouncer/v0.0.24-debian-pragmatic-<long UUID>\r\nX-Api-Key: <API key>\r\n\r\n"
time="22-01-2023 15:04:00" level=trace msg="auth-api response: HTTP/1.1 200 OK\r\nContent-Length: 184\r\nContent-Type: application/json; charset=utf-8\r\nDate: Sun, 22 Jan 2023 04:04:00 GMT\r\n\r\n{\"deleted\":null,\"new\":[{\"duration\":\"3h59m58.296214104s\",\"id\":3620822,\"origin\":\"crowdsec\",\"scenario\":\"crowdsecurity/postscreen-rbl\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":\"185.225.73.170\"}]}"
time="22-01-2023 15:04:00" level=debug msg="resp-api: http 200"
time="22-01-2023 15:04:00" level=debug msg="[headers] Content-Type : [application/json; charset=utf-8]"
time="22-01-2023 15:04:00" level=debug msg="[headers] Date : [Sun, 22 Jan 2023 04:04:00 GMT]"
time="22-01-2023 15:04:00" level=debug msg="[headers] Content-Length : [184]"
time="22-01-2023 15:04:00" level=debug msg="Response: HTTP/1.1 200 OK\r\nContent-Length: 184\r\nContent-Type: application/json; charset=utf-8\r\nDate: Sun, 22 Jan 2023 04:04:00 GMT\r\n\r\n{\"deleted\":null,\"new\":[{\"duration\":\"3h59m58.296214104s\",\"id\":3620822,\"origin\":\"crowdsec\",\"scenario\":\"crowdsecurity/postscreen-rbl\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":\"185.225.73.170\"}]}"
time="22-01-2023 15:04:00" level=debug msg="Adding '185.225.73.170' for '3h59m58.296214104s'"
time="22-01-2023 15:04:00" level=debug msg="committing added decisions"
time="22-01-2023 15:04:00" level=debug msg="adding 185.225.73.170 to buffer "
time="22-01-2023 15:04:00" level=debug msg="committed added decisions"
time="22-01-2023 15:04:00" level=info msg="1 decision added"When the bouncer starts, it sees other IPs already in a decision, just not the magical IP:
time="22-01-2023 14:28:40" level=debug msg="Adding '185.225.73.170' for '35m1.949202109s'"
time="22-01-2023 14:28:40" level=debug msg="Adding '2.57.122.215' for '1h16m13.776833835s'"
time="22-01-2023 14:28:40" level=debug msg="Adding '162.142.125.213' for '3h15m18.263834884s'"
time="22-01-2023 14:28:40" level=debug msg="committing added decisions"
time="22-01-2023 14:28:40" level=debug msg="adding 162.142.125.213 to buffer "
time="22-01-2023 14:28:40" level=debug msg="adding 185.225.73.170 to buffer "
time="22-01-2023 14:28:40" level=debug msg="adding 2.57.122.215 to buffer "
time="22-01-2023 14:28:40" level=debug msg="committed added decisions"
time="22-01-2023 14:28:40" level=info msg="3 decisions added"The LAPI notices this weird IP, logs about it, but strangely never gets reported to the bouncer:
time="24-01-2023 09:22:15" level=info msg="Bucket overflow" bucket_id=hidden-shadow capacity=0 cfg=hidden-hill file=/etc/crowdsec/scenarios/postfix-spam.yaml name=crowdsecurity/postscreen-rbl partition=1ebdee2c8992678b607ef9fae0516d5b0352680b
time="24-01-2023 09:22:15" level=info msg="Ip 193.56.29.178 performed 'crowdsecurity/postscreen-rbl' (1 events over 91ns) at 2023-01-23 22:22:15.463192835 +0000 UTC"
time="24-01-2023 09:22:15" level=info msg="(xxx/crowdsec) crowdsecurity/postscreen-rbl by ip 193.56.29.178 (PL/210228) : 4h ban on Ip 193.56.29.178"
time="24-01-2023 09:22:16" level=info msg="sent email to [xxx]" @module=email-plugin.email_default
time="24-01-2023 09:22:25" level=info msg="Signal push: 1 signals to push"
time="24-01-2023 09:23:44" level=info msg="flushed 1/1058 alerts because they were created 7d ago or more"Due to this, the nftables rule never gets installed, the spammer keeps trying to spam, I keep getting notifications.
lingfish
commented
1 year ago The plot thickens... I have now at least one other IP doing the same thing:
LaurenceJJones
commented
1 year ago Could there be some sort of bypass they are doing? as going to be straight up and say we cant really replicate without steps....
lingfish
commented
1 year ago Could there be some sort of bypass they are doing?
No. The bouncer, running on my boundary firewall, should be blocking them (and hence no further RBL stuff for 4 hours), and it isn't, because the decisions aren't showing up in the API, as posted above.
we cant really replicate without steps....
I understand, but surely we can do some debugging? Logging? Replicate with a dump of my DB?
LaurenceJJones
commented
1 year ago You pretty much have all the logging on in the previous comment, so those IP's are most likely in our community blocklist so the filter you are applying for debugging could be not returning it cause the will have the source CAPI.
Is the bouncer running in ipset mode only? I would for debugging create an api key, send the same request with no filters with startup true (send all current IP's) then grep the output for those if they are there, then some reason the LAPI already thinks the bouncer should know these.
lingfish
commented
1 year ago so the filter you are applying for debugging could be not returning it cause the will have the source CAPI
I think you've nailed it; I'm using those filters because yes, I've set origins: ["cscli", "crowdsec"] in the bouncer config.
Is the bouncer running in ipset mode only?
Yep, sure is.
I would for debugging create an api key, send the same request with no filters with startup true
Yep, done, and you're right... both IPs are in the list, origin CAPI.
So, being that origins is relatively new, I guess this is a kind of "feature" bug maybe? I would expect that the bouncer would be told to ban, if one intentionally wasn't using the CAPI list.
LaurenceJJones
commented
1 year ago I understand issue clearly now. I will need to have a chat with @buixor and see if I can replicate easily without CAPI I can spoof the source but will reply once I get some progress.
lingfish
commented
1 year ago Any news here?
LaurenceJJones
commented
1 year ago Just to keep you in the loop. We have tested a potential fix, however, it seems the way ent is converting the syntax to a sql statement means it takes minutes to returns a response. So we need to investigate it further.
lingfish
commented
1 year ago Hi, has this been resolved yet?
LaurenceJJones
commented
1 year ago No, as of yet we haven't found an optimal solution plus we are classing it as an edge case that a user wants to supply signals to CAPI but not use the community blocklist within a remediation component. So we have tagged this a low priority since we got multiple internal projects that we want to finish firstly.
LaurenceJJones
commented
1 year ago The only idea I had to workaround until a full feature fix is implemented is add additional config to CAPI configuration to allow you to set a flag to "send only"
Example
url: https://api.crowdsec.net/
login: XXXX
password: XXXX
share_only : true ## setting to true will cancel pull tombThen we create if statement for pull tomb
Proof of concept #2362
What happened?
I run a multi server install, and there is one particular IP that keeps getting a ban decision made, but the bouncer (nftables) doesn't ban it because it would seem the IP doesn't show up in
/v1/decisions/stream?origins=cscli,crowdsec&startup=trueWhat did you expect to happen?
The IP to be banned.
How can we reproduce it (as minimally and precisely as possible)?
I'm not entirely sure. Even if I delete the decision manually, it seems to initially block it, and then keep alerting, but not block it.
Anything else we need to know?
The IP is
193.56.29.178. It appears in decision lists, and alerts, but when the bouncer calls the API using/v1/decisions/stream?origins=cscli,crowdsec&startup=true(orfalse), the IP doesn't appear; other banned IPs do.The IP does appear in the sqlite DB.
Example, after running a curl against
/v1/decisions/stream?origins=cscli,crowdsec&startup=true:Crowdsec version
OS version
Enabled collections and parsers
Acquisition config
Config show
Prometheus metrics
Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.