Open pschiffe opened 4 months ago
github-actions[bot]
commented
4 months ago @pschiffe: Thanks for opening an issue, it is currently awaiting triage.
In the meantime, you can:
blotus
commented
4 months ago Hello,
I saw on your discord thread that you had multiple loki data sources configured. Can you confirm that none of the different data sources will get the same logs? Crowdsec does not perform any deduplication on the logs it reads, so you must ensure a single line will only be seen once.
Do you have a rough idea of how many logs per second are pushed to loki ?
You can also set the data source to trace (by setting log_level: trace in the acquis configuration) to see exactly what crowdsec read (just to be aware, it will likely spam the logs a lot)
pschiffe
commented
4 months ago Hi @blotus, thank you for your reply. I do have multiple loki data sources configured, but none of them reads the same logs, the queries are distinct. You can see the acquis log files in the details in the issue description above.
Roughly there's 10 to 25 logs per second pushed to loki now in total, but crowdsec is reading maybe half of that.
I'll check the trace log if I can see something there.
pschiffe
commented
4 months ago Here's a trace log from single loki datasource: crowdsec_trace.txt
$ cat /etc/container-crowdsec/acquis.d/loki_mail.yaml
source: loki
log_level: trace
url: http://loki:3100/
headers:
X-Scope-OrgID: someorg
query: |
{container=~"mailcowdockerized-postfix-mailcow-1|mailcowdockerized-dovecot-mailcow-1"} | json | line_format `{{.MESSAGE}}`
labels:
type: syslogYou can try to search for a log line - connect from arsyline2.vshosting.cz[185.59.208.36] (some remote server), and you can see that it's there multiple times.
pschiffe
commented
4 months ago Let me know if you need anything else from my side.
LaurenceJJones
commented
4 months ago Let me know if you need anything else from my side.
Hi 👋🏻 we just need find time to replicate, currently we have internal projects and upcoming release of 1.6.1. If we don't have time before 1.6.1 we will dedicate time for 1.6.2
pschiffe
commented
4 months ago Hello, I have an update that may or may not be related. Roughly after a day or two of crowdsec running, the loki integration dies with:
time="2024-03-03T20:20:40Z" level=warning msg="loki is not available, will retry for 30s" component=lokiclient source="http://loki:3100/" type=loki
time="2024-03-03T20:21:13Z" level=error msg="loki didn't manage to recover after 30s, giving up" component=lokiclient source="http://loki:3100/" type=lokiHowever loki is working fine, receiving data and I can browse the logs in Grafana. After crowdsec restart, it works again for a couple of days and then dies...
Previously when it died, the metrics said that the loki acquisition read around 8 millions of lines.
LaurenceJJones
commented
4 months ago Hello, I have an update that may or may not be related. Roughly after a day or two of crowdsec running, the loki integration dies with:
time="2024-03-03T20:20:40Z" level=warning msg="loki is not available, will retry for 30s" component=lokiclient source="http://loki:3100/" type=loki time="2024-03-03T20:21:13Z" level=error msg="loki didn't manage to recover after 30s, giving up" component=lokiclient source="http://loki:3100/" type=lokiHowever loki is working fine, receiving data and I can browse the logs in Grafana. After crowdsec restart, it works again for a couple of days and then dies...
Previously when it died, the metrics said that the loki acquisition read around 8 millions of lines.
Hmm there is an incremental backoff features but personally I would not expect the acquisition to end if it has already made successful connections in the past since it knows the configuration is valid. 🤔
What happened?
For example, this IP was banned:
91.73.194.178. In the logs I only see:But the alert says there was 6 events:
This is causing me false positives with scenarios such as
crowdsecurity/postfix-spam,crowdsecurity/http-crawl-non_statics,LePresidente/http-generic-403-bf. Other scenarios are working fine.Acquisition for postfix logs:
What did you expect to happen?
Less events based on logs.
How can we reproduce it (as minimally and precisely as possible)?
See above.
Anything else we need to know?
No response
Crowdsec version
OS version
Enabled collections and parsers
Acquisition config
Config show
Prometheus metrics
Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.
No response