Closed grasshide closed 1 year ago
buixor
commented
2 years ago Hello,
Thanks for the issue ! Would you be able to provide more context so we can look into this ?
Please let us know :)
grasshide
commented
2 years ago Sure :)
is the I/O caused by crowdsec or the bouncer ? It is caused by crowdsec. The screenshot is from the crowdsec docker container. Yesterday I disabled the bouncer completely and already got another 1 GB i/o since then.
what are the log file size ? I don't parse any logs/ no logs are mounted in the container. I just started using crowdsec and wanted to leverage the crowd-collected malicious IP's in a first attempt.
how big is crowdsec's database ? 5,4M crowdsec.db
can you share your config.yaml if it's not a stock config It is the stock config
what tool did you use to collect the stats ? None. I currently use the cli to see how crowdsec is doing.
blotus
commented
2 years ago Hello,
That's really weird, if you haven't configured anything, the only writes crowdsec should perform are when the IPs are pulled from CAPI (to update the database) or when the bouncer queries LAPI (for the access logs mostly).
Regarding the metrics, we meant which tool provided you with the graphs ? (cscli does not know how to graph anything, and it would definitely not be that pretty :) and the metrics endpoint does not give anything related to the I/O as far as I can tell).
Do you see anything in crowdsec logs ? (so in your case docker logs ....)
grasshide
commented
2 years ago That's really weird, if you haven't configured anything, the only writes crowdsec should perform are when the IPs are pulled from CAPI (to update the database) or when the bouncer queries LAPI (for the access logs mostly).
Regarding the metrics, we meant which tool provided you with the graphs ? (cscli does not know how to graph anything, and it would definitely not be that pretty :) and the metrics endpoint does not give anything related to the I/O as far as I can tell).
That's Portainer (www.portainer.io). :)
Do you see anything in crowdsec logs ? (so in your case
docker logs ....) I don't see anything wrong here...time="25-04-2022 23:49:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 00:19:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 00:49:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 00:50:46" level=info msg="capi/community-blocklist : 0 explicit deletions" time="26-04-2022 00:51:31" level=info msg="crowdsecurity/community-blocklist : added 13358 entries, deleted 13338 entries (alert:92)" time="26-04-2022 01:19:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 01:32:35" level=info msg="flushed 1/71 alerts because they were created 7d ago or more" time="26-04-2022 01:49:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 02:19:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 02:49:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 02:50:46" level=info msg="capi/community-blocklist : 0 explicit deltions" time="26-04-2022 02:51:31" level=info msg="crowdsecurity/community-blocklist : added 13361 entries, deleted 13338 entries (alert:93)" time="26-04-2022 03:19:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 03:32:36" level=info msg="flushed 1/71 alerts because they were created 7d ago or more" time="26-04-2022 03:49:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 04:19:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 04:49:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 04:50:46" level=info msg="capi/community-blocklist : 0 explicit deletions" time="26-04-2022 04:51:31" level=info msg="crowdsecurity/community-blocklist : added 13369 entries, deleted 13343 entries (alert:94)" time="26-04-2022 05:19:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 05:32:36" level=info msg="flushed 1/71 alerts because they were created 7d ago or more" time="26-04-2022 05:49:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 06:19:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 06:49:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 06:50:46" level=info msg="capi/community-blocklist : 0 explicit deletions" time="26-04-2022 06:51:31" level=info msg="crowdsecurity/community-blocklist : added 13359 entries, deleted 13337 entries (alert:95)" time="26-04-2022 07:19:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 07:32:36" level=info msg="flushed 1/71 alerts because they were created 7d ago or more" time="26-04-2022 07:49:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 08:19:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 08:49:44" level=info msg="capi metrics: metrics sent successfully" time="26-04-2022 08:50:46" level=info msg="capi/community-blocklist : 0 explicit deletions" time="26-04-2022 08:51:31" level=info msg="crowdsecurity/community-blocklist : added 13366 entries, deleted 13347 entries (alert:96)"
I added the traefik log in the meantime to see if crowdsec needed some useful data to work. But that did not change anything.
# cat acquis.yaml
filenames:
- /var/log/traefik/*
labels:
type: traefik
---
filenames:
- /var/log/auth.log
- /var/log/syslog
labels:
type: syslogI have now 2 GB i/o for 16 hours runtime of the container.
LaurenceJJones
commented
2 years ago Hey! Going through open issues, how are you getting along with this? on the latest version are you seeing the same figures?
grasshide
commented
2 years ago Ran another test. It's still the same with the most current docker image. With over 5 GB I/O for almost no traffic (not even 4MB total on the network side of the container), there is certainly something very wrong.
docker-compose: ` crowdsec: image: crowdsecurity/crowdsec container_name: crowdsec environment:
traefik
crowdsec-bouncer: image: fbonalair/traefik-crowdsec-bouncer container_name: crowdsec-bouncer environment: CROWDSEC_BOUNCER_API_KEY: e5e6...1463 CROWDSEC_AGENT_HOST: crowdsec:8080 GIN_MODE: release networks:
LaurenceJJones
commented
2 years ago Even though you only see 4mb on eth0, the bouncer communication wont happen on this network. What are the stats within the docker network? cause the traefik bouncer queries per request and there is no cache from what I know.
EDIT: if you run docker stats whilst on root machine does it show same stats? Cause I dont know how portainer grabs these metrics as docker stats does
NET I/O | The amount of data the container has sent and received over its network interfaceHmmm reading into it eth0 is the first network the container is connected but if you container is connected to multiple it will only ever show the first.... https://github.com/portainer/portainer/issues/2736 What the configuration of traefik network?
LaurenceJJones
commented
2 years ago So tried to get a test environment up and running without portainer, then going to try a stack within portainer.
10 min nikto scanning
Portainer results
So the image once running has an internal ticket to get blocklist which should be only read/write operation since the rest are database transactions. Do you have any other crons to restart the container or exec a command inside?
For us to look deeper could you provide acquis.yaml and config.yaml from ./crowdsec/config
grasshide
commented
2 years ago Let it run for a day. It did not immediately go up for me either.
Am Fr., 4. Nov. 2022 um 13:32 Uhr schrieb Laurence Jones < @.***>:
So tried to get a test environment up and running without portainer, then going to try a stack within portainer. 10 min nikto scanning [image: image] https://user-images.githubusercontent.com/23139695/199973358-d611205c-cd99-4021-9ce0-811f2a458ab7.png
— Reply to this email directly, view it on GitHub https://github.com/crowdsecurity/crowdsec/issues/1476#issuecomment-1303436754, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABBKECKQJKB6J2WIN3ZMACDWGT65XANCNFSM5UE3PFVQ . You are receiving this because you authored the thread.Message ID: @.***>
LaurenceJJones
commented
2 years ago 👍🏻 Okay, I am leaving my environment to run with a scan every 5 mins to make it more like a live environment.
LaurenceJJones
commented
2 years ago So ran it since last spoke.
Browser crashed but here the stats
However I can explain the read operations on the latter picture. I was running watch -n 15 docker exec <id> cscli decisions list. Running this makes cscli read all configuration files so running this every 15 seconds would make it climb.
grasshide
commented
2 years ago So I ran it again. This time only for one hour. I did not even start the bouncer container. So the crowdsec container got no requests at all.
cat acquis.yaml
filenames:
- /var/log/traefik/*
labels:
type: traefik
---
filenames:
- /var/log/auth.log
- /var/log/syslog
labels:
type: syslogcat config.yaml
common:
daemonize: false
pid_dir: /var/run/
log_media: stdout
log_level: info
log_dir: /var/log/
working_dir: .
config_paths:
config_dir: /etc/crowdsec/
data_dir: /var/lib/crowdsec/data/
simulation_path: /etc/crowdsec/simulation.yaml
hub_dir: /etc/crowdsec/hub/
index_path: /etc/crowdsec/hub/.index.json
notification_dir: /etc/crowdsec/notifications/
plugin_dir: /usr/local/lib/crowdsec/plugins/
crowdsec_service:
acquisition_path: /etc/crowdsec/acquis.yaml
parser_routines: 1
plugin_config:
user: nobody
group: nobody
cscli:
output: human
db_config:
log_level: info
type: sqlite
db_path: /var/lib/crowdsec/data/crowdsec.db
#user:
#password:
#db_name:
#host:
#port:
flush:
max_items: 5000
max_age: 7d
api:
client:
insecure_skip_verify: false
credentials_path: /etc/crowdsec/local_api_credentials.yaml
server:
log_level: info
listen_uri: 0.0.0.0:8080
profiles_path: /etc/crowdsec/profiles.yaml
online_client: # Central API credentials (to push signals and receive bad IPs)
credentials_path: /etc/crowdsec/online_api_credentials.yaml
#credentials_path: /etc/crowdsec/online_api_credentials.yaml
# tls:
# cert_file: /etc/crowdsec/ssl/cert.pem
# key_file: /etc/crowdsec/ssl/key.pem
prometheus:
enabled: true
level: full
listen_addr: 0.0.0.0
listen_port: 6060The only thing strange to me is the prometheus config. I don't runn any prometheus on my system. That came with the default config.
grasshide
commented
2 years ago Just saw this spike while leaving the monitor running:
Currently still nothing connected. No bouncer, no monitoring, nothing.
Edit Here is the logfile:
time="05-11-2022 13:00:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:00:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 14.631074ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:01:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:01:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 13.072738ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:02:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:02:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 15.383846ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:03:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:03:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 13.767319ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:04:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:04:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 15.195959ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:05:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:05:38 UTC] \"POST /v1/watchers/login HTTP/1.1 200 108.257959ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:05:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:05:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 14.472844ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:06:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:06:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 18.753496ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:07:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:07:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 15.257476ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:07:38" level=info msg="capi metrics: metrics sent successfully"
time="05-11-2022 13:08:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:08:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 14.908636ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:08:44" level=info msg="capi/community-blocklist : 0 explicit deletions"
time="05-11-2022 13:09:37" level=info msg="crowdsecurity/community-blocklist : added 17376 entries, deleted 17295 entries (alert:103)"
time="05-11-2022 13:09:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:09:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 13.447132ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:10:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:10:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 15.238409ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:11:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:11:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 15.596767ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:12:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:12:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 12.711819ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:13:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:13:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 15.550116ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:14:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:14:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 15.626747ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:15:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:15:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 14.473281ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:16:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:16:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 14.725815ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:17:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:17:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 14.52024ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:18:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:18:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 14.922283ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:19:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:19:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 14.831311ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:20:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:20:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 14.397151ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:21:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:21:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 15.07662ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""
time="05-11-2022 13:22:38" level=info msg="127.0.0.1 - [Sat, 05 Nov 2022 12:22:38 UTC] \"GET /v1/heartbeat HTTP/1.1 200 15.050284ms \"crowdsec/v1.4.1-linux-e1954adc325baa9e3420c324caabd50b7074dd77\" \""So with the timestamps, it is the community blocklist. However it list shouldn't be this big. How many scenarios do you have installed?
cscli scenarios listClosing due to stale, in version 1.5 we have some IO improvements in the form of file acquisition. Re reading over the issue this should have no effect.
Describe the bug
Im worried about my SSD.
To Reproduce Install crowdsec on debian bullseye via docker and watch the stats. I use the following bouncer cotainer with traefik: fbonalair/traefik-crowdsec-bouncer.
Expected behavior much less i/o.
Screenshots see above.
Technical Information (please complete the following information):
Additional context The i/o grows even as I write this article for about 500 MB.