Closed Simbiat closed 1 month ago
github-actions[bot]
commented
3 months ago @Simbiat: Thanks for opening an issue, it is currently awaiting triage.
In the meantime, you can:
LaurenceJJones
commented
3 months ago Struggling to replicate the issue, could be an external factor (onedrive sync) or WSL specific.
The files are not created before the collection is installed, within the staging directory is everything for /etc/crowdsec the datafiles are within /var/lib/crowdsec/data/ so it could be this directory has some issue?
Logs within details:
Simbiat
commented
3 months ago Here are my logs
And then from the log file:
time="2024-07-08T11:56:05Z" level=warning msg="Machine is not allowed to synchronize decisions, you can enable it with `cscli console enable console_management`"
time="2024-07-08T11:56:05Z" level=warning msg="scenario list is empty, will not pull yet"
time="2024-07-08T11:56:05Z" level=error msg="open /var/lib/crowdsec/data/cloudflare_ips.txt: no such file or directory"
time="2024-07-08T11:56:05Z" level=error msg="open /var/lib/crowdsec/data/cloudflare_ip6s.txt: no such file or directory"
time="2024-07-08T11:56:05Z" level=error msg="open /var/lib/crowdsec/data/rdns_seo_bots.txt: no such file or directory"
time="2024-07-08T11:56:05Z" level=error msg="open /var/lib/crowdsec/data/rdns_seo_bots.regex: no such file or directory"
time="2024-07-08T11:56:05Z" level=error msg="open /var/lib/crowdsec/data/ip_seo_bots.txt: no such file or directory"
time="2024-07-08T11:56:05Z" level=warning msg="No matching files for pattern /var/log/syslog" type=file
time="2024-07-08T11:56:06Z" level=warning msg="/var/log/auth.log is a directory, ignoring it." type=fileI do not think it's related to OneDrive, it looks like you did not clear the data directory, because you have
crowdsec | updated rdns_seo_bots.txt
crowdsec | updated rdns_seo_bots.regex
crowdsec | updated ip_seo_bots.txt
crowdsec | time="2024-07-08T10:39:47Z" level=info msg="Enabled postoverflows: crowdsecurity/seo-bots-whitelist"
crowdsec | time="2024-07-08T10:39:47Z" level=info msg="crowdsecurity/cdn-whitelist: OK"
crowdsec | updated cloudflare_ips.txt
crowdsec | updated cloudflare_ip6s.txtin your logs, and I get similar entries only after 2nd launch of container. If data folder is empty at the time of container launch, you will get appropriate warnings.
LaurenceJJones
commented
3 months ago So here is my compose, I dont use volume I just specified a test folder which does not exist:
services:
crowdsec:
container_name: crowdsec
image: crowdsecurity/crowdsec:v1.6.2
restart: unless-stopped
environment:
GID: 1000
COLLECTIONS: crowdsecurity/caddy crowdsecurity/whitelist-good-actors crowdsecurity/http-cve crowdsecurity/http-dos crowdsecurity/base-http-scenarios crowdsecurity/appsec-crs crowdsecurity/appsec-generic-rules crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-wordpress crowdsecurity/wordpress crowdsecurity/discord-crawler-whitelist
volumes:
- ./test/:/var/lib/crowdsec/data/:rw
#Required to read syslog. This will be valid only on UNIX
- /var/log/:/var/log/:ro
#Caddy logs folder is RW, because we're also writing Crowdsec logs here
- ./logs:/usr/local/logs/:rw
security_opt:
- no-new-privileges=true
healthcheck:
test: [ "CMD", "cscli", "lapi", "status" ]
start_period: 120s
interval: 10s
timeout: 5s
retries: 3Disk check to see there is no test folder
root@bookworm:/tmp# ls -la
total 44
drwxrwxrwt 10 root root 4096 Jul 8 12:23 .
drwxr-xr-x 18 root root 4096 May 15 16:13 ..
drwxrwxrwt 2 root root 4096 Jul 8 10:35 .ICE-unix
drwxrwxrwt 2 root root 4096 Jul 8 10:35 .X11-unix
drwxrwxrwt 2 root root 4096 Jul 8 10:35 .XIM-unix
drwxrwxrwt 2 root root 4096 Jul 8 10:35 .font-unix
drwxr-xr-x 3 root root 4096 Jul 8 10:36 config
-rw-r--r-- 1 root root 0 Jul 8 10:37 config.yaml.local
-rw-r--r-- 1 root root 1046 Jul 8 10:48 docker-compose.yaml
drwxr-xr-x 2 root root 4096 Jul 8 10:36 logs
drwx------ 3 root root 4096 Jul 8 10:35 systemd-private-0640d25d1096422496910f954a1f546b-chrony.service-6lZV62
drwx------ 3 root root 4096 Jul 8 10:35 systemd-private-0640d25d1096422496910f954a1f546b-systemd-logind.service-DKpJ5ISame cant replicate: but what do you mean from And then from the log file: ? as all logs are printed to stdout unless your config.yaml.local has specified to print to a log file?
When the directory is empty you get logs stating it will download them
crowdsec | level=info msg="Downloaded /var/lib/crowdsec/data/thinkphp_cve_2018-20062.txt"
crowdsec | updated /var/lib/crowdsec/data/thinkphp_cve_2018-20062.txt
crowdsec | updated /var/lib/crowdsec/data/log4j2_cve_2021_44228.txt
crowdsec | level=info msg="Downloaded /var/lib/crowdsec/data/log4j2_cve_2021_44228.txt"
crowdsec | updated /var/lib/crowdsec/data/jira_cve_2021-26086.txt
crowdsec | level=info msg="Downloaded /var/lib/crowdsec/data/jira_cve_2021-26086.txt"
crowdsec | level=info msg="/etc/crowdsec/collections/http-cve.yaml already exists."
crowdsec | level=info msg="Enabled collections: crowdsecurity/http-cve"
crowdsec | level=info msg="Enabled crowdsecurity/http-cve"
crowdsec | level=info msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
crowdsec | installed crowdsecurity/http-cve
crowdsec | Running: cscli collections install "crowdsecurity/http-dos"
crowdsec | level=info msg="Enabled scenarios: crowdsecurity/http-dos-bypass-cache"
crowdsec | level=info msg="Enabled scenarios: crowdsecurity/http-dos-random-uri"
crowdsec | level=info msg="Enabled scenarios: crowdsecurity/http-dos-switching-ua"
crowdsec | level=info msg="Enabled scenarios: crowdsecurity/http-dos-invalid-http-versions"
crowdsec | level=info msg="Enabled collections: crowdsecurity/http-dos"
crowdsec | level=info msg="Enabled crowdsecurity/http-dos"
crowdsec | level=info msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
crowdsec | installed crowdsecurity/http-dos
crowdsec | Running: cscli collections install "crowdsecurity/base-http-scenarios"
crowdsec | level=info msg="Downloaded /var/lib/crowdsec/data/bad_user_agents.regex.txt"
crowdsec | updated /var/lib/crowdsec/data/bad_user_agents.regex.txt
crowdsec | updated /var/lib/crowdsec/data/http_path_traversal.txt
crowdsec | level=info msg="Downloaded /var/lib/crowdsec/data/http_path_traversal.txt"
crowdsec | level=info msg="Downloaded /var/lib/crowdsec/data/sensitive_data.txt"
crowdsec | updated /var/lib/crowdsec/data/sensitive_data.txt
crowdsec | level=info msg="Downloaded /var/lib/crowdsec/data/sqli_probe_patterns.txt"
crowdsec | updated /var/lib/crowdsec/data/sqli_probe_patterns.txt
crowdsec | updated /var/lib/crowdsec/data/xss_probe_patterns.txt
crowdsec | level=info msg="Downloaded /var/lib/crowdsec/data/xss_probe_patterns.txt"
crowdsec | level=info msg="Downloaded /var/lib/crowdsec/data/backdoors.txt"
crowdsec | updated /var/lib/crowdsec/data/backdoors.txt
crowdsec | updated /var/lib/crowdsec/data/admin_interfaces.txt
crowdsec | level=info msg="Downloaded /var/lib/crowdsec/data/admin_interfaces.txt"
crowdsec | level=info msg="Downloaded /var/lib/crowdsec/data/trendy_cves.txt"The wording updated is misleading it just means the file was modified.
Simbiat
commented
3 months ago My config.yaml.local is pointing to a file, yes. Interesting, that your logs do not have a line like
2024-07-08 14:55:49 hub/collections/crowdsecurity/whitelist-good-actors.yamlwhen configs are being copied. Some difference in the image? latest seems to be the same as 1.6.2, though, from what I see 🤔
LaurenceJJones
commented
3 months ago Managed to replicate the error by setting config.yaml.local to log to a file instead
time="2024-07-08T12:46:57Z" level=debug msg="adding expression any(File('cloudflare_ips.txt'), { IpInRange(evt.Overflow.Alert.Source.IP ,#)}) to whitelists" id=cool-wind name=crowdsecurity/cdn-whitelist stage=s01-whitelist
time="2024-07-08T12:46:57Z" level=debug msg="adding expression any(File('cloudflare_ip6s.txt'), { IpInRange(evt.Overflow.Alert.Source.IP ,#)}) to whitelists" id=cool-wind name=crowdsecurity/cdn-whitelist stage=s01-whitelist
time="2024-07-08T12:46:57Z" level=debug msg="init (folder:/var/lib/crowdsec/data) (file:cloudflare_ips.txt) (type:string)"
time="2024-07-08T12:46:57Z" level=error msg="open /var/lib/crowdsec/data/cloudflare_ips.txt: no such file or directory"
time="2024-07-08T12:46:57Z" level=debug msg="init (folder:/var/lib/crowdsec/data) (file:cloudflare_ip6s.txt) (type:string)"
time="2024-07-08T12:46:57Z" level=error msg="open /var/lib/crowdsec/data/cloudflare_ip6s.txt: no such file or directory"
time="2024-07-08T12:46:57Z" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/postoverflows/s01-whitelist/cdn-whitelist.yaml stage=s01-whitelist
time="2024-07-08T12:46:57Z" level=debug msg="loading parser file '{/etc/crowdsec/postoverflows/s01-whitelist/discord-crawler-whitelist.yaml s01-whitelist}'"
time="2024-07-08T12:46:57Z" level=debug msg="adding expression evt.Enriched.reverse_dns endsWith '.ptr.discord.com.' to whitelists" id=holy-sunset name=crowdsecurity/discord-crawler-whitelist stage=s01-whitelist
time="2024-07-08T12:46:57Z" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/postoverflows/s01-whitelist/discord-crawler-whitelist.yaml stage=s01-whitelist
time="2024-07-08T12:46:57Z" level=debug msg="loading parser file '{/etc/crowdsec/postoverflows/s01-whitelist/seo-bots-whitelist.yaml s01-whitelist}'"
time="2024-07-08T12:46:57Z" level=debug msg="adding expression any(File('rdns_seo_bots.txt'), { len(#) > 0 && evt.Enriched.reverse_dns endsWith #}) to whitelists" id=misty-snowflake name=crowdsecurity/seo-bots-whitelist stage=s01-whitelist
time="2024-07-08T12:46:57Z" level=debug msg="adding expression RegexpInFile(evt.Enriched.reverse_dns, 'rdns_seo_bots.regex') to whitelists" id=misty-snowflake name=crowdsecurity/seo-bots-whitelist stage=s01-whitelist
time="2024-07-08T12:46:57Z" level=debug msg="adding expression any(File('ip_seo_bots.txt'), { len(#) > 0 && IpInRange(evt.Overflow.Alert.Source.IP ,#)}) to whitelists" id=misty-snowflake name=crowdsecurity/seo-bots-whitelist stage=s01-whitelist
time="2024-07-08T12:46:57Z" level=debug msg="init (folder:/var/lib/crowdsec/data) (file:rdns_seo_bots.txt) (type:string)"
time="2024-07-08T12:46:57Z" level=error msg="open /var/lib/crowdsec/data/rdns_seo_bots.txt: no such file or directory"
time="2024-07-08T12:46:57Z" level=debug msg="init (folder:/var/lib/crowdsec/data) (file:rdns_seo_bots.regex) (type:regexp)"
time="2024-07-08T12:46:57Z" level=error msg="open /var/lib/crowdsec/data/rdns_seo_bots.regex: no such file or directory"
time="2024-07-08T12:46:57Z" level=debug msg="init (folder:/var/lib/crowdsec/data) (file:ip_seo_bots.txt) (type:string)"
time="2024-07-08T12:46:57Z" level=error msg="open /var/lib/crowdsec/data/ip_seo_bots.txt: no such file or directory"
time="2024-07-08T12:46:57Z" level=info msg="Loaded 1 parser nodes" file=/etc/crowdsec/postoverflows/s01-whitelist/seo-bots-whitelist.yaml stage=s01-whitelistlet me dig further
Simbiat
commented
3 months ago If it will help, my configs are in https://github.com/Simbiat/simbiat.ru/tree/master/config/crowdsec
LaurenceJJones
commented
3 months ago Right we managed to find the cause and it is indeed a bug ✨ thank you for reporting it, but TLDR; is when we decided to ship all the hub files using rsync there was an oversight in this, when the file exists in the /etc/crowdsec/hub directory it already believes it has been remotely download (which is not the case for docker since they are installed at build time). This then causes cscli to not bother downloading the file because it already thinks they exist when they dont. The reason the second time to container is brought back it knows to download them is cscli hub upgrade is ran as the first item and this then trigger all files to be downloaded.
Simbiat
commented
3 months ago That's somewhat similar to what I was wondering in relationship to
if [ ! -e "/etc/crowdsec/local_api_credentials.yaml" ] && [ ! -e "/etc/crowdsec/config.yaml" ]; thenthen. Documentation kind of suggests, that using a .local file is the way to go, instead of regular config.yaml. But since I do not have a config.yaml the entrypoint copies all yaml files from staging, including default config.yaml and default acquis.yaml.
While the former may not a big deal, the latter one can be - it has multiple acquisitions in 1 config (which is supposed to be deprecated), and they may not be needed even. Since I have my acquisitions in acquis.d folder, I have to have an empty acquis.yaml file, so that it does not get overwritten.
While this is a separate issue (I can create a ticket for that, if required), it may be coming from the same assumptions of how things will be setup in a container. While I have at least 2 ideas how to handle acquis.yaml, I do not know what other potential issues may be hidden here.
LaurenceJJones
commented
1 month ago Will be resolved as next release as off #3120
Classing issue as completed until then you can use 1.6.3 RC image tags
What happened?
When creating Docker container for the first time, you can see some warnings/errors in logs like
Not sure about the first one, but the rest seem strange, because these files are created in
stagingbefore we get to this point (or at least it looks that way), and copying of the files from there should be happening after collections installation, if I am readingdocker_start.shcorrectly. As result to truly utilize the container I am forced to restart it (or at the least the CrowdSec service). Technically, the same is required in case I update Docker Compose and add/remove some collections/parsers/configs. Need for restart may not be obvious, though, if the output if I doup -d, and the output from CrowdSec goes to container, instead of file. And technically, this should not even be required, since the service is not even up yet.What did you expect to happen?
The
should happen after components installation/removal, that is after
How can we reproduce it (as minimally and precisely as possible)?
Have a docker compose service like this:
Create and start the container
Anything else we need to know?
No response
Crowdsec version
OS version
Enabled collections and parsers
Acquisition config
Config show
Prometheus metrics
N/A
Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.
N/A