search

crowdsecurity / crowdsec

CrowdSec - the open-source and participative security solution offering crowdsourced protection against malicious IPs and access to the most advanced real-world CTI.
https://crowdsec.net
MIT License
8.85k stars 459 forks source link

docker-compose COLLECTIONS Environment Variable not managing installed collections #1987

Closed xenolithis closed 1 year ago

xenolithis commented 1 year ago

What happened?

Hello,

when trying to manage hub collections with the COLLECTIONS Environment Variable I am not seeing the installed collections updating as expected.

What did you expect to happen?

expecting all collections specified in environment to be installed on container start. In the compose(below) I am testing with LePresidente/jellyseerr

How can we reproduce it (as minimally and precisely as possible)?

compose file

---
services:
  crowdsec:
    image: docker.io/crowdsecurity/crowdsec:latest
    container_name: crowdsec
    environment:
      COLLECTIONS: "crowdsecurity/iptables LePresidente/jellyseerr"
      TZ: US/Eastern
      GID: 1000
      LEVEL_DEBUG: true
    volumes:
      - path_to_config:/etc/crowdsec:rw
      - path_to_data:/var/lib/crowdsec/data:rw
      - /var/log:/var/log/host:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro 
      - /var/log/journal:/run/log/journal:ro
    networks:
      - my-network
    restart: unless-stopped
    security_opt:
      - no-new-privileges=true
networks:
  my-network:
    external: true

Anything else we need to know?

checking for new collection after compose up

> cscli collections list

COLLECTIONS
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 Name                                  📦 Status   Version   Local Path                                           
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────
 crowdsecurity/base-http-scenarios     ✔️ enabled   0.6       /etc/crowdsec/collections/base-http-scenarios.yaml   
 crowdsecurity/http-cve                ✔️ enabled   1.7       /etc/crowdsec/collections/http-cve.yaml              
 crowdsecurity/iptables                ✔️ enabled   0.1       /etc/crowdsec/collections/iptables.yaml              
 crowdsecurity/linux                   ✔️ enabled   0.2       /etc/crowdsec/collections/linux.yaml                 
 crowdsecurity/nginx                   ✔️ enabled   0.2       /etc/crowdsec/collections/nginx.yaml                 
 crowdsecurity/sshd                    ✔️ enabled   0.2       /etc/crowdsec/collections/sshd.yaml                  
 crowdsecurity/whitelist-good-actors   ✔️ enabled   0.1       /etc/crowdsec/collections/whitelist-good-actors.yaml 
──────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Crowdsec version

``` > cscli version 2023/01/12 23:48:15 version: v1.4.4-8d0af73a4fb24ef8416e598d891c6e5e5501ede5 2023/01/12 23:48:15 Codename: alphaga 2023/01/12 23:48:15 BuildDate: 2023-01-05_10:41:00 2023/01/12 23:48:15 GoVersion: 1.19.4 2023/01/12 23:48:15 Platform: linux 2023/01/12 23:48:15 Constraint_parser: >= 1.0, <= 2.0 2023/01/12 23:48:15 Constraint_scenario: >= 1.0, < 3.0 2023/01/12 23:48:15 Constraint_api: v1 2023/01/12 23:48:15 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

``` > docker version Client: Docker Engine - Community Version: 20.10.22 API version: 1.41 Go version: go1.18.9 Git commit: 3a2c30b Built: Thu Dec 15 22:28:04 2022 OS/Arch: linux/amd64 Context: default Experimental: true Server: Docker Engine - Community Engine: Version: 20.10.22 API version: 1.41 (minimum version 1.12) Go version: go1.18.9 Git commit: 42c8b31 Built: Thu Dec 15 22:25:49 2022 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.14 GitCommit: 9ba4b250366a5ddde94bb7c9d1def331423aa323 runc: Version: 1.1.4 GitCommit: v1.1.4-0-g5fd4c4d docker-init: Version: 0.19.0 GitCommit: de40ad0 ```

Enabled collections and parsers

```console $ cscli hub list -o raw > cscli hub list -o raw crowdsecurity/base-http-scenarios,enabled,0.6,http common : scanners detection,collections crowdsecurity/http-cve,enabled,1.7,,collections crowdsecurity/iptables,enabled,0.1,iptables support : logs and port-scans detection scenarios,collections crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections crowdsecurity/nginx,enabled,0.2,nginx support : parser and generic http scenarios,collections crowdsecurity/sshd,enabled,0.2,sshd support : parser and brute-force detection,collections crowdsecurity/whitelist-good-actors,enabled,0.1,Good actors whitelists,collections crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/docker-logs,enabled,0.1,docker json logs parser,parsers crowdsecurity/geoip-enrich,enabled,0.2,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/http-logs,enabled,1.1,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers crowdsecurity/iptables-logs,enabled,0.3,Parse iptables drop logs,parsers crowdsecurity/nginx-logs,enabled,1.3,Parse nginx access and error logs,parsers crowdsecurity/sshd-logs,enabled,2.0,Parse openSSH logs,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers crowdsecurity/CVE-2022-26134,enabled,0.1,Detect CVE-2022-26134 exploits,scenarios crowdsecurity/CVE-2022-35914,enabled,0.1,Detect CVE-2022-35914 exploits,scenarios crowdsecurity/CVE-2022-37042,enabled,0.1,Detect CVE-2022-37042 exploits,scenarios crowdsecurity/CVE-2022-40684,enabled,0.2,Detect cve-2022-40684 exploitation attempts,scenarios crowdsecurity/CVE-2022-41082,enabled,0.3,Detect CVE-2022-41082 exploits,scenarios crowdsecurity/CVE-2022-41697,enabled,0.1,Detect CVE-2022-41697 enumeration,scenarios crowdsecurity/CVE-2022-42889,enabled,0.2,Detect CVE-2022-42889 exploits (Text4Shell),scenarios crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.4,Detect cve-2021-44228 exploitation attemps,scenarios crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.1,Detect cve-2020-5902 exploitation attemps,scenarios crowdsecurity/fortinet-cve-2018-13379,enabled,0.2,Detect cve-2018-13379 exploitation attemps,scenarios crowdsecurity/grafana-cve-2021-43798,enabled,0.1,Detect cve-2021-43798 exploitation attemps,scenarios crowdsecurity/http-backdoors-attempts,enabled,0.3,Detect attempt to common backdoors,scenarios crowdsecurity/http-bad-user-agent,enabled,0.7,Detect bad user-agents,scenarios crowdsecurity/http-crawl-non_statics,enabled,0.3,Detect aggressive crawl from single ip,scenarios crowdsecurity/http-cve-2021-41773,enabled,0.1,cve-2021-41773,scenarios crowdsecurity/http-cve-2021-42013,enabled,0.1,cve-2021-42013,scenarios crowdsecurity/http-generic-bf,enabled,0.3,Detect generic http brute force,scenarios crowdsecurity/http-open-proxy,enabled,0.3,Detect scan for open proxy,scenarios crowdsecurity/http-path-traversal-probing,enabled,0.2,Detect path traversal attempt,scenarios crowdsecurity/http-probing,enabled,0.2,Detect site scanning/probing from a single ip,scenarios crowdsecurity/http-sensitive-files,enabled,0.2,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios crowdsecurity/http-sqli-probing,enabled,0.2,A scenario that detects SQL injection probing with minimal false positives,scenarios crowdsecurity/http-xss-probing,enabled,0.2,A scenario that detects XSS probing with minimal false positives,scenarios crowdsecurity/iptables-scan-multi_ports,enabled,0.1,ban IPs that are scanning us,scenarios crowdsecurity/jira_cve-2021-26086,enabled,0.1,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios crowdsecurity/nginx-req-limit-exceeded,enabled,0.1,Detects IPs which violate nginx's user set request limit.,scenarios crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.2,Detect cve-2019-11510 exploitation attemps,scenarios crowdsecurity/spring4shell_cve-2022-22965,enabled,0.2,Detect cve-2022-22965 probing,scenarios crowdsecurity/ssh-bf,enabled,0.1,Detect ssh bruteforce,scenarios crowdsecurity/ssh-slow-bf,enabled,0.2,Detect slow ssh bruteforce,scenarios crowdsecurity/thinkphp-cve-2018-20062,enabled,0.3,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios crowdsecurity/vmware-cve-2022-22954,enabled,0.2,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.1,Detect VMSA-2021-0027 exploitation attemps,scenarios ltsich/http-w00tw00t,enabled,0.1,detect w00tw00t,scenarios crowdsecurity/cdn-whitelist,enabled,0.4,Whitelist CDN providers,postoverflows crowdsecurity/rdns,enabled,0.2,Lookup the DNS associated to the source IP only for overflows,postoverflows crowdsecurity/seo-bots-whitelist,enabled,0.4,Whitelist good search engine crawlers,postoverflows ```

Acquisition config

```console > cat acquis.yaml filenames: - /var/log/nginx/*.log - ./tests/nginx/nginx.log - /var/log/swag/* labels: type: nginx --- filenames: - /var/log/auth.log - /var/log/syslog - /var/log/host/auth.log* labels: type: syslog --- filename: /var/log/apache2/*.log labels: type: apache2 --- source: docker container_name: - jellyseerr labels: type: jellyseerr ```

Config show

```console > cscli config show Global: - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log/ - Log level : info - Log Media : stdout Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 cscli: - Output : human - Hub Branch : - Hub Folder : /etc/crowdsec/hub Local API Server: - Listen URL : 0.0.0.0:8080 - Profile File : /etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : sqlite - Path : /var/lib/crowdsec/data/crowdsec.db - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console > cscli metrics Acquisition Metrics: ╭───────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────╮ │ Source │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │ ├───────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┤ │ docker:jellyseerr │ 142 │ - │ 142 │ - │ │ file:/var/log/host/auth.log │ 2 │ - │ 2 │ - │ │ file:/var/log/swag/access.log │ 4 │ 4 │ - │ - │ ╰───────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────╯ Parser Metrics: ╭─────────────────────────────────┬──────┬────────┬──────────╮ │ Parsers │ Hits │ Parsed │ Unparsed │ ├─────────────────────────────────┼──────┼────────┼──────────┤ │ child-crowdsecurity/http-logs │ 12 │ 8 │ 4 │ │ child-crowdsecurity/nginx-logs │ 4 │ 4 │ - │ │ child-crowdsecurity/syslog-logs │ 2 │ 2 │ - │ │ crowdsecurity/dateparse-enrich │ 4 │ 4 │ - │ │ crowdsecurity/geoip-enrich │ 4 │ 4 │ - │ │ crowdsecurity/http-logs │ 4 │ 4 │ - │ │ crowdsecurity/nginx-logs │ 4 │ 4 │ - │ │ crowdsecurity/non-syslog │ 146 │ 146 │ - │ │ crowdsecurity/syslog-logs │ 2 │ 2 │ - │ │ crowdsecurity/whitelists │ 4 │ 4 │ - │ ╰─────────────────────────────────┴──────┴────────┴──────────╯ Local Api Metrics: ╭────────────────────┬────────┬──────╮ │ Route │ Method │ Hits │ ├────────────────────┼────────┼──────┤ │ /v1/decisions │ GET │ 4 │ │ /v1/heartbeat │ GET │ 20 │ │ /v1/watchers/login │ POST │ 2 │ ╰────────────────────┴────────┴──────╯ Local Api Machines Metrics: ╭─────────┬───────────────┬────────┬──────╮ │ Machine │ Route │ Method │ Hits │ ├─────────┼───────────────┼────────┼──────┤ │ thor │ /v1/heartbeat │ GET │ 20 │ ╰─────────┴───────────────┴────────┴──────╯ Local Api Bouncers Metrics: ╭─────────┬───────────────┬────────┬──────╮ │ Bouncer │ Route │ Method │ Hits │ ├─────────┼───────────────┼────────┼──────┤ │ swag │ /v1/decisions │ GET │ 4 │ ╰─────────┴───────────────┴────────┴──────╯ Local Api Bouncers Decisions: ╭─────────┬───────────────┬───────────────────╮ │ Bouncer │ Empty answers │ Non-empty answers │ ├─────────┼───────────────┼───────────────────┤ │ swag │ 4 │ 0 │ ╰─────────┴───────────────┴───────────────────╯ Local Api Decisions: ╭────────────────────────────────────────────┬──────────┬────────┬───────╮ │ Reason │ Origin │ Action │ Count │ ├────────────────────────────────────────────┼──────────┼────────┼───────┤ │ crowdsecurity/http-probing │ CAPI │ ban │ 2886 │ │ crowdsecurity/http-sensitive-files │ CAPI │ ban │ 84 │ │ crowdsecurity/thinkphp-cve-2018-20062 │ CAPI │ ban │ 31 │ │ crowdsecurity/CVE-2022-26134 │ CAPI │ ban │ 1 │ │ crowdsecurity/http-crawl-non_statics │ CAPI │ ban │ 391 │ │ crowdsecurity/http-cve-2021-41773 │ CAPI │ ban │ 38 │ │ crowdsecurity/ssh-slow-bf │ CAPI │ ban │ 3899 │ │ crowdsecurity/vmware-cve-2022-22954 │ CAPI │ ban │ 3 │ │ ltsich/http-w00tw00t │ CAPI │ ban │ 3 │ │ crowdsecurity/http-bad-user-agent │ CAPI │ ban │ 3395 │ │ crowdsecurity/http-bad-user-agent │ crowdsec │ ban │ 1 │ │ crowdsecurity/http-generic-bf │ CAPI │ ban │ 8 │ │ crowdsecurity/nginx-req-limit-exceeded │ CAPI │ ban │ 312 │ │ crowdsecurity/http-path-traversal-probing │ CAPI │ ban │ 105 │ │ crowdsecurity/jira_cve-2021-26086 │ CAPI │ ban │ 35 │ │ crowdsecurity/ssh-bf │ CAPI │ ban │ 9986 │ │ crowdsecurity/CVE-2022-35914 │ CAPI │ ban │ 2 │ │ crowdsecurity/apache_log4j2_cve-2021-44228 │ CAPI │ ban │ 17 │ │ crowdsecurity/f5-big-ip-cve-2020-5902 │ CAPI │ ban │ 4 │ │ crowdsecurity/http-open-proxy │ CAPI │ ban │ 137 │ │ crowdsecurity/spring4shell_cve-2022-22965 │ CAPI │ ban │ 1 │ │ crowdsecurity/fortinet-cve-2018-13379 │ CAPI │ ban │ 37 │ │ crowdsecurity/http-backdoors-attempts │ CAPI │ ban │ 91 │ │ crowdsecurity/http-cve-2021-42013 │ CAPI │ ban │ 1 │ ╰────────────────────────────────────────────┴──────────┴────────┴───────╯ Local Api Alerts: ╭───────────────────────────────────────┬───────╮ │ Reason │ Count │ ├───────────────────────────────────────┼───────┤ │ crowdsecurity/CVE-2022-41082 │ 3 │ │ crowdsecurity/fortinet-cve-2018-13379 │ 1 │ │ crowdsecurity/http-bad-user-agent │ 14 │ │ crowdsecurity/http-open-proxy │ 3 │ │ manual 'ban' from 'host' │ 1 │ ╰───────────────────────────────────────┴───────╯ ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

github-actions[bot] commented 1 year ago

@xenolithis: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.
xenolithis commented 1 year ago

manually adding collections is successful cscli collections install LePresidente/jellyseerr

LaurenceJJones commented 1 year ago

Fix has been released a v1.4.5 tag, however, latest does not point to this. So if you are experiencing this issue change tag to v.1.4.5

mmetc commented 1 year ago

Fix has been released a v1.4.5 tag, however, latest does not point to this. So if you are experiencing this issue change tag to v.1.4.5

Latest, slim and latest-debian have been updated including latest on ghcr.io

xenolithis commented 1 year ago

can confirm that latest tag has the change. Thanks for resolving this!