search

crowdsecurity / crowdsec

CrowdSec - the open-source and participative security solution offering crowdsourced protection against malicious IPs and access to the most advanced real-world CTI.
https://crowdsec.net
MIT License
8.85k stars 459 forks source link

/etc/crowdsec/profile: no such file or directory #2054

Closed Brettdah closed 1 year ago

Brettdah commented 1 year ago

What happened?

As suggested to the contributing lines I'm openning this issue, I already sent a PR to Fix it but it was closed automaticaly? or without any human comment.

I have found a bug while bind mounting my config.yml on a fresh deployment of crowdsec via docker compose with this compose file :

version: '3.8'

networks:
  proxy:
    external: true

services:
  crowdsec:
    image: crowdsecurity/crowdsec:v1.4.5-slim
    #restart: unless-stopped
    environment:
      TZ: Europe/Paris
      GID: "${GID-1000}"
      LEVEL_INFO: "true"
      COLLECTIONS: "crowdsecurity/linux crowdsecurity/traefik LePresidente/authelia Dominic-Wagner/vaultwarden"
    volumes:
      - /etc/timezone:/etc/timezone:ro
      - crowdsec-db:/var/lib/crowdsec/data/
      - crowdsec-config:/etc/crowdsec/
      - $PWD/data/etc/crowdsec/acquisitions:/etc/crowdsec/acquisitions
      - $PWD/data/etc/crowdsec/config.yaml:/etc/crowdsec/config.yaml
    networks:
      - proxy

volumes:
  crowdsec-db:
  crowdsec-config:

What did you expect to happen?

the container should start with no errors

How can we reproduce it (as minimally and precisely as possible)?

Just use my compose file and add the acquisition file ssh.yml in the directory I'm using for any of my acquis config, it look messy to have more than 1 config in one file:

---
# SSH context
filenames:
  - /var/log/dockerhost/auth.log
labels:
  type: syslog

Then depending on you docker and especialy docker compose installation:

# if docker compose is installed as a plugin
docker compose up -d
# else
docker-compose up -d
#or if using swarm
docker stack deploy -c docker-compose.yml stack-securiy

Anything else we need to know?

when I bind mount the docker-start.sh with this change:

original:

# Check and prestage /etc/crowdsec
if [ ! -e "/etc/crowdsec/local_api_credentials.yaml" ] && [ ! -e "/etc/crowdsec/config.yaml" ]; then
    echo "Populating configuration directory..."
    # don't overwrite existing configuration files, which may come
    # from bind-mount or even be read-only (configmaps)
    if [ -e /staging/etc/crowdsec ]; then
        mkdir -p /etc/crowdsec/
        # if you change this, check that it still works
        # under alpine and k8s, with and without tls
        cp -an /staging/etc/crowdsec/* /etc/crowdsec/
    fi
fi

mine

echo "Populating configuration directory..."
# don't overwrite existing configuration files, which may come
# from bind-mount or even be read-only (configmaps)
if [ -e /staging/etc/crowdsec ]; then
    mkdir -p /etc/crowdsec/
    # if you change this, check that it still works
    # under alpine and k8s, with and without tls
    cp -an /staging/etc/crowdsec/* /etc/crowdsec/
fi

the container start ! why check if the file exists before the copy, with the option -n (--no-cobbler) the existing files will not be replaced... I opened the PR 2035 as suggested by Loz on discord but it was closed... :(

Crowdsec version

```console $ cscli version 2023/02/11 17:09:24 version: v1.4.5-a9a2186a76af63551352aa3bc296bdbe80ca4893 2023/02/11 17:09:24 Codename: alphaga 2023/02/11 17:09:24 BuildDate: 2023-01-25_08:57:05 2023/02/11 17:09:24 GoVersion: 1.19.5 2023/02/11 17:09:24 Platform: linux 2023/02/11 17:09:24 Constraint_parser: >= 1.0, <= 2.0 2023/02/11 17:09:24 Constraint_scenario: >= 1.0, < 3.0 2023/02/11 17:09:24 Constraint_api: v1 2023/02/11 17:09:24 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

```console $ cat /etc/os-release NAME="Ubuntu" VERSION="20.04.5 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.5 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=focal UBUNTU_CODENAME=focal $ uname -r Linux dockerhost 5.4.0-139-generic #156-Ubuntu SMP Fri Jan 20 17:27:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ``` Tried on a different machine with ubuntu 22.04 fresh install same problem

Enabled collections and parsers

```console $ cscli hub list -o raw Dominic-Wagner/vaultwarden,enabled,0.1,Vaultwarden support : parser and brute-force detection,collections LePresidente/authelia,enabled,0.2,Authelia Support : parser and brute-force detection,collections crowdsecurity/base-http-scenarios,enabled,0.6,http common : scanners detection,collections crowdsecurity/http-cve,enabled,1.9,,collections crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections crowdsecurity/sshd,enabled,0.2,sshd support : parser and brute-force detection,collections crowdsecurity/traefik,enabled,0.1,traefik support: parser and generic http scenarios,collections Dominic-Wagner/vaultwarden-logs,enabled,0.1,Parse vaultwarden logs,parsers LePresidente/authelia-logs,enabled,0.3,Parse Authelia logs,parsers crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/docker-logs,enabled,0.1,docker json logs parser,parsers crowdsecurity/geoip-enrich,enabled,0.2,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/http-logs,enabled,1.1,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers crowdsecurity/sshd-logs,enabled,2.0,Parse openSSH logs,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers crowdsecurity/traefik-logs,enabled,0.5,Parse Traefik access logs,parsers crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers Dominic-Wagner/vaultwarden-bf,enabled,0.1,Detect vaultwarden bruteforce,scenarios LePresidente/authelia-bf,enabled,0.2,Detect authelia bruteforce,scenarios crowdsecurity/CVE-2022-26134,enabled,0.1,Detect CVE-2022-26134 exploits,scenarios crowdsecurity/CVE-2022-35914,enabled,0.1,Detect CVE-2022-35914 exploits,scenarios crowdsecurity/CVE-2022-37042,enabled,0.1,Detect CVE-2022-37042 exploits,scenarios crowdsecurity/CVE-2022-40684,enabled,0.2,Detect cve-2022-40684 exploitation attempts,scenarios crowdsecurity/CVE-2022-41082,enabled,0.3,Detect CVE-2022-41082 exploits,scenarios crowdsecurity/CVE-2022-41697,enabled,0.1,Detect CVE-2022-41697 enumeration,scenarios crowdsecurity/CVE-2022-42889,enabled,0.2,Detect CVE-2022-42889 exploits (Text4Shell),scenarios crowdsecurity/CVE-2022-44877,enabled,0.2,Detect CVE-2022-44877 exploits,scenarios crowdsecurity/CVE-2022-46169,enabled,0.1,Detect CVE-2022-46169 brute forcing,scenarios crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.4,Detect cve-2021-44228 exploitation attemps,scenarios crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.1,Detect cve-2020-5902 exploitation attemps,scenarios crowdsecurity/fortinet-cve-2018-13379,enabled,0.2,Detect cve-2018-13379 exploitation attemps,scenarios crowdsecurity/grafana-cve-2021-43798,enabled,0.1,Detect cve-2021-43798 exploitation attemps,scenarios crowdsecurity/http-backdoors-attempts,enabled,0.3,Detect attempt to common backdoors,scenarios crowdsecurity/http-bad-user-agent,enabled,0.7,Detect bad user-agents,scenarios crowdsecurity/http-crawl-non_statics,enabled,0.3,Detect aggressive crawl from single ip,scenarios crowdsecurity/http-cve-2021-41773,enabled,0.1,cve-2021-41773,scenarios crowdsecurity/http-cve-2021-42013,enabled,0.1,cve-2021-42013,scenarios crowdsecurity/http-generic-bf,enabled,0.4,Detect generic http brute force,scenarios crowdsecurity/http-open-proxy,enabled,0.3,Detect scan for open proxy,scenarios crowdsecurity/http-path-traversal-probing,enabled,0.2,Detect path traversal attempt,scenarios crowdsecurity/http-probing,enabled,0.2,Detect site scanning/probing from a single ip,scenarios crowdsecurity/http-sensitive-files,enabled,0.2,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios crowdsecurity/http-sqli-probing,enabled,0.2,A scenario that detects SQL injection probing with minimal false positives,scenarios crowdsecurity/http-xss-probing,enabled,0.2,A scenario that detects XSS probing with minimal false positives,scenarios crowdsecurity/jira_cve-2021-26086,enabled,0.1,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.2,Detect cve-2019-11510 exploitation attemps,scenarios crowdsecurity/spring4shell_cve-2022-22965,enabled,0.2,Detect cve-2022-22965 probing,scenarios crowdsecurity/ssh-bf,enabled,0.1,Detect ssh bruteforce,scenarios crowdsecurity/ssh-slow-bf,enabled,0.2,Detect slow ssh bruteforce,scenarios crowdsecurity/thinkphp-cve-2018-20062,enabled,0.3,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios crowdsecurity/vmware-cve-2022-22954,enabled,0.2,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.1,Detect VMSA-2021-0027 exploitation attemps,scenarios ltsich/http-w00tw00t,enabled,0.1,detect w00tw00t,scenarios ```

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml => does not exist $ ls /etc/crowdsec/acquisitions/ docker.yml ssh.yml $ cat /etc/crowdsec/acquisitions/* --- # Docker context source : docker container_name_regexp: - ^[a-zA-Z0-9.-]*$ labels: type: log_type --- # SSH context filenames: - /var/log/dockerhost/auth.log labels: type: syslog ```

Config show

```console $ cscli config show Global: - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log/ - Log level : info - Log Media : stdout Crowdsec: - Acquisition File : - Parsers routines : 1 - Acquisition Folder : /etc/crowdsec/acquisitions/ cscli: - Output : human - Hub Branch : - Hub Folder : /etc/crowdsec/hub Local API Server: - Listen URL : 0.0.0.0:8080 - Profile File : /etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : sqlite - Path : /var/lib/crowdsec/data/crowdsec.db - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

The problem is not linked to metrics ```console $ cscli metrics # paste output here ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

github-actions[bot] commented 1 year ago

@Brettdah: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.
mmetc commented 1 year ago

Hi @Brettdah

I don't know or remember why the PR was closed, but I planned to fix the issue after landing the functional tests for docker containers in the master branch (directory docker/tests)

I'll have a look and get back to you tomorrow. Thanks!

Brettdah commented 1 year ago

it seems it was my mistake I'm not used to work with public repos, in the company I was when I submited the PR I was deleting the branch as soon as I pushed sent the PR (partly because I was the one to accept the PR too) so that's why it was claused xD

so yeah, PEBCAK on me^^

as I said the with the cp -n there should not be any problem but if I were you I would build the image with the /etc/crowdsec already filled with what's in the /staging/etc/crowdsec/ the bing directive will put any personal config on top of your default one and take precedence over the default file present in the Image (hope I'm clear enough, english is not my first language)

thanks for your reply ! waiting to read you soon :)

mmetc commented 1 year ago

Hi @Brettdah

To make it short, the proper way to achieve what you need is to

  1. Mount a file /etc/crowdsec/config.yaml.local containing only the following
crowdsec_service:
  acquisition_dir: /etc/crowdsec/acquisitions

and any other configuration that you many need to change over the default one. This way you don't need to mount config.yaml to the container anymore.

  1. There is no 2. You don't need to change the image or anything else. But it's better to name the directory "acquis.d" instead of "acquisitions" because it's the default name and it will be configured by default in the next version so you won't need the .local file in 1.5.

The .local and acquis.d features have been there for over a year, it's just that the docker documentation was not updated to take advantage of them.

Let us know, and thanks again