OneDrive vs Geolite2: Name or type not allowed

Simbiat commented 2 months ago

What happened?

I have a Windows host machine running Docker Desktop with WSL2. When running Crowdsec container and mounting a path that's a OneDrive path in Windows, OneDrive starts complaining about GeoLite2-ASN.mmdb and GeoLite2-City.mmdb saying name or type not allowed. It complains because the files are symlinks, and from what I understand OneDrive can be picky about those. In this case it would be doubly picky, since the links are to WSL areas, which OneDrive clearly can't access.

What did you expect to happen?

Files should be accepted by OneDrive normally

How can we reproduce it (as minimally and precisely as possible)?

Have a Docker Compose service like

  crowdsec:
    container_name: crowdsec
    image: crowdsecurity/crowdsec:latest
    restart: unless-stopped
    environment:
      GID: 1000
      COLLECTIONS: crowdsecurity/caddy crowdsecurity/whitelist-good-actors crowdsecurity/appsec-crs
      BOUNCER_KEY_CADDY: ${CROWDSEC_API_KEY}
    networks:
      - webserver
    volumes:
      - ${CROWDSEC_DATA_DIR}:/var/lib/crowdsec/data/:rw
      - ./config/crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml:rw
      - ./logs:/var/log/caddy:ro
    security_opt:
      - no-new-privileges=true
    healthcheck:
      test: [ "CMD", "cscli", "version" ]
      start_period: 2m
      interval: 10s
      timeout: 5s
      retries: 3

with ${CROWDSEC_DATA_DIR} being a OneDrive folder. Start the service and wait for the files to be created and OneDrive to try syncing them.

Anything else we need to know?

This is obviously not a bug, but it would be nice to find a way to work around t his, because as is I have to remove the links every time. To be honest, I am unsure what's the point of using symlinks here, since the files are still within the same container in non-persistent storage, thus it seems reasonable to just use files from /staging/var/lib/crowdsec/data

Crowdsec version

```console version: v1.6.2-16bfab86 Codename: alphaga BuildDate: 2024-06-05_14:25:55 GoVersion: 1.22.3 Platform: docker libre2: C++ User-Agent: crowdsec/v1.6.2-16bfab86-docker Constraint_parser: >= 1.0, <= 3.0 Constraint_scenario: >= 1.0, <= 3.0 Constraint_api: v1 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

```console BuildNumber Caption OSArchitecture Version 22631 Microsoft Windows 11 Pro 64-bit 10.0.22631 ```

Enabled collections and parsers

```console name,status,version,description,type crowdsecurity/appsec-logs,enabled,0.5,Parse Appsec events,parsers crowdsecurity/caddy-logs,enabled,0.8,Parse caddy logs,parsers crowdsecurity/cri-logs,enabled,0.1,CRI logging format parser,parsers crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/docker-logs,enabled,0.1,docker json logs parser,parsers crowdsecurity/geoip-enrich,enabled,0.3,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/http-logs,enabled,1.2,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers crowdsecurity/sshd-logs,enabled,2.3,Parse openSSH logs,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers crowdsecurity/cdn-whitelist,enabled,0.4,Whitelist CDN providers,postoverflows crowdsecurity/rdns,enabled,0.3,Lookup the DNS associated to the source IP only for overflows,postoverflows crowdsecurity/seo-bots-whitelist,enabled,0.5,Whitelist good search engine crawlers,postoverflows crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.6,Detect cve-2021-44228 exploitation attemps,scenarios crowdsecurity/CVE-2017-9841,enabled,0.2,Detect CVE-2017-9841 exploits,scenarios crowdsecurity/CVE-2019-18935,enabled,0.2,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios crowdsecurity/CVE-2022-26134,enabled,0.2,Detect CVE-2022-26134 exploits,scenarios crowdsecurity/CVE-2022-35914,enabled,0.2,Detect CVE-2022-35914 exploits,scenarios crowdsecurity/CVE-2022-37042,enabled,0.2,Detect CVE-2022-37042 exploits,scenarios crowdsecurity/CVE-2022-40684,enabled,0.3,Detect cve-2022-40684 exploitation attempts,scenarios crowdsecurity/CVE-2022-41082,enabled,0.4,Detect CVE-2022-41082 exploits,scenarios crowdsecurity/CVE-2022-41697,enabled,0.2,Detect CVE-2022-41697 enumeration,scenarios crowdsecurity/CVE-2022-42889,enabled,0.3,Detect CVE-2022-42889 exploits (Text4Shell),scenarios crowdsecurity/CVE-2022-44877,enabled,0.3,Detect CVE-2022-44877 exploits,scenarios crowdsecurity/CVE-2022-46169,enabled,0.2,Detect CVE-2022-46169 brute forcing,scenarios crowdsecurity/CVE-2023-22515,enabled,0.1,Detect CVE-2023-22515 exploitation,scenarios crowdsecurity/CVE-2023-22518,enabled,0.2,Detect CVE-2023-22518 exploits,scenarios crowdsecurity/CVE-2023-49103,enabled,0.3,Detect owncloud CVE-2023-49103 exploitation attempts,scenarios crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.2,Detect cve-2020-5902 exploitation attemps,scenarios crowdsecurity/fortinet-cve-2018-13379,enabled,0.3,Detect cve-2018-13379 exploitation attemps,scenarios crowdsecurity/grafana-cve-2021-43798,enabled,0.2,Detect cve-2021-43798 exploitation attemps,scenarios crowdsecurity/http-admin-interface-probing,enabled,0.4,Detect generic HTTP admin interface probing,scenarios crowdsecurity/http-backdoors-attempts,enabled,0.6,Detect attempt to common backdoors,scenarios crowdsecurity/http-bad-user-agent,enabled,1.2,Detect usage of bad User Agent,scenarios crowdsecurity/http-crawl-non_statics,enabled,0.7,Detect aggressive crawl on non static resources,scenarios crowdsecurity/http-cve-2021-41773,enabled,0.2,cve-2021-41773,scenarios crowdsecurity/http-cve-2021-42013,enabled,0.2,cve-2021-42013,scenarios crowdsecurity/http-cve-probing,enabled,0.2,Detect generic HTTP cve probing,scenarios crowdsecurity/http-generic-bf,enabled,0.6,Detect generic http brute force,scenarios crowdsecurity/http-open-proxy,enabled,0.5,Detect scan for open proxy,scenarios crowdsecurity/http-path-traversal-probing,enabled,0.4,Detect path traversal attempt,scenarios crowdsecurity/http-probing,enabled,0.4,Detect site scanning/probing from a single ip,scenarios crowdsecurity/http-sensitive-files,enabled,0.4,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios crowdsecurity/http-sqli-probing,enabled,0.4,A scenario that detects SQL injection probing with minimal false positives,scenarios crowdsecurity/http-wordpress-scan,enabled,0.2,Detect WordPress scan: vuln hunting,scenarios crowdsecurity/http-xss-probing,enabled,0.4,A scenario that detects XSS probing with minimal false positives,scenarios crowdsecurity/jira_cve-2021-26086,enabled,0.3,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios crowdsecurity/netgear_rce,enabled,0.3,Detect Netgear RCE DGN1000/DGN220 exploitation attempts,scenarios crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.3,Detect cve-2019-11510 exploitation attemps,scenarios crowdsecurity/spring4shell_cve-2022-22965,enabled,0.3,Detect cve-2022-22965 probing,scenarios crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios crowdsecurity/thinkphp-cve-2018-20062,enabled,0.6,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios crowdsecurity/vmware-cve-2022-22954,enabled,0.3,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.2,Detect VMSA-2021-0027 exploitation attemps,scenarios ltsich/http-w00tw00t,enabled,0.2,detect w00tw00t,scenarios crowdsecurity/bf_base,enabled,0.1,,contexts crowdsecurity/http_base,enabled,0.2,,contexts crowdsecurity/crs,enabled,0.2,,appsec-configs crowdsecurity/crs,enabled,0.4,,appsec-rules crowdsecurity/appsec-crs,enabled,0.4,Appsec: Modsecurity core rule set rules,collections crowdsecurity/base-http-scenarios,enabled,1.0,http common : scanners detection,collections crowdsecurity/caddy,enabled,0.1,caddy support : parser and generic http scenarios,collections crowdsecurity/http-cve,enabled,2.6,Detect CVE exploitation in http logs,collections crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections crowdsecurity/sshd,enabled,0.3,sshd support : parser and brute-force detection,collections crowdsecurity/whitelist-good-actors,enabled,0.1,Good actors whitelists,collections ```

Acquisition config

```console filenames: - /var/log/caddy/access.log labels: type: caddy ```

Config show

```console Global: - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log - Log level : info - Log Media : stdout Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /etc/crowdsec/acquis.d cscli: - Output : human - Hub Branch : API Client: - URL : http://0.0.0.0:8080/ - Login : localhost - Credentials File : /etc/crowdsec/local_api_credentials.yaml Local API Server: - Listen URL : 0.0.0.0:8080 - Listen Socket : - Profile File : /etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : sqlite - Path : /var/lib/crowdsec/data/crowdsec.db - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console Local API Metrics: ╭────────────────────┬────────┬──────╮ │ Route │ Method │ Hits │ ├────────────────────┼────────┼──────┤ │ /v1/heartbeat │ GET │ 130 │ │ /v1/watchers/login │ POST │ 3 │ ╰────────────────────┴────────┴──────╯ Local API Machines Metrics: ╭───────────┬───────────────┬────────┬──────╮ │ Machine │ Route │ Method │ Hits │ ├───────────┼───────────────┼────────┼──────┤ │ localhost │ /v1/heartbeat │ GET │ 130 │ ╰───────────┴───────────────┴────────┴──────╯ ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

No response

github-actions[bot] commented 2 months ago

@Simbiat: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

Check Crowdsec Documentation to see if your issue can be self resolved.
You can also join our Discord.
Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.

Simbiat commented 2 months ago

Adding DISABLE_PARSERS: crowdsecurity/geoip-enrich to compose does not help: the links are still created on every start. Maybe

ln -s "/staging/var/lib/crowdsec/data/$geodb" /var/lib/crowdsec/data/

can be replaced with something like this?

if istrue "$GEODB_COPY"; then
   cp -f "/staging/var/lib/crowdsec/data/$geodb" /var/lib/crowdsec/data/
else
    ln -s "/staging/var/lib/crowdsec/data/$geodb" /var/lib/crowdsec/data/
fi

I can make a PR for this (and adding the variable to docker documentation), if this is acceptable

LaurenceJJones commented 2 months ago

Honestly, we don't want to add another option just for this edge case. We symlink them because they already exist within the container image and copying the file just uses extra storage. As you stated after the container is running you can just CP manually and then the file are the correct ones and if the volume is persisted then there should be no issues after the fact.

There no need to onedrive these files, the only thing I would say is the database to which I would expect onedrive to be able to turn on or off depending if they are valid.

Simbiat commented 2 months ago

On my test environment my whole project, including data folders are on OneDrive. I know it's an edge case, but what about another more probable scenario: sharing of geo files between containers? From what I understand, if we still make links, it will not allow that, because links in other containers will be failing.

LaurenceJJones commented 2 months ago

On my test environment my whole project, including data folders are on OneDrive. I know it's an edge case, but what about another more probable scenario: sharing of geo files between containers? From what I understand, if we still make links, it will not allow that, because links in other containers will be failing.

If you share the same mount point between all crowdsec containers it follows the symlinks which will result in the same location no matter. If its another application then sorry its out of scope for this project to support having geo databases read by another application.

crowdsecurity / crowdsec