k8s - hash mismatch on persistent volume

usma0118 commented 2 months ago

What happened?

crowdsec agent startup failed with following error:

Defaulted container "crowdsec-agent" out of: crowdsec-agent, wait-for-lapi (init)
/etc/crowdsec_data was found in a volume
Running hub update
Skipping hub update, index file is recent
Skipping hub upgrade, data directory is not in a volume
Running: cscli  parsers install "crowdsecurity/docker-logs"
installed crowdsecurity/docker-logs
level=info msg="Enabled crowdsecurity/docker-logs"
level=info msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
Running: cscli  parsers install "crowdsecurity/cri-logs"
installed crowdsecurity/cri-logs
level=info msg="Enabled crowdsecurity/cri-logs"
level=info msg="Run 'sudo systemctl reload crowdsec' for the new configuration to be effective."
Running: cscli  collections install "crowdsecurity/nginx"
level=fatal msg="error while installing 'crowdsecurity/nginx': while downloading crowdsecurity/nginx-logs: while downloading crowdsecurity/nginx-logs to https://hub-cdn.crowdsec.net/master/parsers/s01-parse/crowdsecurity/nginx-logs.yaml: hash mismatch: expected 1948e74edab6e6fa23f70675e2883b726d4e0394314dafaad2b9819762b92b34, got 538990ce5b01974ddd29c948de56322b92de56f6d9e70fc7f45415ce8af3858d"
Failed to install collections/crowdsecurity/nginx, running hub update before retrying
Skipping hub update, index file is recent
level=fatal msg="error while installing 'crowdsecurity/nginx': while downloading crowdsecurity/nginx-logs: while downloading crowdsecurity/nginx-logs to https://hub-cdn.crowdsec.net/master/parsers/s01-parse/crowdsecurity/nginx-logs.yaml: hash mismatch: expected 1948e74edab6e6fa23f70675e2883b726d4e0394314dafaad2b9819762b92b34, got 538990ce5b01974ddd29c948de56322b92de56f6d9e70fc7f45415ce8af3858d"

What did you expect to happen?

for crowdsect agent to startup and load collections.

How can we reproduce it (as minimally and precisely as possible)?

upgrade running helm version: 0.10.0 to version: 0.11.0

Anything else we need to know?

No response

Crowdsec version

```console $ cscli version # paste output here ```

OS version

```console # On Linux: $ cat /etc/os-release # paste output here $ uname -a # paste output here # On Windows: C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture # paste output here ```

Enabled collections and parsers

```console $ cscli hub list -o raw # paste output here ```

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* filenames: - /var/log/nginx/*.log - ./tests/nginx/nginx.log #this is not a syslog log, indicate which kind of logs it is labels: type: nginx --- filenames: - /var/log/auth.log - /var/log/syslog labels: type: syslog --- filename: /var/log/apache2/*.log labels: type: apache2 # On Windows: C:\> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml # paste output here

Config show

```console $ cscli config show E0827 09:26:12.661835 65905 websocket.go:296] Unknown stream id 1, discarding message Global: - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log - Log level : info - Log Media : stdout Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /etc/crowdsec/acquis.d cscli: - Output : human - Hub Branch : API Client: - URL : http://localhost:8080/ - Login : localhost - Credentials File : /etc/crowdsec/local_api_credentials.yaml Local API Server: - Listen URL : 0.0.0.0:8080 - Listen Socket : - Profile File : /etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : sqlite - Path : /var/lib/crowdsec/data/crowdsec.db - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console $ cscli metrics ╭────────────────────────────────────────────┬────────┬────────┬───────╮ │ Reason │ Origin │ Action │ Count │ ├────────────────────────────────────────────┼────────┼────────┼───────┤ │ crowdsecurity/http-sensitive-files │ CAPI │ ban │ 216 │ │ crowdsecurity/apache_log4j2_cve-2021-44228 │ CAPI │ ban │ 47 │ │ crowdsecurity/http-bad-user-agent │ CAPI │ ban │ 8248 │ │ crowdsecurity/nginx-req-limit-exceeded │ CAPI │ ban │ 397 │ │ crowdsecurity/thinkphp-cve-2018-20062 │ CAPI │ ban │ 84 │ │ crowdsecurity/http-generic-bf │ CAPI │ ban │ 25 │ │ crowdsecurity/http-path-traversal-probing │ CAPI │ ban │ 146 │ │ crowdsecurity/netgear_rce │ CAPI │ ban │ 15 │ │ crowdsecurity/CVE-2019-18935 │ CAPI │ ban │ 19 │ │ crowdsecurity/http-crawl-non_statics │ CAPI │ ban │ 314 │ │ crowdsecurity/ssh-slow-bf │ CAPI │ ban │ 7957 │ │ crowdsecurity/http-backdoors-attempts │ CAPI │ ban │ 148 │ │ crowdsecurity/http-cve-probing │ CAPI │ ban │ 8 │ │ crowdsecurity/CVE-2023-22515 │ CAPI │ ban │ 3 │ │ crowdsecurity/pgsql-bf │ CAPI │ ban │ 21 │ │ crowdsecurity/CVE-2022-35914 │ CAPI │ ban │ 2 │ │ crowdsecurity/CVE-2022-37042 │ CAPI │ ban │ 2 │ │ crowdsecurity/CVE-2022-26134 │ CAPI │ ban │ 9 │ │ crowdsecurity/http-cve-2021-41773 │ CAPI │ ban │ 187 │ │ ltsich/http-w00tw00t │ CAPI │ ban │ 3 │ │ crowdsecurity/fortinet-cve-2018-13379 │ CAPI │ ban │ 10 │ │ crowdsecurity/http-admin-interface-probing │ CAPI │ ban │ 228 │ │ crowdsecurity/http-cve-2021-42013 │ CAPI │ ban │ 3 │ │ crowdsecurity/http-open-proxy │ CAPI │ ban │ 1435 │ │ crowdsecurity/http-probing │ CAPI │ ban │ 2991 │ │ crowdsecurity/http-wordpress-scan │ CAPI │ ban │ 236 │ │ crowdsecurity/jira_cve-2021-26086 │ CAPI │ ban │ 15 │ │ crowdsecurity/ssh-bf │ CAPI │ ban │ 6351 │ │ crowdsecurity/CVE-2017-9841 │ CAPI │ ban │ 161 │ │ crowdsecurity/CVE-2023-49103 │ CAPI │ ban │ 85 │ │ crowdsec_paris_2024_intelligence │ lists │ ban │ 6455 │ ╰────────────────────────────────────────────┴────────┴────────┴───────╯ ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

github-actions[bot] commented 2 months ago

@usma0118: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

Check Crowdsec Documentation to see if your issue can be self resolved.
You can also join our Discord.
Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.

usma0118 commented 2 months ago

already tried #2946 without any success.

LaurenceJJones commented 2 months ago

already tried #2946 without any success.

Can you provide the docker run command that was executed?

usma0118 commented 2 months ago

already tried #2946 without any success.

Can you provide the docker run command that was executed?

I am using k8s with helm, not docker. if you want to see chart values, those can be seen here: https://pastebin.com/mAZgwKV5

blotus commented 2 months ago

Hello,

We are working on a long-term fix for this type of issue that should be part of 1.6.3, but in the meantime, you can try the following:

Create a file pod.yaml with the following content (make sure the crowdsec version specified is the same as the one you use):

apiVersion: v1
kind: Pod
metadata:
name: temp-cscli-update
spec:
containers:
- name: temp-cscli-update
image: crowdsecurity/crowdsec:v1.6.2
command: ["sh", "-c", "ln -s /etc/crowdsec_data /etc/crowdsec && cscli hub update"]
volumeMounts:
- mountPath: /etc/crowdsec_data
  name: crowdsec-agent-config
volumes:
- name: crowdsec-agent-config
persistentVolumeClaim:
  claimName: crowdsec-agent-config-pvc
restartPolicy: Never

Run this pod: kubectl apply -f pod.yaml
It should take just a few seconds, you can check the hub was updated properly with kubectl logs temp-cscli-update
If it succeeded, you can delete the pod: kubectl delete -f pod.yaml

LaurenceJJones commented 2 months ago

@usma0118 Did the provided workaround managed to fix the issue? if not please reopen the issue and provided relevant details as to why it did not work or resolve the issue

crowdsecurity / crowdsec