search

crowdsecurity / crowdsec

CrowdSec - the open-source and participative security solution offering crowdsourced protection against malicious IPs and access to the most advanced real-world CTI.
https://crowdsec.net
MIT License
9k stars 467 forks source link

Crowdsec update issue - Ubuntu 22.04 #2204

Closed modem7 closed 1 year ago

modem7 commented 1 year ago

What happened?

sudo aptitude update && sudo aptitude safe-upgrade -y
Hit http://security.ubuntu.com/ubuntu jammy-security InRelease
Get: 1 http://download.opensuse.org/repositories/shells:/zsh-users:/zsh-autosuggestions/xUbuntu_22.04  InRelease [1,567 B]
Get: 2 http://download.opensuse.org/repositories/shells:/zsh-users:/zsh-syntax-highlighting/xUbuntu_22.04  InRelease [1,579 B]
Hit https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64  InRelease
Get: 3 https://cli.github.com/packages stable InRelease [3,917 B]
Hit https://esm.ubuntu.com/apps/ubuntu jammy-apps-security InRelease
Hit https://esm.ubuntu.com/apps/ubuntu jammy-apps-updates InRelease
Hit https://repo.netdata.cloud/repos/stable/ubuntu jammy/ InRelease
Hit https://esm.ubuntu.com/infra/ubuntu jammy-infra-security InRelease
Hit http://archive.ubuntu.com/ubuntu jammy InRelease
Hit https://esm.ubuntu.com/infra/ubuntu jammy-infra-updates InRelease
Hit https://repo.netdata.cloud/repos/repoconfig/ubuntu jammy/ InRelease
Hit http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit https://download.docker.com/linux/ubuntu jammy InRelease
Hit http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Ign https://download.webmin.com/download/repository sarge InRelease
Hit https://download.webmin.com/download/repository sarge Release
Hit https://packagecloud.io/crowdsec/crowdsec/ubuntu jammy InRelease
Fetched 7,063 B in 2s (3,531 B/s)

The following partially installed packages will be configured:
  crowdsec
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.
Setting up crowdsec (1.5.0) ...
Updating hub
WARN[16-05-2023 22:50:32] Crowdsec is not the latest version. Current version is 'v1.5.0-debian-pragmatic' and the latest stable version is 'v1.5.0'. Please update it!
WARN[16-05-2023 22:50:32] As a result, you will not be able to use parsers/scenarios/collections added to Crowdsec Hub after CrowdSec v1.5.0
FATA[16-05-2023 22:50:32] Failed to get Hub index : failed to download index: bad http code 404 while requesting https://hub-cdn.crowdsec.net/v1.5.0-debian-pragmatic/.index.json
dpkg: error processing package crowdsec (--configure):
 installed crowdsec package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 crowdsec
E: Sub-process /usr/bin/dpkg returned an error code (1)
Setting up crowdsec (1.5.0) ...
Updating hub
WARN[16-05-2023 22:50:35] Crowdsec is not the latest version. Current version is 'v1.5.0-debian-pragmatic' and the latest stable version is 'v1.5.0'. Please update it!
WARN[16-05-2023 22:50:35] As a result, you will not be able to use parsers/scenarios/collections added to Crowdsec Hub after CrowdSec v1.5.0
FATA[16-05-2023 22:50:35] Failed to get Hub index : failed to download index: bad http code 404 while requesting https://hub-cdn.crowdsec.net/v1.5.0-debian-pragmatic/.index.json
dpkg: error processing package crowdsec (--configure):
 installed crowdsec package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 crowdsec

What did you expect to happen?

Crowdsec to update normally

How can we reproduce it (as minimally and precisely as possible)?

sudo aptitude update && sudo aptitude safe-upgrade -y

Anything else we need to know?

No response

Crowdsec version

```console ❯ cscli version 2023/05/16 22:52:50 version: v1.5.0-debian-pragmatic-0ddd42c01f28411d24fc60084f9c48af55f7f1f3 2023/05/16 22:52:50 Codename: alphaga 2023/05/16 22:52:50 BuildDate: 2023-05-16_14:10:28 2023/05/16 22:52:50 GoVersion: 1.20.1 2023/05/16 22:52:50 Platform: linux 2023/05/16 22:52:50 Constraint_parser: >= 1.0, <= 2.0 2023/05/16 22:52:50 Constraint_scenario: >= 1.0, < 3.0 2023/05/16 22:52:50 Constraint_api: v1 2023/05/16 22:52:50 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

```console # On Linux: ❯ cat /etc/os-release PRETTY_NAME="Ubuntu 22.04.2 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.2 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy ❯ uname -a Linux HDA 5.15.0-72-generic #79-Ubuntu SMP Wed Apr 19 08:22:18 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux ```

Enabled collections and parsers

```console ❯ sudo cscli hub list -o raw crowdsecurity/base-http-scenarios,enabled,0.6,http common : scanners detection,collections crowdsecurity/http-cve,enabled,2.0,,collections crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections crowdsecurity/smb,enabled,0.1,smb support : parser and brute-force scenario,collections crowdsecurity/sshd,enabled,0.2,sshd support : parser and brute-force detection,collections crowdsecurity/whitelist-good-actors,enabled,0.1,Good actors whitelists,collections crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/geoip-enrich,enabled,0.2,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/http-logs,enabled,1.1,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers crowdsecurity/smb-logs,enabled,0.2,Parse SMB logs,parsers crowdsecurity/sshd-logs,enabled,2.0,Parse openSSH logs,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers crowdsecurity/CVE-2019-18935,enabled,0.1,Detect Telerik CVE-2019-18935 exploitation attempts,scenarios crowdsecurity/CVE-2022-26134,enabled,0.1,Detect CVE-2022-26134 exploits,scenarios crowdsecurity/CVE-2022-35914,enabled,0.1,Detect CVE-2022-35914 exploits,scenarios crowdsecurity/CVE-2022-37042,enabled,0.1,Detect CVE-2022-37042 exploits,scenarios crowdsecurity/CVE-2022-40684,enabled,0.2,Detect cve-2022-40684 exploitation attempts,scenarios crowdsecurity/CVE-2022-41082,enabled,0.3,Detect CVE-2022-41082 exploits,scenarios crowdsecurity/CVE-2022-41697,enabled,0.1,Detect CVE-2022-41697 enumeration,scenarios crowdsecurity/CVE-2022-42889,enabled,0.2,Detect CVE-2022-42889 exploits (Text4Shell),scenarios crowdsecurity/CVE-2022-44877,enabled,0.2,Detect CVE-2022-44877 exploits,scenarios crowdsecurity/CVE-2022-46169,enabled,0.1,Detect CVE-2022-46169 brute forcing,scenarios crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.4,Detect cve-2021-44228 exploitation attemps,scenarios crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.1,Detect cve-2020-5902 exploitation attemps,scenarios crowdsecurity/fortinet-cve-2018-13379,enabled,0.2,Detect cve-2018-13379 exploitation attemps,scenarios crowdsecurity/grafana-cve-2021-43798,enabled,0.1,Detect cve-2021-43798 exploitation attemps,scenarios crowdsecurity/http-backdoors-attempts,enabled,0.3,Detect attempt to common backdoors,scenarios crowdsecurity/http-bad-user-agent,enabled,0.7,Detect bad user-agents,scenarios crowdsecurity/http-crawl-non_statics,enabled,0.3,Detect aggressive crawl from single ip,scenarios crowdsecurity/http-cve-2021-41773,enabled,0.1,cve-2021-41773,scenarios crowdsecurity/http-cve-2021-42013,enabled,0.1,cve-2021-42013,scenarios crowdsecurity/http-generic-bf,enabled,0.4,Detect generic http brute force,scenarios crowdsecurity/http-open-proxy,enabled,0.3,Detect scan for open proxy,scenarios crowdsecurity/http-path-traversal-probing,enabled,0.2,Detect path traversal attempt,scenarios crowdsecurity/http-probing,enabled,0.2,Detect site scanning/probing from a single ip,scenarios crowdsecurity/http-sensitive-files,enabled,0.2,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios crowdsecurity/http-sqli-probing,enabled,0.2,A scenario that detects SQL injection probing with minimal false positives,scenarios crowdsecurity/http-xss-probing,enabled,0.2,A scenario that detects XSS probing with minimal false positives,scenarios crowdsecurity/jira_cve-2021-26086,enabled,0.1,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.2,Detect cve-2019-11510 exploitation attemps,scenarios crowdsecurity/smb-bf,enabled,0.1,Detect smb bruteforce,scenarios crowdsecurity/spring4shell_cve-2022-22965,enabled,0.2,Detect cve-2022-22965 probing,scenarios crowdsecurity/ssh-bf,enabled,0.1,Detect ssh bruteforce,scenarios crowdsecurity/ssh-slow-bf,enabled,0.2,Detect slow ssh bruteforce,scenarios crowdsecurity/thinkphp-cve-2018-20062,enabled,0.3,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios crowdsecurity/vmware-cve-2022-22954,enabled,0.2,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.1,Detect VMSA-2021-0027 exploitation attemps,scenarios ltsich/http-w00tw00t,enabled,0.1,detect w00tw00t,scenarios crowdsecurity/cdn-whitelist,enabled,0.4,Whitelist CDN providers,postoverflows crowdsecurity/rdns,enabled,0.2,Lookup the DNS associated to the source IP only for overflows,postoverflows crowdsecurity/seo-bots-whitelist,enabled,0.4,Whitelist good search engine crawlers,postoverflows ```

Acquisition config

```console # On Linux: ❯ cat /etc/crowdsec/acquis.yaml #Generated acquisition file - wizard.sh (service: sshd) / files : /var/log/auth.log filenames: - /var/log/auth.log labels: type: syslog --- #Generated acquisition file - wizard.sh (service: linux) / files : /var/log/syslog /var/log/kern.log filenames: - /var/log/syslog - /var/log/kern.log labels: type: syslog ---

Config show

```console ❯ sudo cscli config show Global: - Configuration Folder : /etc/crowdsec - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log/ - Log level : info - Log Media : file Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /etc/crowdsec/acquis.d cscli: - Output : human - Hub Branch : - Hub Folder : /etc/crowdsec/hub API Client: - URL : http://127.0.0.1:8080/ - Login : e5d5d79cad044640b7f9d8ef3a35584bVXtkYKe4FqmxwcRP - Credentials File : /etc/crowdsec/local_api_credentials.yaml Local API Server: - Listen URL : 127.0.0.1:8080 - Profile File : /etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : sqlite - Path : /var/lib/crowdsec/data/crowdsec.db - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console ❯ sudo cscli metrics FATA[16-05-2023 22:55:05] failed to fetch prometheus metrics : executing GET request for URL "http://127.0.0.1:6060/metrics" failed: Get "http://127.0.0.1:6060/metrics": dial tcp 127.0.0.1:6060: connect: connection refused ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

github-actions[bot] commented 1 year ago

@modem7: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

  1. Check Crowdsec Documentation to see if your issue can be self resolved.
  2. You can also join our Discord.
  3. Check Releases to make sure your agent is on the latest version.
Details I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.
buixor commented 1 year ago

Hello,

Thanks for the report, we are looking into it.

What version did you have before ? The install log says it was already partially installed.

(Didn't manage to reproduce on ubuntu 20)

Edit: didn't manage to reproduce on ubuntu 22.04 from 1.4.6 using the same aptitude commands

modem7 commented 1 year ago

Hello,

Thanks for the report, we are looking into it.

What version did you have before ? The install log says it was already partially installed.

(Didn't manage to reproduce on ubuntu 20)

Edit: didn't manage to reproduce on ubuntu 22.04 from 1.4.6 using the same aptitude commands

Heya

❯ sudo cscli version
2023/05/16 23:36:30 version: v1.4.6-debian-pragmatic-5f71037b40c498045e1b59923504469e2b8d0140
2023/05/16 23:36:30 Codename: alphaga
2023/05/16 23:36:30 BuildDate: 2023-02-09_14:33:16
2023/05/16 23:36:30 GoVersion: 1.19.2
2023/05/16 23:36:30 Platform: linux
2023/05/16 23:36:30 Constraint_parser: >= 1.0, <= 2.0
2023/05/16 23:36:30 Constraint_scenario: >= 1.0, < 3.0
2023/05/16 23:36:30 Constraint_api: v1
2023/05/16 23:36:30 Constraint_acquis: >= 1.0, < 2.0

Cheers!

modem7 commented 1 year ago

Will close this for now.

After a VM restore and crowdsec reinstall, it seems to have behaved again. No idea why this occurred in first place, however.