[hub] introduce cscli hub fix command

LaurenceJJones commented 1 month ago

There has been an outstanding issue for a long time if a user "accidentally" install the debian packages and then upgrades to our repository version all symlinks point towards deleted files.

An idea could be to have cscli hub fix command that goes through the current symlinks and tries to rectify the broken symlinks for example after installing the debian package and upgrading to our this is the output of cscli parsers list

root@bookworm:~# cscli parsers list
WARN link target does not exist: /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml -> /var/lib/crowdsec/hub/parsers/s00-raw/crowdsecurity/syslog-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s01-parse/apache2-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/apache2-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s01-parse/nginx-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/nginx-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml -> /var/lib/crowdsec/hub/parsers/s01-parse/crowdsecurity/sshd-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml -> /var/lib/crowdsec/hub/parsers/s02-enrich/crowdsecurity/dateparse-enrich.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s02-enrich/http-logs.yaml -> /var/lib/crowdsec/hub/parsers/s02-enrich/crowdsecurity/http-logs.yaml
WARN link target does not exist: /etc/crowdsec/parsers/s02-enrich/whitelists.yaml -> /var/lib/crowdsec/hub/parsers/s02-enrich/crowdsecurity/whitelists.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-26134.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-26134.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-35914.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-35914.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-37042.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-37042.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-40684.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-40684.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-41082.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-41082.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-41697.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-41697.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-42889.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-42889.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-44877.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-44877.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/CVE-2022-46169.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/CVE-2022-46169.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/apache_log4j2_cve-2021-44228.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/apache_log4j2_cve-2021-44228.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/f5-big-ip-cve-2020-5902.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/f5-big-ip-cve-2020-5902.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/fortinet-cve-2018-13379.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/fortinet-cve-2018-13379.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/grafana-cve-2021-43798.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/grafana-cve-2021-43798.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-backdoors-attempts.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-backdoors-attempts.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-bad-user-agent.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-bad-user-agent.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-crawl-non_statics.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-crawl-non_statics.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-cve-2021-41773.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-cve-2021-41773.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-cve-2021-42013.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-cve-2021-42013.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-generic-bf.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-generic-bf.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-open-proxy.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-open-proxy.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-path-traversal-probing.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-path-traversal-probing.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-probing.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-probing.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-sensitive-files.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-sensitive-files.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-sqli-probing.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-sqli-probing.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-w00tw00t.yaml -> /var/lib/crowdsec/hub/scenarios/ltsich/http-w00tw00t.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/http-xss-probing.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/http-xss-probing.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/jira_cve-2021-26086.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/jira_cve-2021-26086.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/nginx-req-limit-exceeded.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/nginx-req-limit-exceeded.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/pulse-secure-sslvpn-cve-2019-11510.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/pulse-secure-sslvpn-cve-2019-11510.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/spring4shell_cve-2022-22965.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/spring4shell_cve-2022-22965.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/ssh-bf.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/ssh-bf.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/ssh-slow-bf.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/ssh-slow-bf.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/thinkphp-cve-2018-20062.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/thinkphp-cve-2018-20062.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/vmware-cve-2022-22954.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/vmware-cve-2022-22954.yaml
WARN link target does not exist: /etc/crowdsec/scenarios/vmware-vcenter-vmsa-2021-0027.yaml -> /var/lib/crowdsec/hub/scenarios/crowdsecurity/vmware-vcenter-vmsa-2021-0027.yaml
WARN link target does not exist: /etc/crowdsec/collections/apache2.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/apache2.yaml
WARN link target does not exist: /etc/crowdsec/collections/base-http-scenarios.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/base-http-scenarios.yaml
WARN link target does not exist: /etc/crowdsec/collections/http-cve.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/http-cve.yaml
WARN link target does not exist: /etc/crowdsec/collections/linux.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/linux.yaml
WARN link target does not exist: /etc/crowdsec/collections/nginx.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/nginx.yaml
WARN link target does not exist: /etc/crowdsec/collections/sshd.yaml -> /var/lib/crowdsec/hub/collections/crowdsecurity/sshd.yaml

PARSERS
──────────────────────────────────────
Name  📦 Status  Version  Local Path
──────────────────────────────────────
──────────────────────────────────────

I did used to have a script https://gist.github.com/LaurenceJJones/6960107296145e8e365009973b9d7f6d that would fix this, however, with recent changes to the hub no items are displayed and it cannot be fixed like this anymore.

Edit: This will improve the user experience then having to completely remove the package (potentially remove there own custom configuration) and having to restart all over again if they happen to notice this late in the process.

github-actions[bot] commented 1 month ago

@LaurenceJJones: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

Check Crowdsec Documentation to see if your issue can be self resolved.
You can also join our Discord.
Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.

github-actions[bot] commented 1 month ago

@LaurenceJJones: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

/kind feature
/kind enhancement
/kind refactoring
/kind bug
/kind packaging

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.

crowdsecurity / crowdsec

[hub] introduce cscli hub fix command #3264