getDecisionStream does not return custom scope decisions by default

laurentgoudet commented 10 months ago

What happened?

Without the scopes parameter set, getDecisions returns Ip, Range, and custom scopes (UserAgent here) decisions:

curl -sS -H "X-Api-Key: 9E1uMkxZzSDYQZU/Qbm9Og" -k 'https://127.0.0.1:8080/v1/decisions?origins=cscli%2Ccrowdsec' | jq .
[
  {
    "duration": "3h50m28.543341271s",
    "id": 6177362,
    "origin": "cscli",
    "scenario": "manual 'ban' from ''",
    "scope": "UserAgent",
    "type": "ban",
    "value": "crowdsec"
  },
  {
    "duration": "3h50m42.543340911s",
    "id": 6177363,
    "origin": "cscli",
    "scenario": "manual 'ban' from ''",
    "scope": "Ip",
    "type": "ban",
    "value": "1.2.3.4"
  },
  {
    "duration": "3h59m54.543340425s",
    "id": 6177364,
    "origin": "cscli",
    "scenario": "manual 'ban' from ''",
    "scope": "Range",
    "type": "ban",
    "value": "1.2.3.0/24"
  }
]

However, by default (i.e. without scopes being set) getDecisionsStream does not return the custom scopes decisions, acting as in an scopes=Ip,Range filter was set:

curl -sS -H "X-Api-Key: 9E1uMkxZzSDYQZU/Qbm9Og" -k 'https://127.0.0.1:8080/v1/decisions/stream?startup=true&origins=cscli%2Ccrowdsec' | jq .
{
  "deleted": null,
  "new": [
    {
      "duration": "3h49m55.067403477s",
      "id": 6177363,
      "origin": "cscli",
      "scenario": "manual 'ban' from ''",
      "scope": "Ip",
      "type": "ban",
      "uuid": "dff4470c-6e10-4b75-acf1-3a5d7dd4084c",
      "value": "1.2.3.4"
    },
    {
      "duration": "3h59m7.067401742s",
      "id": 6177364,
      "origin": "cscli",
      "scenario": "manual 'ban' from ''",
      "scope": "Range",
      "type": "ban",
      "uuid": "798e70b5-2d84-43aa-8cfd-8abe42b2ab71",
      "value": "1.2.3.0/24"
    }
  ]
}

This prevents me from using https://github.com/crowdsecurity/cs-custom-bouncer for acting on custom scope decisions, since those are not returned by default by the stream API that the bouncer is using.

Note that manually setting scopes to Ip,Range,UserAgent makes getDecisionsStream return those decisions:

curl -sS -H "X-Api-Key: 9E1uMkxZzSDYQZU/Qbm9Og" -k 'https://127.0.0.1:8080/v1/decisions/stream?startup=true&scopes=Ip,Range,UserAgent&origins=cscli%2Ccrowdsec' | jq .
{
  "deleted": null,
  "new": [
    {
      "duration": "3h44m57.2303507s",
      "id": 6177362,
      "origin": "cscli",
      "scenario": "manual 'ban' from ''",
      "scope": "UserAgent",
      "type": "ban",
      "uuid": "ae12d4ba-fedf-42d2-8e5e-20aaafeab32e",
      "value": "crowdsec"
    },
    {
      "duration": "3h45m11.230350125s",
      "id": 6177363,
      "origin": "cscli",
      "scenario": "manual 'ban' from ''",
      "scope": "Ip",
      "type": "ban",
      "uuid": "dff4470c-6e10-4b75-acf1-3a5d7dd4084c",
      "value": "1.2.3.4"
    },
    {
      "duration": "3h54m23.230349652s",
      "id": 6177364,
      "origin": "cscli",
      "scenario": "manual 'ban' from ''",
      "scope": "Range",
      "type": "ban",
      "uuid": "798e70b5-2d84-43aa-8cfd-8abe42b2ab71",
      "value": "1.2.3.0/24"
    }
  ]
}

What did you expect to happen?

I would expect all scopes to be returned by default if scopes is not set, matching how the non-stream endpoint behaves.

How can we reproduce it (as minimally and precisely as possible)?

Add decisions with

cscli decision add -i 1.2.3.4
cscli decision add -r 1.2.3.0/24
cscli decision add --scope UserAgent --value crowdsec

Register an API key

cscli bouncer add MyTestBouncer

Use that API key to cURL the getDecisionsStream endpoint:

curl -sS -H "X-Api-Key: 9E1uMkxZzSDYQZU/Qbm9Og" -k 'https://127.0.0.1:8080/v1/decisions?origins=cscli%2Ccrowdsec' | jq .

Notice that the custom scope (UserAgent) decision is missing from the list returned by getDecisionsStream

Anything else we need to know?

No response

Crowdsec version

```console $ cscli version 2023/11/27 13:00:52 version: v1.5.5-d2d788c5dc0a9e387635276623c6781774a9dfd4 2023/11/27 13:00:52 Codename: alphaga 2023/11/27 13:00:52 BuildDate: 2023-10-24_08:13:35 2023/11/27 13:00:52 GoVersion: 1.21.3 2023/11/27 13:00:52 Platform: docker 2023/11/27 13:00:52 libre2: C++ 2023/11/27 13:00:52 Constraint_parser: >= 1.0, <= 2.0 2023/11/27 13:00:52 Constraint_scenario: >= 1.0, < 3.0 2023/11/27 13:00:52 Constraint_api: v1 2023/11/27 13:00:52 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

```console # On Linux: $ cat /etc/os-release NAME="Alpine Linux" ID=alpine VERSION_ID=3.18.4 PRETTY_NAME="Alpine Linux v3.18" HOME_URL="https://alpinelinux.org/" BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues" $ uname -a Linux ip-172-18-18-8 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 Linux ```

Enabled collections and parsers

```console $ cscli hub list -o raw # paste output here ```

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* filenames: - /var/log/nginx/*.log - ./tests/nginx/nginx.log #this is not a syslog log, indicate which kind of logs it is labels: type: nginx --- filenames: - /var/log/auth.log - /var/log/syslog labels: type: syslog --- filename: /var/log/apache2/*.log labels: type: apache2 cat: can't open '/etc/crowdsec/acquis.d/*': No such file or directory

Config show

```console $ cscli config show Global: - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log/ - Log level : info - Log Media : file Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /etc/crowdsec/acquis.d cscli: - Output : human - Hub Branch : - Hub Folder : /etc/crowdsec/hub API Client: - URL : https://127.0.0.1:8080/ - Login : - Credentials File : /etc/crowdsec/local_api_credentials.yaml Local API Server: - Listen URL : 0.0.0.0:8080 - Profile File : /etc/crowdsec/profiles.yaml - Cert File : /tmp/server.pem - Key File : /tmp/server-key.pem - CA Cert : /tmp/ca_bundle.pem - Allowed Agents OU : Freelancer Ltd - Allowed Bouncers OU : Freelancer Ltd - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : mysql - Host : db-crowdsec.syd1.fln-dev.flnltd.com - Port : 3306 - User : crowdsec - DB Name : crowdsec - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console $ cscli metrics Local API Metrics: ╭──────────────────────┬────────┬──────╮ │ Route │ Method │ Hits │ ├──────────────────────┼────────┼──────┤ │ /v1/alerts │ GET │ 4 │ │ /v1/alerts │ POST │ 5 │ │ /v1/decisions │ GET │ 10 │ │ /v1/decisions/stream │ GET │ 244 │ │ /v1/heartbeat │ GET │ 78 │ │ /v1/watchers/login │ POST │ 12 │ ╰──────────────────────┴────────┴──────╯ Local API Machines Metrics: ╭──────────────────────────────┬───────────────┬────────┬──────╮ │ Machine │ Route │ Method │ Hits │ ├──────────────────────────────┼───────────────┼────────┼──────┤ │ crowdsec_agent@127.0.0.1 │ /v1/alerts │ GET │ 2 │ │ crowdsec_agent@127.0.0.1 │ /v1/alerts │ POST │ 4 │ │ crowdsec_agent@172.18.14.178 │ /v1/heartbeat │ GET │ 37 │ │ crowdsec_agent@172.18.17.100 │ /v1/alerts │ GET │ 2 │ │ crowdsec_agent@172.18.17.100 │ /v1/alerts │ POST │ 1 │ │ crowdsec_agent@172.18.17.100 │ /v1/heartbeat │ GET │ 37 │ ╰──────────────────────────────┴───────────────┴────────┴──────╯ Local API Bouncers Metrics: ╭────────────────────────────┬──────────────────────┬────────┬──────╮ │ Bouncer │ Route │ Method │ Hits │ ├────────────────────────────┼──────────────────────┼────────┼──────┤ │ MyTestClient │ /v1/decisions/stream │ GET │ 12 │ │ MyTestClient │ /v1/decisions │ GET │ 10 │ │ crowdsec_bouncer@127.0.0.1 │ /v1/decisions/stream │ GET │ 232 │ ╰────────────────────────────┴──────────────────────┴────────┴──────╯ Local API Bouncers Decisions: ╭──────────────┬───────────────┬───────────────────╮ │ Bouncer │ Empty answers │ Non-empty answers │ ├──────────────┼───────────────┼───────────────────┤ │ MyTestClient │ 0 │ 10 │ ╰──────────────┴───────────────┴───────────────────╯ Local API Decisions: ╭────────────────────────────────────────────┬──────────┬─────────┬───────╮ │ Reason │ Origin │ Action │ Count │ ├────────────────────────────────────────────┼──────────┼─────────┼───────┤ │ crowdsecurity/f5-big-ip-cve-2020-5902 │ CAPI │ ban │ 22 │ │ crowdsecurity/ssh-slow-bf │ CAPI │ ban │ 12 │ │ crowdsecurity/CVE-2022-41082 │ CAPI │ ban │ 367 │ │ crowdsecurity/CVE-2022-42889 │ CAPI │ ban │ 4 │ │ crowdsecurity/http-cve-2021-41773 │ CAPI │ ban │ 20 │ │ crowdsecurity/http-path-traversal-probing │ CAPI │ ban │ 24 │ │ crowdsecurity/CVE-2022-37042 │ CAPI │ ban │ 14 │ │ crowdsecurity/CVE-2023-22518 │ CAPI │ ban │ 15 │ │ crowdsecurity/http-backdoors-attempts │ CAPI │ ban │ 255 │ │ crowdsecurity/netgear_rce │ CAPI │ ban │ 6 │ │ flnltd/http-bruteforce-login │ crowdsec │ captcha │ 1 │ │ crowdsecurity/apache_log4j2_cve-2021-44228 │ CAPI │ ban │ 306 │ │ crowdsecurity/http-probing │ CAPI │ ban │ 742 │ │ crowdsecurity/grafana-cve-2021-43798 │ CAPI │ ban │ 57 │ │ crowdsecurity/http-sensitive-files │ CAPI │ ban │ 8 │ │ crowdsecurity/jira_cve-2021-26086 │ CAPI │ ban │ 19 │ │ manual 'ban' from '' │ cscli │ ban │ 4 │ │ crowdsecurity/thinkphp-cve-2018-20062 │ CAPI │ ban │ 14 │ │ crowdsecurity/CVE-2019-18935 │ CAPI │ ban │ 32 │ │ crowdsecurity/CVE-2022-35914 │ CAPI │ ban │ 45 │ │ crowdsecurity/fortinet-cve-2018-13379 │ CAPI │ ban │ 76 │ │ crowdsecurity/http-bad-user-agent │ CAPI │ ban │ 2402 │ │ crowdsecurity/http-crawl-non_statics │ CAPI │ ban │ 147 │ │ crowdsecurity/http-open-proxy │ CAPI │ ban │ 267 │ │ crowdsecurity/nginx-req-limit-exceeded │ CAPI │ ban │ 47 │ │ crowdsecurity/CVE-2022-26134 │ CAPI │ ban │ 165 │ │ crowdsecurity/CVE-2023-22515 │ CAPI │ ban │ 4 │ │ crowdsecurity/http-generic-bf │ CAPI │ ban │ 8 │ │ crowdsecurity/ssh-bf │ CAPI │ ban │ 9922 │ ╰────────────────────────────────────────────┴──────────┴─────────┴───────╯ Local API Alerts: ╭──────────────────────────────┬───────╮ │ Reason │ Count │ ├──────────────────────────────┼───────┤ │ flnltd/http-bruteforce-login │ 2 │ │ manual 'ban' from '' │ 4 │ ╰──────────────────────────────┴───────╯ ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

github-actions[bot] commented 10 months ago

@laurentgoudet: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

Check Crowdsec Documentation to see if your issue can be self resolved.
You can also join our Discord.
Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.

LaurenceJJones commented 10 months ago

Hi thank you for your report,

We intentionally set those scopes if non is provided

https://github.com/crowdsecurity/crowdsec/blob/67cdf91f94a16fdb7122c2e450addd88c931ca6c/pkg/apiserver/controllers/v1/decisions.go#L353-L355

I will ask the team if this should be the same as non stream or if we should not do this.

laurentgoudet commented 10 months ago

Thanks. Arguably the inconsistency between getDecisions & getDecisionsStream is just a minor annoyance, what I am really trying to achieve here is being able to use https://github.com/crowdsecurity/cs-custom-bouncer with custom scopes, without having to rebuild the whole thing myself just to pass a custom scopes filter.

LaurenceJJones commented 10 months ago

Thanks. Arguably the inconsistency between getDecisions & getDecisionsStream is just a minor annoyance, what I am really trying to achieve here is being able to use https://github.com/crowdsecurity/cs-custom-bouncer with custom scopes, without having to rebuild the whole thing myself just to pass a custom scopes filter.

You shouldnt have to rebuild it, our go SDK already has scopes yaml keys.

https://github.com/crowdsecurity/go-cs-bouncer/blob/ff0ac9af53a6110b88e415b0a0e876cfc82341ea/stream_bouncer.go#L39C2-L39C8

So within all bouncers you can add the following to the configuration as an example

scopes:
  - ip
  - range
  - useragent

crowdsecurity / crowdsec