Improvement/LAPI: Add 'origin' query to decisions local API

[RoboMagus]

Is your feature request related to a problem? Please describe. I'd like to be able to query decisions from within HomeAssistant. Mostly to see if one of my own devices has been blocked such that I can take action on it. When listing decisions using cscli decisions list you only get the local decisions by default. Using LAPI however you get a huge list that includes decisions from the CAPI origin.

In order to create a sensor for the local decisions I'd like to be able to filter based on the decision origin, much like is currently possible based on 'type', 'range', 'scope', etc.

Describe the solution you'd like Adding a query parameter for 'origin' to LAPI. e.g. curl -X GET "https://127.0.0.1/v1/decisions?origin=cscli" -H "accept: application/json"

Describe alternatives you've considered I've tried a workaround using the command line call for docker exec crowdsec sh -c "cscli decisions list -o json". However this is really something that should just be present on the API.

edit: Added example to expected solution to clarify improvements requested for LAPI

[buixor]

Hello !

Unless I'm mistaken, wouldn't the --origin string restrict to this origin (ie. lists,CAPI,cscli) of cscli decisions list do the job ? :)

[RoboMagus]

for cscli all the options are there, the request is to expose those query filtering options to LAPI as well ;)

[buixor]

Hello, it is available in the alerts endpoint, but isn't in the decision endpoint.

is the alerts endpoint enough for you ?

[LaurenceJJones]

Behind the scenes cscli decisions list is using the alerts endpoint. https://github.com/crowdsecurity/crowdsec/blob/f1dbe8c9ddfa3e60f3b72e83ac7a9004332b151f/pkg/apiclient/alerts_service.go#L101

[RoboMagus]

What is the lifetime of such an alert? Will they be available for as long as the decision is active, and removed once the timeout on the decision expires?

If those match up then it should be an option. However when performing a GET on the alerts endpoint I'm presented with a cookie error. This would make querying for e.g. a HomeAssistant sensor impossible, as it is expected that a single command that includes the authorization token should be sufficient to retrieve the required data.

curl -H "X-Api-Key: [...]" http://crowdsec:8080/v1/alerts

{"code":401,"message":"cookie token is empty"}

Could you enlighten me on the choice why there are inconsistencies on the API w.r.t. authorization methods? (Login+cookie vs. Token)?

Also, I noticed that where for 'Decisions' cscli does have the 'origin' argument, where for LAPI it does not. And for 'Alerts' the 'origin' argument is missing on cscli, where on LAPI this is available. This seems quite illogical. Perhaps it would be an improvement to at least match this up between cscli and LAPI. Similarly for the time-range that is missing on the decisions api, but present on cscli.

[LaurenceJJones]

Hello, I cannot speak on behalf of the team as I'm just a user! There are 2 types of tokens X-Api-Key and "Authorization: Bearer" I may not be fully correct, but I believe the two types of tokens are there for the ability to have bouncers on another machine. You only want the bouncers to be able to read decisions and not run "admin-level" endpoints.

As you can see from the swagger documentation here you would need the ability to send 2 curl requests one to /v1/watchers/login to generate a JWT session token and another to /v1/alerts that includes the header "Authorization: Bearer [token]". Note: maybe @buixor can explain why the login method needs the scenarios as an array? I tried on my VM but didn't see any impact on what I could retrieve.

I tested a few options but a single liner I found was this

curl -H "Authorization: Bearer $(curl -s -X POST -H 'Content-Type: application/json' -d '{"machine_id": "","password": ""}' http://127.0.0.1:8080/v1/watchers/login | jq -r ".token")" http://127.0.0.1:8080/v1/alerts?origin=cscli

package JQ needs to be installed on the target machine easy way to get the token out of the response body. Edit: if you cannot get JQ on the machine replace | jq .... with:

| sed -E 's/.*"token":"?([^,"]*)"?.*/\1/'

The machine and password are kept under /etc/crowdsec/local_api_credentials.yaml

[LaurenceJJones]

What is the lifetime of such an alert? Will they be available for as long as the decision is active, and removed once the timeout on the decision expires?

The alert endpoint shows all alerts even if the decision has expired. Example output:

[{
"capacity": 0,
"created_at": "2022-05-16T11:12:01Z",
"decisions": [
{
"duration": "-1h5m36.045304492s",
"id": 87624,
"origin": "cscli",
"scenario": "manual 'ban' from 'xxxx'",
"scope": "Ip",
"simulated": false,
"type": "ban",
"value": "10.10.10.10"
}
],
"events": null,
"events_count": 1,
"id": 56,
"labels": null,
"leakspeed": "0",
"machine_id": "xxxx",
"message": "manual 'ban' from 'xxxx'",
"scenario": "manual 'ban' from 'xxxx'",
"scenario_hash": "",
"scenario_version": "",
"simulated": false,
"source": {
"ip": "10.10.10.10",
"scope": "Ip",
"value": "10.10.10.10"
},
"start_at": "2022-05-16 11:12:01 +0000 UTC",
"stop_at": "2022-05-16 11:12:01 +0000 UTC"
}.....]

The alert endpoint does take additional query params docs

[AlteredCoder]

Hello, in crowdsec v1.3.4 you are able to query the local API decisions endpoints by filtering the origins. We will add the new parameters in the swagger documentation. Example:

• For the stream mode:

  
  curl -H "X-Api-Key: <API_KEY>" "http://localhost:8080/v1/decisions/stream?startup=true&origins=crowdsec,cscli"

and then

curl -H "X-Api-Key: " "http://localhost:8080/v1/decisions/stream?startup=false&origins=crowdsec,cscli"

- For the live mode:

curl -H "X-Api-Key: " "http://localhost:8080/v1/decisions?origins=crowdsec,cscli"

[buixor]

@RoboMagus :

What is the lifetime of such an alert? Will they be available for as long as the decision is active, and removed once the timeout on the decision expires?

Alerts stays "forever", but Decisions expire (in the way : are not returned anymore by the API) once they expired or are deleted

[RoboMagus]

@AlteredCoder, Thanks for letting me know! That's exactly what I was looking for. My setup was still stuck on v1.3.1. As The API spec was v1.0.0 and for more recent releases this had not changed I figured there was no use in updating.
So erm, Devs, VERSIONING MATTERS! If you have versioned software components or API specs, please do not forget to update those numbers whenever functionality or spec changes. ;-)

Nevertheless, Great work and thanks for adding this query option!

@LaurenceJJones, thanks for your input. I had also been playing around with the various options to get valuable data out. As the 2 command login flow does not seem compatible with a simple state-less REST sensor within home-assistant I started looking for options to exploit the commandline as well. But in that case interaction throuhg cscli would be easier than chaining curl commands.

The alert endpoint does take additional query params docs

The LAPI endpoint does, yes, but here I was refering to the cscli options not accepting to filter on 'origin'