[loki] request headers are not set for `"loki/api/v1/query_rang` url

pschiffe commented 5 months ago

What happened?

I cannot connect from crowdsec to loki. Config:

source: loki
log_level: trace
url: http://lgtm-loki:3100/
headers:
  X-Scope-OrgID: someorg
query: |
  {container="traefik"}
labels:
  type: traefik

Logs:

time="2024-02-06T19:57:48Z" level=info msg="loading acquisition file : /etc/crowdsec/acquis.yaml"
time="2024-02-06T19:57:48Z" level=info msg="Resetting since" type=loki
time="2024-02-06T19:57:48Z" level=info msg="Since value: 0s" type=loki
time="2024-02-06T19:57:48Z" level=info msg="Starting processing data"
time="2024-02-06T19:57:48Z" level=info msg="127.0.0.1 - [Tue, 06 Feb 2024 19:57:48 UTC] \"POST /v1/watchers/login HTTP/1.1 200 106.24133ms \"crowdsec/v1.6.0-4192af30\" \""
time="2024-02-06T19:57:48Z" level=debug msg="Checking if Loki is ready" component=lokiclient source="http://lgtm-loki:3100/" type=loki
time="2024-02-06T19:57:48Z" level=info msg="Loki is ready" component=lokiclient source="http://lgtm-loki:3100/" type=loki
time="2024-02-06T19:57:48Z" level=debug msg="Since: 0s (2024-02-06 19:57:48.580313608 +0000 UTC m=+3.292162486)" component=lokiclient source="http://lgtm-loki:3100/" type=loki
time="2024-02-06T19:57:48Z" level=info msg="Connecting to http://lgtm-loki:3100/loki/api/v1/query_range?direction=forward&end=1707249468580288408&limit=100&query=%7Bcontainer%3D%22traefik%22%7D%0A&start=1707249468580286688" component=lokiclient source="http://lgtm-loki:3100/" type=loki
time="2024-02-06T19:57:48Z" level=warning msg="loki is not available, will retry for 30s" component=lokiclient source="http://lgtm-loki:3100/" type=loki
time="2024-02-06T19:57:52Z" level=info msg="127.0.0.1 - [Tue, 06 Feb 2024 19:57:52 UTC] \"GET /health HTTP/1.1 200 159.401µs \"Wget\" \""
time="2024-02-06T19:58:02Z" level=info msg="127.0.0.1 - [Tue, 06 Feb 2024 19:58:02 UTC] \"GET /health HTTP/1.1 200 42.68µs \"Wget\" \""
time="2024-02-06T19:58:12Z" level=info msg="127.0.0.1 - [Tue, 06 Feb 2024 19:58:12 UTC] \"GET /health HTTP/1.1 200 53.36µs \"Wget\" \""
time="2024-02-06T19:58:21Z" level=error msg="loki didn't manage to recover after 30s, giving up" component=lokiclient source="http://lgtm-loki:3100/" type=loki

I don't have auth configured for internal docker network, but my loki requires X-Scope-OrgID header. In the loki logs I see:

{"caller":"logging.go:118","level":"debug","msg":"GET /loki/api/v1/query_range?direction=forward&end=1707249712871610258&limit=100&query=%7Bcontainer%3D%22traefik%22%7D%0A&start=1707249712871606218 (401) 84.64µs","traceID":"570ba695bf8e8e26","ts":"2024-02-06T20:01:59.180945417Z"}

401 http code means there's no X-Scope-OrgID header. From inside of the crowdsec container:

a8720664422b:/# wget -O- 'http://lgtm-loki:3100/loki/api/v1/query_range?direction=forward&end=1707249468580288408&limit=100&query=%7Bcontainer%3D%22traefik%22%7D%0A&start=1707249468580286688'
Connecting to lgtm-loki:3100 (172.18.0.6:3100)
wget: server returned error: HTTP/1.1 401 Unauthorized
a8720664422b:/# wget -O- --header='X-Scope-OrgID: someorg' 'http://lgtm-loki:3100/loki/api/v1/query_range?direction=forward&end=1707249468580288408&limit=100&query=%7Bcontainer%3D%22traefik%22%7D%0A&start=1707249468580286688'
Connecting to lgtm-loki:3100 (172.18.0.6:3100)
writing to stdout
{"status":"success","data":{"resultType":"streams","result":[],"stats":{"summary":{"bytesProcessedPerSecond":0,"linesProcessedPerSecond":0,"totalBytesProcessed":0,"totalLinesProcessed":0,"execTime":0.017962,"queueTime":0.000067,"subqueries":0,"totalEntriesReturned":0,"splits":0,"shards":0,"totalPostFilterLines":0,"totalStructuredMetadataBytesProcessed":0},"querier":{"store":{"totalChunksRef":0,"totalChunksDownloaded":0,"chunksDownloadTime":0,"chunk":{"headChunkBytes":0,"headChunkLines":0,"decompressedBytes":0,"decompressedLines":0,"compressedBytes":0,"totalDuplicates":0,"postFilterLines":0,"headChunkStructuredMetadataBytes":0,"decompressedStructuredMetadataBytes":0}}},"ingester":{"totalReached":1,"totalChunksMatched":0,"totalBatches":1,"totalLinesSent":0,"store":{"totalChunksRef":0,"totalChunksDownloaded":0,"chunksDownloadTime":0,"chunk":{"headChunkBytes":0,"headChunkLines":0,"decompressedBytes":0,"decompressedLines":0,"compressedBytes":0,"totalDuplicates":0,"postFilterLines":0,"headChunkStructuredMetadataBytes":0,"decompressedStructuredMetadataBytes":0}}},"cache":{"chunk":{"entriesFound":0,"entriesRequested":0,"entriesStored":0,"bytesReceived":0,"bytesSent":0,"requests":0,"downloadTime":0},"index":{"entriesFound":0,"entriesRequested":0,"entriesStored":0,"bytesReceived":0,"bytesSent":0,"requests":0,"downloadTime":0},"result":{"entriesFound":0,"entriesRequested":0,"entriesStored":0,"bytesReceived":0,"bytesSent":0,"requests":0,"downloadTime":0},"statsResult":{"entriesFound":0,"entriesRequested":0,"entriesStored":0,"bytesReceived":0,"bytesSent":0,"requests":0,"downloadTime":0}}}}}
-                    100% |*******************************************************************************************************************************************************************************************************************************************************************************|  1605  0:00:00 ETA
written to stdout

In the source code here, it looks like the requestHeader is not set.

Also, a little bit verbose logging would be useful as well, such as the http error status code.

cc @lperdereau

What did you expect to happen?

loki working

How can we reproduce it (as minimally and precisely as possible)?

see above

Anything else we need to know?

No response

Crowdsec version

```console $ cscli version 2024/02/06 19:32:12 version: v1.6.0-4192af30 2024/02/06 19:32:12 Codename: alphaga 2024/02/06 19:32:12 BuildDate: 2024-01-31_13:41:30 2024/02/06 19:32:12 GoVersion: 1.21.6 2024/02/06 19:32:12 Platform: docker 2024/02/06 19:32:12 libre2: C++ 2024/02/06 19:32:12 Constraint_parser: >= 1.0, <= 3.0 2024/02/06 19:32:12 Constraint_scenario: >= 1.0, <= 3.0 2024/02/06 19:32:12 Constraint_api: v1 2024/02/06 19:32:12 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

```console # On Linux: $ cat /etc/os-release NAME="Alpine Linux" ID=alpine VERSION_ID=3.19.1 PRETTY_NAME="Alpine Linux v3.19" HOME_URL="https://alpinelinux.org/" BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues" $ uname -a Linux a8720664422b 6.7.3-100.fc38.aarch64 #1 SMP PREEMPT_DYNAMIC Thu Feb 1 04:22:10 UTC 2024 aarch64 Linux ```

Enabled collections and parsers

```console $ cscli hub list -o raw name,status,version,description,type crowdsecurity/cri-logs,enabled,0.1,CRI logging format parser,parsers crowdsecurity/dateparse-enrich,enabled,0.2,,parsers crowdsecurity/docker-logs,enabled,0.1,docker json logs parser,parsers crowdsecurity/geoip-enrich,enabled,0.2,"Populate event with geoloc info : as, country, coords, source range.",parsers crowdsecurity/sshd-logs,enabled,2.2,Parse openSSH logs,parsers crowdsecurity/syslog-logs,enabled,0.8,,parsers crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers crowdsecurity/ssh-bf,enabled,0.3,Detect ssh bruteforce,scenarios crowdsecurity/ssh-slow-bf,enabled,0.4,Detect slow ssh bruteforce,scenarios crowdsecurity/bf_base,enabled,0.1,,contexts crowdsecurity/linux,enabled,0.2,core linux support : syslog+geoip+ssh,collections crowdsecurity/sshd,enabled,0.3,sshd support : parser and brute-force detection,collections ```

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* source: loki log_level: trace url: http://lgtm-loki:3100/ headers: X-Scope-OrgID: someorg query: | {container="traefik"} labels: type: traefik cat: can't open '/etc/crowdsec/acquis.d/*': No such file or directory ```

Config show

```console $ cscli config show Global: - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log - Log level : info - Log Media : stdout Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /etc/crowdsec/acquis.d cscli: - Output : human - Hub Branch : API Client: - URL : http://0.0.0.0:8080/ - Login : localhost - Credentials File : /etc/crowdsec/local_api_credentials.yaml Local API Server: - Listen URL : 0.0.0.0:8080 - Profile File : /etc/crowdsec/profiles.yaml - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : mysql - Host : mysql - Port : 3306 - User : crowdsec - DB Name : crowdsec - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console $ cscli metrics Local API Metrics: ╭────────────────────┬────────┬──────╮ │ Route │ Method │ Hits │ ├────────────────────┼────────┼──────┤ │ /v1/heartbeat │ GET │ 6 │ │ /v1/watchers/login │ POST │ 1 │ ╰────────────────────┴────────┴──────╯ Local API Machines Metrics: ╭───────────┬───────────────┬────────┬──────╮ │ Machine │ Route │ Method │ Hits │ ├───────────┼───────────────┼────────┼──────┤ │ localhost │ /v1/heartbeat │ GET │ 6 │ ╰───────────┴───────────────┴────────┴──────╯ Local API Decisions: ╭───────────────────────────┬────────┬────────┬───────╮ │ Reason │ Origin │ Action │ Count │ ├───────────────────────────┼────────┼────────┼───────┤ │ crowdsecurity/ssh-bf │ CAPI │ ban │ 14942 │ │ crowdsecurity/ssh-slow-bf │ CAPI │ ban │ 58 │ ╰───────────────────────────┴────────┴────────┴───────╯ ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

No response

github-actions[bot] commented 5 months ago

@pschiffe: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

Check Crowdsec Documentation to see if your issue can be self resolved.
You can also join our Discord.
Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.

LaurenceJJones commented 5 months ago

FYI we are planning 1.6.1 soon but for yourself, you can use the dev container tag to get access to this fix.

pschiffe commented 4 months ago

Awesome, thank you for quick fix. My server is arm64, so I will have to wait for the proper release I think.

crowdsecurity / crowdsec