Environment variables in local_api_credentials.yaml.local are not expanded

andreasbrett commented 2 weeks ago

What happened?

When using environment variables in local_api_credentials.yaml (in my case local_api_credentials.yaml.local) these are not expanded and can thus not be used.

What did you expect to happen?

I expected environment variables to be expanded as per https://docs.crowdsec.net/docs/configuration/crowdsec_configuration/#environment-variables. This is particularly useful for creating re-usable config files that can be controlled via environment variables when being deployed to different machines.

How can we reproduce it (as minimally and precisely as possible)?

define environment variable CONNECT_URI and point it at e.g. https://localhost:8080
provide url definition in local_api_credentials.yaml (url: ${CONNECT_URI})
run cscli config show and see how it is not expanded
run cscli lapi status and see how it is not interpreted

d99d6639663a:/# cat /etc/crowdsec/local_api_credentials.yaml.local
url: ${CONNECT_URI}

d99d6639663a:/# env | grep CONNECT
CONNECT_URI=https://localhost:8080

d99d6639663a:/# cscli config show
INFO[2024-06-12T13:29:53+02:00] Loading yaml file: '/etc/crowdsec/local_api_credentials.yaml' with additional values from '/etc/crowdsec/local_api_credentials.yaml.local'
Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /etc/crowdsec/hub
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log
   - Log level              : info
   - Log Media              : stdout
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              :
API Client:
  - URL                     : ${CONNECT_URI}
  - Login                   : localhost
  - Credentials File        : /etc/crowdsec/local_api_credentials.yaml
Local API Server:
  - Listen URL              : 0.0.0.0:8080
  - Listen Socket           :
  - Profile File            : /etc/crowdsec/profiles.yaml
  - Cert File : /tls/server-cert.pem
  - Key File  : /tls/server-key.pem

  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : postgresql
      - Host                : db
      - Port                : 5432
      - User                : crowdsec
      - DB Name             : crowdsec
      - Flush age           : 7d
      - Flush size          : 5000

d99d6639663a:/# cscli lapi status
INFO[2024-06-12T13:31:24+02:00] Loading yaml file: '/etc/crowdsec/local_api_credentials.yaml' with additional values from '/etc/crowdsec/local_api_credentials.yaml.local'
INFO[2024-06-12T13:31:24+02:00] Loaded credentials from /etc/crowdsec/local_api_credentials.yaml
INFO[2024-06-12T13:31:24+02:00] Trying to authenticate with username localhost on ${CONNECT_URI}
FATA[2024-06-12T13:31:24+02:00] failed to authenticate to Local API (LAPI): BaseURL must have a trailing slash, but "$%7BCONNECT_URI%7D" does not

Anything else we need to know?

No response

Crowdsec version

```console $ cscli version d99d6639663a:/# cscli version 2024/06/12 13:34:19 version: v1.6.1-c6e40191 2024/06/12 13:34:19 Codename: alphaga 2024/06/12 13:34:19 BuildDate: 2024-04-18_13:47:41 2024/06/12 13:34:19 GoVersion: 1.21.9 2024/06/12 13:34:19 Platform: docker 2024/06/12 13:34:19 libre2: C++ 2024/06/12 13:34:19 Constraint_parser: >= 1.0, <= 3.0 2024/06/12 13:34:19 Constraint_scenario: >= 1.0, <= 3.0 2024/06/12 13:34:19 Constraint_api: v1 2024/06/12 13:34:19 Constraint_acquis: >= 1.0, < 2.0 ```

OS version

```console # On Linux: $ cat /etc/os-release # paste output here $ uname -a # paste output here # On Windows: C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture # paste output here ``` CS is used in a docker environment

Enabled collections and parsers

```console $ cscli hub list -o raw # paste output here ```

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* # paste output here # On Windows: C:\> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml # paste output here

Config show

```console $ cscli config show INFO[2024-06-12T13:35:31+02:00] Loading yaml file: '/etc/crowdsec/local_api_credentials.yaml' with additional values from '/etc/crowdsec/local_api_credentials.yaml.local' Global: - Configuration Folder : /etc/crowdsec - Data Folder : /var/lib/crowdsec/data - Hub Folder : /etc/crowdsec/hub - Simulation File : /etc/crowdsec/simulation.yaml - Log Folder : /var/log - Log level : info - Log Media : stdout Crowdsec: - Acquisition File : /etc/crowdsec/acquis.yaml - Parsers routines : 1 - Acquisition Folder : /etc/crowdsec/acquis.d cscli: - Output : human - Hub Branch : API Client: - URL : ${CONNECT_URI} - Login : localhost - Credentials File : /etc/crowdsec/local_api_credentials.yaml Local API Server: - Listen URL : 0.0.0.0:8080 - Listen Socket : - Profile File : /etc/crowdsec/profiles.yaml - Cert File : /tls/server-cert.pem - Key File : /tls/server-key.pem - Trusted IPs: - 127.0.0.1 - ::1 - Database: - Type : postgresql - Host : db - Port : 5432 - User : crowdsec - DB Name : crowdsec - Flush age : 7d - Flush size : 5000 ```

Prometheus metrics

```console $ cscli metrics # paste output here ```

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

github-actions[bot] commented 2 weeks ago

@andreasbrett: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

Check Crowdsec Documentation to see if your issue can be self resolved.
You can also join our Discord.
Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/crowdsec/blob/master/.github/governance.yml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the [BirthdayResearch/oss-governance-bot](https://github.com/BirthdayResearch/oss-governance-bot) repository.

andreasbrett commented 2 weeks ago

Side note: other env var expands work. I furthermore assume this has to do with the following change https://github.com/crowdsecurity/crowdsec/pull/2012

crowdsecurity / crowdsec