Env variable substitution not working when using target allocator and custom scrape-config

[cmergenthaler]

Component(s)

collector, target allocator

What happened?

Description

When using target allocator together with custom scrape configs, env variable substitution on the collector does not seem to work properly.

Steps to Reproduce

• Having a remote prometheus metrics endpoint, which should be scraped by custom scrape-config
• Scrape target in otel-collector is configured using env variables
• Env variables are set on otel-collector and target-allocator

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: remotepromendpoint
          scheme: https
          basic_auth:
            username: "${env::USER_ENV}"
            password_file: /etc/my-mounted-password
          tls_config:
            insecure_skip_verify: true
          static_configs:
            - targets: [ "${env::URL_ENV}" ]

Expected Result

Env variables are substituted by otel-collector and/or target-allocator

Actual Result

Otel-collector tries to scrape https://${env::URL_ENV}/metrics

Kubernetes Version

1.26.6

Operator version

0.85.0

Collector version

0.85.0

Environment information

No response

Log output

No response

Additional context

No response

[jaronoff97]

@anunarapureddy any ideas what may be happening here? I have a few theories, but don't have a ton of time to look in to this currently.

[anunarapureddy]

@jaronoff97 I can take a look into this issue.

[cole-christensen]

I am having the same issue, looks like it has been a while since there was any activity here

[jaronoff97]

@cole-christensen Can you link your configuration and explain what isn't working? Also share any logs from the operator, collector, target allocator that may be relevant when you get a chance. thank you!