Config reloads are not done if only a secret has changed

What's wrong?

Secrets (e.g. passwords) in configs files are hidden during marshalling to yaml. They are displayed as <secret>, in order to not expose them in logs and UIs. This causes issues for a static mode feature that configs won't be reloaded if a reload was triggered but the config hasn't changed. When Agent checks if the config changed, it marshals the yaml but the secrets are displayed as <secret> and it doesn't detect if they've changed.

There is a NoScrubbedSecretsHook function which is often used in our codebase to prevent this problem. However, it doesn't aways work. We may need to compare the config structs instead of marshaling to yaml.

This bug may be present for all of logs, metrics, and traces subsystems. Also, NoScrubbedSecretsHook currently doesn't handle OTel secrets. And I'm not sure if we can implement it in a way that it will catch all possible secret types that come up in the future.

Steps to reproduce

Run a config like this:

metrics:
  configs:
  - scrape_configs:
    - job_name: ""
      honor_timestamps: false
      track_timestamps_staleness: false
      basic_auth:
        username: user
        password: pass
      follow_redirects: false
      enable_http2: false

Then change the password in the config file and reload it. Check if the config was really reloaded. Do a similar test for logs and traces.

To illustrate the issue with NoScrubbedSecretsHook, this test doesn't pass:

func TestMarshalWithHook2(t *testing.T) {
    cfg := metrics.Config{
        Configs: []instance.Config{
            {
                ScrapeConfigs: []*prom_config.ScrapeConfig{
                    {
                        HTTPClientConfig: config_util.HTTPClientConfig{
                            BasicAuth: &config_util.BasicAuth{
                                Username: "user",
                                Password: config_util.Secret("pass"),
                            },
                        },
                    },
                },
            },
        },
    }

    expectedYaml := `configs:
- scrape_configs:
  - job_name: ""
    honor_timestamps: false
    track_timestamps_staleness: false
    basic_auth:
      username: user
      password: pass
    follow_redirects: false
    enable_http2: false
`

    actualYaml, err := util.MarshalWithHook(cfg, util.NoScrubbedSecretsHook)
    require.NoError(t, err)

    require.Equal(t, expectedYaml, string(actualYaml))
}

The reason is the call to MarshalConfig(&c, true) in /static/metrics/instance/instance.go. I can't think of a good way to propagate the preference to not marshal to <secret>. Global variables won't work, since there could be more than one thread marshalling at any given time.

System information

No response

Software version

No response

Configuration

No response

Logs

No response

grafana / agent