Add support for configuration via YAML/JSON

[wilfriedroset]

Background

When deploying grafana-agent-flow on baremetal/vm to gather observability materials (metrics, logs, profiles...) the end users might want to use a configuration management tool such as Puppet or Ansible. The new configuration, River, is powerful but also introduce a fairly complex templating for configuration management. Below is an example around profiles scraping:

logging {
    level = "<%= $log_level %>"
}

pyroscope.write "default" {
    endpoint {
        url = "<%= $pyroscope_endpoint %>"

        basic_auth {
            username = "<%= $pyroscope_basic_auth_username %>"
            password = "<%= $pyroscope_basic_auth_password %>"
        }
    }

    <%- if $external_labels.length > 0 { -%>
    external_labels = {
        <%- $external_labels.each |$key, $value| { -%>
            "<%= $key %>" = "<%= $value %>",
        <%- } -%>
    }
    <%- } -%>
}

pyroscope.scrape "default" {
    targets = [
        <%- $scrape_targets.each |$service_name, $address| { -%>
        { "__address__" = "<%= $address %>", "service_name" = "<%= $service_name %>" },
        <%- } -%>
    ]
    forward_to = [pyroscope.write.default.receiver]

    scrape_interval = "<%= $scrape_interval %>"
    scheme = "<%= $scrape_scheme %>"

    <%- if $scrape_insecure_skip_verify { -%>
    tls_config {
        insecure_skip_verify = <%= String($scrape_insecure_skip_verify, "%t") %>
    }
    <% } -%>

    <% if $job_name { -%>
    job_name = "<%= $job_name %>"
    <% } -%>

    profiling_config {

        profile.goroutine {
            enabled = <%= String($goroutine_profiling_enabled, "%t") %>
            path = "<%= $goroutine_profiling_path %>"
            delta = false
        }

        profile.process_cpu {
            enabled = <%= String($process_cpu_profiling_enabled, "%t") %>
            path = "<%= $process_cpu_profiling_path %>"
            delta = true
        }

        profile.memory {
            enabled = <%= String($memory_profiling_enabled, "%t") %>
            path = "<%= $memory_profiling_path %>"
            delta = false
        }

        profile.mutex {
            enabled = <%= String($mutex_profiling_enabled, "%t") %>
            path = "<%= $mutex_profiling_path %>"
            delta = false
        }

        profile.block {
            enabled = <%= String($block_profiling_enabled, "%t") %>
            path = "<%= $block_profiling_path %>"
            delta = false
        }
    }
}

The associated Puppet code to deploy the template above could be similar to:

  file { '/etc/grafana-agent-flow.river':
    ensure  => 'file',
    content => epp('grafana_agent/config.river.epp',
      {
        'pyroscope_endpoint'            => $pyroscope_endpoint,
        'pyroscope_basic_auth_username' => $pyroscope_basic_auth_username,
        'pyroscope_basic_auth_password' => $pyroscope_basic_auth_password,
        'external_labels'               => $external_labels,
        'job_name'                      => $job_name,
        'log_level'                     => $log_level,
        'scrape_targets'                => $scrape_targets,
        'scrape_scheme'                 => $scrape_scheme,
        'scrape_interval'               => $scrape_interval,
        'scrape_insecure_skip_verify'   => $scrape_insecure_skip_verify,
        'goroutine_profiling_enabled'   => $goroutine_profiling_enabled,
        'goroutine_profiling_path'      => $goroutine_profiling_path,
        'process_cpu_profiling_enabled' => $process_cpu_profiling_enabled,
        'process_cpu_profiling_path'    => $process_cpu_profiling_path,
        'memory_profiling_enabled'      => $memory_profiling_enabled,
        'memory_profiling_path'         => $memory_profiling_path,
        'mutex_profiling_enabled'       => $mutex_profiling_enabled,
        'mutex_profiling_path'          => $mutex_profiling_path,
        'block_profiling_enabled'       => $block_profiling_enabled,
        'block_profiling_path'          => $block_profiling_path,
      }),
    owner   => $owner,
    group   => $group,
    mode    => '0640',
  }

The template can be optimized to be less verbose and more flexible but the general idea stays the same, we need to take into account possible cases before hand and maintain a complex template.

Proposal

One alternative to that would be to configuration grafana-agent-flow with either yaml or JSON. The proposal has already been discussed before here.

I don’t know the implied complexity of such request but it would facilitate everything from the configuration management point. Puppet, but also Ansible, knows how to merge several configuration pieces but also render the result of this merge in a jsonor yaml file. This also apply to Terraform as well.

@rfratto examples in his proposal enable such Puppet code:

  file { '/etc/grafana-agent-flow.river':
    ensure       => 'file',
    content      => to_yaml($config_hash),
    owner        => $owner,
    group        => $group,
    mode         => '0640',
  }

The Ansible alternative could be similar to:

  - name: create grafana agent flow configuration
    ansible.builtin.copy:
      dest: /etc/grafana-agent-flow.river
      content: "{{ config_hash | to_nice_yaml }}"

The end users could benefits from both its configuration management tools feature and grafana-agent-flow. The code becomes less verbose and there is no need to maintain a template.

[rfratto]

I discussed this with the wider team of developers working on the agent. The current consensus is that while this is a problem we're hearing more and more, we don't have the time to prioritize it right now, as we're currently working towards on solidifying a path to a 1.0 release with the features we have.

We're planning to place this proposal on hold until after 1.0; though prioritization may change if demand for this skyrockets.

[rfratto]

I'm going to remove the proposal label from this; I think this is a good enhancement request (and something currently on my radar), but at the moment it lacks sufficient detail for what YAML/JSON support would look like to be considered a proposal. I anticipate having a more detailed proposal for this within the coming months.

[thequailman]

FWIW I don't see the value of river and the expressiveness of the config. I'm deploying alloy from a configuration management tool or Helm, all of the logic you're trying to abstract in the config is part of my configuration management tool. Please let us provide a vanilla YAML or JSON file without any fancy interpretation from Alloy!