A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Hi, I really hope that log_to_metric transformer could tag expansion similar to Loki's label expansion.

This is what I got from tap (output from alloy_web_vitals_fcp) Notice that the tag contains actual * sign.

{"name":"value_fcp_total","namespace":"faro_web_vitals","tags":{"*":"{\"app\":\"@faro-sdk\",\"app_environment\":\"beta\",\"app_name\":\"my-app\",\"app_version\":\"dev\",\"browser_name\":\"Chrome\",\"browser_os\":\"os\",\"browser_version\":\"random\"}"},"timestamp":"2024-11-16T14:51:51.072530Z","kind":"absolute","gauge":{"value":85.29999923706055}}
{"name":"value_fcp_total","namespace":"faro_web_vitals","tags":{"*":"{\"app\":\"@faro-sdk\",\"app_environment\":\"beta\",\"app_name\":\"my-app\",\"app_version\":\"latest\",\"browser_name\":\"Chrome\",\"browser_os\":\"os\",\"browser_version\":\"random\"}"},"timestamp":"2024-11-16T14:51:54.071667Z","kind":"absolute","gauge":{"value":190.8000030517578}}

The full self-contained config is provides as below session.

data_dir: "${VECTOR_DATA_DIR}"
api:
  enabled: true
  graphql: true
  playground: true
  address: "0.0.0.0:8686"
sources:
  faro_receiver_logs:
    type: demo_logs
    format: shuffle
    lines:
    - entry="timestamp=\"2024-11-15 10:07:17.856 +0000 UTC\" kind=measurement type=web-vitals ttfb=8.799999 value_ttfb=8.799999237060547 sdk_version=latest app_name=my-app app_version=dev app_environment=beta  browser_name=Chrome browser_version=random browser_os=\"os\" browser_mobile=true view_name=default"
    - entry="timestamp=\"2024-11-15 10:07:17.856 +0000 UTC\" kind=measurement type=web-vitals fcp=85.299999 value_fcp=85.29999923706055 sdk_version=latest app_name=my-app app_version=dev app_environment=beta  browser_name=Chrome browser_version=random browser_os=\"os\" browser_mobile=true view_name=default"
    - entry="timestamp=\"2024-11-15 10:07:17.531 +0000 UTC\" kind=measurement type=web-vitals ttfb=20.400002 value_ttfb=20.400001525878906 sdk_version=latest app_name=my-app app_version=latest app_environment=beta  browser_name=Chrome browser_version=random browser_os=\"os\" browser_mobile=true"
    - entry="timestamp=\"2024-11-15 10:07:17.492 +0000 UTC\" kind=measurement type=web-vitals fcp=190.800003 value_fcp=190.8000030517578 sdk_version=latest app_name=my-app app_version=latest app_environment=beta  browser_name=Chrome browser_version=random browser_os=\"os\" browser_mobile=true"
    - entry="timestamp=\"2024-11-15 10:07:17.231 +0000 UTC\" kind=measurement type=web-vitals ttfb=22.500000 value_ttfb=22.5 sdk_version=latest app_name=my-app app_version=latest app_environment=beta  browser_name=Chrome browser_version=random browser_os=\"os\" browser_mobile=true"
    - entry="timestamp=\"2024-11-15 10:07:17.228 +0000 UTC\" kind=measurement type=web-vitals fcp=210.099998 value_fcp=210.0999984741211 sdk_version=latest app_name=my-app app_version=latest app_environment=beta  browser_name=Chrome browser_version=random browser_os=\"os\" browser_mobile=true"
    - entry="timestamp=\"2024-11-15 10:07:16.995 +0000 UTC\" kind=measurement type=web-vitals cls=0.439619 value_cls=0.4396192758193616 sdk_version=latest app_name=my-app app_version=dev app_environment=beta browser_name=Chrome browser_version=random browser_os=\"os\" browser_mobile=true view_name=default"
transforms:
  alloy_preprocessor:
    type: "remap"
    inputs: [ faro_receiver_logs ]
    source: |
      . = parse_key_value!(.message)
      .loki_labels = {
        "app": "@faro-sdk",
      }
      .prom_labels = .loki_labels
      .parsed_entry = parse_key_value!(.entry)
      optional_labels = [
        "app_name",
        "app_environment",
        "app_version",
        "browser_name",
        "browser_version",
        "browser_os",
      ]
      # only add app related labels to loki_labels
      for_each(optional_labels) -> |_, l| {
        option_label = get!(value: .parsed_entry, path: [l])
        if option_label != null {
          .prom_labels = set!(.prom_labels, [l], option_label)
          if starts_with(l, "app") {
            .loki_labels = set!(.loki_labels, [l], option_label)
          }
        }
      }
      .message = .entry
  alloy_metrics_route:
    type: route
    reroute_unmatched: false
    inputs:
    - alloy_preprocessor
    route:
      web_vitals: .parsed_entry.kind == "measurement"
  alloy_web_vitals_preprocessor:
    type: "remap"
    inputs: [ alloy_metrics_route.web_vitals ]
    source: |
      web_vitals = [
        "value_cls",
        "value_inp",
        "value_fid",
        "value_lcp",
        "value_ttfb",
        "value_fcp",
      ]
      for_each(web_vitals) -> |_, v| {
        metric = get!(value: .parsed_entry, path: [v])
        if metric != null {
          metricf, err = parse_float(metric)
          if err != null {
            log(err, level: "error")
          }
          . = set!(., ["{{ v }}_total"], metricf)
        }
      }
  web_vitals_route:
    type: route
    reroute_unmatched: false
    inputs:
    - alloy_web_vitals_preprocessor
    route:
      fcp: .value_fcp_total != null

  alloy_web_vitals_fcp:
    type: log_to_metric
    inputs:
    - web_vitals_route.fcp
    metrics:
    - type: gauge
      field: value_fcp_total
      namespace: faro_web_vitals
      tags:
        "*": "{{ .prom_labels }}"
sinks:
  prom_exporter:
    type: prometheus_exporter
    inputs:
    - alloy_web_vitals_fcp
    address: 0.0.0.0:9090

Attempted Solutions

Provide each defined tag in tags field, which is very annoyed.

Proposal

Support tags expansion for log_to_metric transformer. I am really wanting to volunteer to support this feature.

References

14744

Version

0.42.0-distroless-libc

vectordotdev / vector

Provide `tags expansion` similar to Loki's label expansion #21815