Relabeling in Promtail is impossibe of syslog rfc5424 data with "." in the id or "-" in label name.

Describe the bug I'm trying to ingest FW flow data from Juniper FW as syslog rfc5424 stream.

An example of the log entry.

Msg: 1 2023-04-09T13:12:48.660+10:00 j210h RT_FLOW - RT_FLOW_SESSION_CLOSE [junos@2636.1.1.1.2.36 reason="idle Timeout" source-address="a.a.a.a" source-port="12" destination-address="b.b.b.b" destination-port="12" service-name="icmp" nat-source-address="x.x.x.x" nat-source-port="22265" nat-destination-address="y.y.y.y" nat-destination-port="12" src-nat-rule-type="source rule" src-nat-rule-name="source-nat-rule" dst-nat-rule-type="N/A" dst-nat-rule-name="N/A" protocol-id="1" policy-name="Outbound" source-zone-name="trust" destination-zone-name="untrust" session-id-32="95612" packets-from-client="1" bytes-from-client="40" packets-from-server="0" bytes-from-server="0" elapsed-time="60" application="UNKNOWN" nested-application="UNKNOWN" username="N/A" roles="N/A" packet-incoming-interface="vlan.10" encrypted="UNKNOWN"] session closed idle Timeout: x.x.x.x/12->y.y.y.y/12 icmp x.x.x.x/22265->y.y.y.y/12 source rule source-nat-rule N/A N/A 1 Outbound trust untrust 95612"

Problem is with id part of the message junos@2636.1.1.1.2.36 Per documentation promtail configured with label_structured_data: yes should transform first label to to label
__syslog_message_sd_junos_2636.1.1.1.2.36_reason.

But I can't use label like that in relabel config, config as bellow

     - source_labels: ['__syslog_message_sd_junos_2636.1.1.1.2.36_reason']
       target_label: 'reason'

Promtail generates error:

Unable to parse config: /etc/promtail/config.yml: "__syslog_message_sd_junos_2636.1.1.1.2.36_reason" is not a valid label name. Use `-config.expand-env=true` flag if you want to expand environment variables in your config file

Also '-' in source-address address also causing same problem.

In short, incoming syslog labels are not sanitized enough, or promtail is too strict with name of internal labels.

This piece of code from https://github.com/grafana/loki/blob/main/clients/pkg/promtail/targets/syslog/syslogtarget.go is to blame.

    if t.config.LabelStructuredData && rfc5424Msg.StructuredData != nil {
        for id, params := range *rfc5424Msg.StructuredData {
            id = strings.ReplaceAll(id, "@", "_")
            for name, value := range params {
                key := "__syslog_message_sd_" + id + "_" + name
                lb.Set(key, value)
            }
        }
    }

Replacing just @ is not enough. Happy to submit pull request with patch but I guess it faster to fix then accept a pull request

To Reproduce

Use promtail Config

scrape_configs:
  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:1514
      listen_protocol: tcp
      idle_timeout: 60s
      label_structured_data: yes
      use_incoming_timestamp: yes
      labels:
        job: "syslog"
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'
      - source_labels: ['__syslog_message_sd_junos_2636.1.1.1.2.36_reason']
        target_label: 'reason'

Expected behavior Syslog message like above should be labeled with label reason="idle Timeout"

Observed behavior Promtail failed to parse config producing error Unable to parse config: /etc/promtail/config.yml: "__syslog_message_sd_junos_2636.1.1.1.2.36_reason" is not a valid label name. Use `-config.expand-env=true` flag if you want to expand environment variables in your config file

Environment:

promtail 2.8.0
syslog-ng in front of it.

grafana / loki

Relabeling in Promtail is impossibe of syslog rfc5424 data with "." in the id or "-" in label name. #9113