grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.68k stars 3.42k forks source link

How do you process syslog messages using the syslog stage on Promtail for Cisco devices? #10948

Open dellthePROgrammer opened 1 year ago

dellthePROgrammer commented 1 year ago

Hello, all, I have been wacking my head around trying to ingest logs of our Cisco devices. I have made a job within our Promtail config with syslog as the stage for the job, and I'm also parsing the log using regex. I know the regex is good (I check an online source and it can parse my logs, which I am watching from Wireshark) and promtail is seeing the logs via the UDP port but it comes up with an error saying

level=warn ts=2023-10-17T18:03:21.5753201Z caller=syslogtarget.go:105 msg="error parsing syslog stream" err="expecting a version value in the range 1-999 [col 5]"
level=warn ts=2023-10-17T18:03:50.8143289Z caller=syslogtarget.go:105 msg="error parsing syslog stream" err="expecting a version value in the range 1-999 [col 5]"
level=warn ts=2023-10-17T18:04:14.4202125Z caller=syslogtarget.go:105 msg="error parsing syslog stream" err="expecting a version value in the range 1-999 [col 5]"

I have tried almost everything to try and get this working, even using a static config and changing the path to udp://0.0.0.0:514 which also doesn't seem to work. If anyone has any ideas, I'm all ears.

Thanks

EDIT 1:

Sorry I forgot to send my config

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: ./position.yaml

clients:
  - url: http://10.0.18.213:30000/loki/api/v1/push

scrape_configs:
  - job_name: windows-application
    pipeline_stages:
      - match:
          selector: '{job="windows-application"}'
          stages:
            - json:
                expressions:
                  message: "message"
                  levelText: "levelText"
                  event_id: "event_id"
                  keywords: "keywords"
                  level: "level"
                  opCodeText: "opCodeText"
                  source: "source"
                  time: "timeCreated"
                  taskText: "taskText"
                  task: "task"
                  processName: "execution_processName"
            - eventlogmessage:
                source: message
                overwrite_existing: true
            - pack:
                labels:
                  - message
                  - levelText
                  - event_id
                  - keywords
                  - level
                  - opCodeText
                  - source
                  - time
                  - taskText
                  - task
                  - processName
    windows_events:
      use_incoming_timestamp: false
      bookmark_path: ./bookmark-app.xml
      eventlog_name: Application
      xpath_query: '*'
      exclude_event_data: false
      labels:
        job: windows-application
        location: SouthField
    relabel_configs:
      - source_labels: ['computer']
        target_label: 'host'

  - job_name: windows-security
    pipeline_stages:
      - match:
          selector: '{job="windows-security"}'
          stages:
            - json:
                expressions:
                  message: "message"
                  levelText: "levelText"
                  event_id: "event_id"
                  keywords: "keywords"
                  level: "level"
                  opCodeText: "opCodeText"
                  source: "source"
                  time: "timeCreated"
                  taskText: "taskText"
                  task: "task"
                  processName: "execution_processName"
            - eventlogmessage:
                source: message
                overwrite_existing: true
            - pack:
                labels:
                  - message
                  - levelText
                  - event_id
                  - keywords
                  - level
                  - opCodeText
                  - source
                  - time
                  - taskText
                  - task
                  - processName
    windows_events:
      use_incoming_timestamp: false
      bookmark_path: ./bookmark-sec.xml
      eventlog_name: Security
      xpath_query: '*'
      exclude_event_data: false
      labels:
        job: windows-security
        location: SouthField
    relabel_configs:
      - source_labels: ['computer']
        target_label: 'host'

  - job_name: windows-system
    pipeline_stages:
      - match:
          selector: '{job="windows-system"}'
          stages:
            - json:
                expressions:
                  message: "message"
                  levelText: "levelText"
                  event_id: "event_id"
                  keywords: "keywords"
                  level: "level"
                  opCodeText: "opCodeText"
                  source: "source"
                  time: "timeCreated"
                  taskText: "taskText"
                  task: "task"
                  processName: "execution_processName"
            - eventlogmessage:
                source: message
                overwrite_existing: true
            - pack:
                labels:
                  - message
                  - levelText
                  - event_id
                  - keywords
                  - level
                  - opCodeText
                  - source
                  - time
                  - taskText
                  - task
                  - processName
    windows_events:
      use_incoming_timestamp: false
      bookmark_path: ./bookmark-sys.xml
      eventlog_name: System
      xpath_query: '*'
      exclude_event_data: false
      labels:
        job: windows-system
        location: SouthField
    relabel_configs:
      - source_labels: ['computer']
        target_label: 'host'

  - job_name: syslog
    syslog: 
      listen_address: "0.0.0.0:514"
      listen_protocol: "udp"
      idle_timeout: 60s
      label_structured_data: true
      labels: 
        job: "syslog"
        type: "syslog"

    pipeline_stages:
      - match:
          selector: '{job="syslog"}'
          stages:
            - regex:
                expression: '/^(?P<seqno>\<[0-9]+\>)(?P<timestamp>[A-Za-z]{3} \d{1,2} \d{4} \d{2}:\d{2}:\d{2}): (?P<facility>\%[A-Z0-9]{3}\-[0-9]\-[0-9]+):(?P<message>.*)$'
            - labels:
                summary:
                timestamp:
                facility:
                severity:
                message:
xh63 commented 1 year ago

you do not send the cisco syslog to promtail directly, the loki won't like the cisco syslog format. you setup a syslog/rsyslog server in front of the promtail and then forward the transformed syslog to promtail. I just did this recently with a good success by following the articles below and google. Few articles for you https://alexandre.deverteuil.net/post/syslog-relay-for-loki/ https://www.syslog-ng.com/community/b/blog/posts/sending-logs-from-syslog-ng-to-grafana-loki https://grafana.com/blog/2021/03/23/how-i-fell-in-love-with-logs-thanks-to-grafana-loki/