[sophos][xg] Failed to parse field

StefanSa commented 1 year ago

Hi there I have a problem here which i do not quite understand. Have deployed sophos.utm/xg (v2.11.0) integration here, via elastic agent (v8.8.2). While sophos.utm parses the fields without problems, this does not happen with sophos.xg. Only the message is transmitted, but not the fields are decoded. The interesting thing is, if i test the pipeline logs-sophos.xg-2.11.0 in Kibana with this message, all fields are decoded correctly (source.ip etc.). This almost looks like the elastic.agent for sophos.xg is not selecting the correct pipeline.

Does anyone have any idea about this behavior ?

logstash output pipeline:

output {
  if "elastic-agent" in [tags] {
      if [metadata][pipeline] {
        elasticsearch {
          hosts => "securityonion"
          ecs_compatibility => v8
          data_stream => true
          user => "so_elastic"
          password => "secret"
          pipeline => "%{[metadata][pipeline]}"
          ssl => true
          ssl_certificate_verification => false
        }
      }
      else {
        elasticsearch {
          hosts => "securityonion"
          ecs_compatibility => v8
          data_stream => true
          user => "so_elastic"
          password => "secret"
          ssl => true
          ssl_certificate_verification => false
        }
     }
  }
}

The transfered value that contains only the message:

{
  "_index": ".ds-logs-sophos.xg-default-2023.09.28-000001",
  "_id": "y3Nc4IoBmLx_PdbIWrDd",
  "_version": 1,
  "_score": 0,
  "_source": {
    "agent": {
      "name": "ingest01",
      "id": "f5040c77-13ff-41b2-8117-a9e105a5a5dc",
      "type": "filebeat",
      "ephemeral_id": "abc517b6-4f46-4088-b665-9ae668096a4c",
      "version": "8.8.2"
    },
    "metadata": {
      "input": {
        "beats": {
          "host": {
            "ip": "172.18.34.221"
          }
        }
      },
      "raw_index": "logs-sophos.xg-default",
      "stream_id": "udp-sophos.xg-419f6380-5e0b-11ee-9ce0-bfd41b12e650",
      "beat": "filebeat",
      "truncated": false,
      "type": "_doc",
      "version": "8.8.2",
      "input_id": "udp-sophos-419f6380-5e0b-11ee-9ce0-bfd41b12e650"
    },
    "log": {
      "source": {
        "address": "192.168.50.50:42611"
      }
    },
    "elastic_agent": {
      "id": "f5040c77-13ff-41b2-8117-a9e105a5a5dc",
      "version": "8.8.2",
      "snapshot": false
    },
    "_conf": {
      "default": "fwgate01.firewall.local",
      "mappings": [
        {
          "hostname": "fwgate01.firewall.local",
          "serial_number": "123456578"
        }
      ],
      "tz_offset": "UTC"
    },
    "message": "<30>device_name=\"SFW\" timestamp=\"2023-09-29T11:54:23+0200\" device_model=\"XG230\" device_serial_id=\"12345678\" log_id=\"010101600001\" log_type=\"Firewall\" log_component=\"Firewall Rule\" log_subtype=\"Allowed\" log_version=1 severity=\"Information\" fw_rule_id=\"60\" fw_rule_name=\"VPN-Wecker_LAN\" fw_rule_section=\"Local rule\" nat_rule_id=\"0\" fw_rule_type=\"USER\" ips_policy_id=17 ether_type=\"Unknown (0x0000)\" in_interface=\"xfrm4\" out_interface=\"Port1\" src_mac=\"C4:F7:D5:B5:47:F4\" dst_mac=\"C8:4F:86:04:0E:71\" src_ip=\"172.17.35.157\" src_country=\"R1\" dst_ip=\"172.16.34.11\" dst_country=\"R1\" protocol=\"UDP\" src_port=59497 dst_port=389 src_zone_type=\"VPN\" src_zone=\"VPN\" dst_zone_type=\"LAN\" dst_zone=\"LAN\" con_event=\"Start\" con_id=\"675172385\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"xfrm4\" out_display_interface=\"Port1\" log_occurrence=\"1\"",
    "tags": [
      "sophos-xg",
      "forwarded",
      "elastic-agent",
      "input-securityonion",
      "beats_input_codec_plain_applied",
      "xg"
    ],
    "input": {
      "type": "udp"
    },
    "@timestamp": "2023-09-29T09:54:23.037Z",
    "ecs": {
      "version": "8.0.0"
    },
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "sophos.xg"
    },
    "@version": "1",
    "event": {
      "agent_id_status": "auth_metadata_missing",
      "ingested": "2023-09-29T09:54:23Z",
      "timezone": "+02:00",
      "module": "sophos",
      "dataset": "sophos.xg"
    }
  },
  "fields": {
    "elastic_agent.version": [
      "8.8.2"
    ],
    "_conf.mappings.hostname": [
      "fwgate01.firewall.local"
    ],
    "_conf.mappings.serial_number": [
      "1234567"
    ],
    "_conf.default": [
      "fwgate01.firewall.local"
    ],
    "agent.type": [
      "filebeat"
    ],
    "event.module": [
      "sophos"
    ],
    "metadata.stream_id": [
      "udp-sophos.xg-419f6380-5e0b-11ee-9ce0-bfd41b12e650"
    ],
    "@version": [
      "1"
    ],
    "agent.name": [
      "ingest01"
    ],
    "elastic_agent.snapshot": [
      false
    ],
    "metadata.raw_index": [
      "logs-sophos.xg-default"
    ],
    "event.agent_id_status": [
      "auth_metadata_missing"
    ],
    "event.timezone": [
      "+02:00"
    ],
    "metadata.type": [
      "_doc"
    ],
    "_conf.tz_offset": [
      "UTC"
    ],
    "metadata.beat": [
      "filebeat"
    ],
    "elastic_agent.id": [
      "f5040c77-13ff-41b2-8117-a9e105a5a5dc"
    ],
    "metadata.input.beats.host.ip": [
      "172.16.34.221"
    ],
    "data_stream.namespace": [
      "default"
    ],
    "metadata.version": [
      "8.8.2"
    ],
    "metadata.truncated": [
      false
    ],
    "input.type": [
      "udp"
    ],
    "message": [
      "<30>device_name=\"SFW\" timestamp=\"2023-09-29T11:54:23+0200\" device_model=\"XG230\" device_serial_id=\"123456\" log_id=\"010101600001\" log_type=\"Firewall\" log_component=\"Firewall Rule\" log_subtype=\"Allowed\" log_version=1 severity=\"Information\" fw_rule_id=\"60\" fw_rule_name=\"VPN-Wecker_LAN\" fw_rule_section=\"Local rule\" nat_rule_id=\"0\" fw_rule_type=\"USER\" ips_policy_id=17 ether_type=\"Unknown (0x0000)\" in_interface=\"xfrm4\" out_interface=\"Port1\" src_mac=\"C4:F7:D5:B5:47:F4\" dst_mac=\"C8:4F:86:04:0E:71\" src_ip=\"172.18.35.157\" src_country=\"R1\" dst_ip=\"172.16.34.11\" dst_country=\"R1\" protocol=\"UDP\" src_port=59497 dst_port=389 src_zone_type=\"VPN\" src_zone=\"VPN\" dst_zone_type=\"LAN\" dst_zone=\"LAN\" con_event=\"Start\" con_id=\"675172385\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"xfrm4\" out_display_interface=\"Port1\" log_occurrence=\"1\""
    ],
    "metadata.input_id": [
      "udp-sophos-419f6380-5e0b-11ee-9ce0-bfd41b12e650"
    ],
    "data_stream.type": [
      "logs"
    ],
    "tags": [
      "sophos-xg",
      "forwarded",
      "elastic-agent",
      "input-securityonion",
      "beats_input_codec_plain_applied",
      "xg"
    ],
    "event.ingested": [
      "2023-09-29T09:54:23.000Z"
    ],
    "@timestamp": [
      "2023-09-29T09:54:23.037Z"
    ],
    "agent.id": [
      "f5040c77-13ff-41b2-8117-a9e105a5a5dc"
    ],
    "ecs.version": [
      "8.0.0"
    ],
    "log.source.address": [
      "192.168.50.50:42611"
    ],
    "data_stream.dataset": [
      "sophos.xg"
    ],
    "agent.ephemeral_id": [
      "abc517b6-4f46-4088-b665-9ae668096a4c"
    ],
    "agent.version": [
      "8.8.2"
    ],
    "event.dataset": [
      "sophos.xg"
    ]
  }
}

Correct result with the test of the pipeline:

{
  "docs": [
    {
      "doc": {
        "_index": ".ds-logs-sophos.xg-default-2023.09.28-000001",
        "_id": "y3Nc4IoBmLx_PdbIWrDd",
        "_version": "1",
        "_source": {
          "agent": {
            "name": "ingest01",
            "id": "f5040c77-13ff-41b2-8117-a9e105a5a5dc",
            "type": "filebeat",
            "ephemeral_id": "abc517b6-4f46-4088-b665-9ae668096a4c",
            "version": "8.8.2"
          },
          "metadata": {
            "input": {
              "beats": {
                "host": {
                  "ip": "172.17.34.221"
                }
              }
            },
            "raw_index": "logs-sophos.xg-default",
            "stream_id": "udp-sophos.xg-419f6380-5e0b-11ee-9ce0-bfd41b12e650",
            "beat": "filebeat",
            "truncated": false,
            "type": "_doc",
            "version": "8.8.2",
            "input_id": "udp-sophos-419f6380-5e0b-11ee-9ce0-bfd41b12e650"
          },
          "log": {
            "level": "Information",
            "source": {
              "address": "192.168.50.50:42611"
            }
          },
          "elastic_agent": {
            "id": "f5040c77-13ff-41b2-8117-a9e105a5a5dc",
            "version": "8.8.2",
            "snapshot": false
          },
          "destination": {
            "port": 389,
            "ip": "172.18.34.11",
            "mac": "C8-4F-86-04-0E-71"
          },
          "rule": {
            "id": "60"
          },
          "source": {
            "port": 59497,
            "ip": "172.17.35.157",
            "mac": "C4-F7-D5-B5-47-F4"
          },
          "tags": [
            "sophos-xg",
            "forwarded",
            "elastic-agent",
            "input-securityonion",
            "beats_input_codec_plain_applied",
            "xg"
          ],
          "network": {
            "community_id": "1:q6S4qlLSpGSnCSE+Gmjx+12Zc7I=",
            "transport": "udp",
            "direction": "internal"
          },
          "input": {
            "type": "udp"
          },
          "observer": {
            "ingress": {
              "zone": "VPN",
              "interface": {
                "name": "xfrm4"
              }
            },
            "product": "XG",
            "vendor": "Sophos",
            "serial_number": "1234567",
            "type": "firewall",
            "egress": {
              "zone": "LAN",
              "interface": {
                "name": "Port1"
              }
            }
          },
          "@timestamp": "2023-09-29T11:54:23.000+02:00",
          "ecs": {
            "version": "8.8.0"
          },
          "related": {
            "hosts": [
              "fwgate01.firewall.local"
            ],
            "ip": [
              "172.17.35.157",
              "172.16.34.11"
            ]
          },
          "sophos": {
            "xg": {
              "device_model": "XG230",
              "con_id": "675172385",
              "fw_rule_type": "USER",
              "ips_policy_id": "17",
              "fw_rule_section": "Local rule",
              "app_is_cloud": "FALSE",
              "device_name": "SFW",
              "log_type": "Firewall",
              "ether_type": "Unknown (0x0000)",
              "log_id": "010101600001",
              "log_component": "Firewall Rule",
              "fw_rule_name": "VPN-Wecker_LAN",
              "log_subtype": "Allowed",
              "dst_zone_type": "LAN",
              "hb_status": "No Heartbeat",
              "src_zone_type": "VPN",
              "con_event": "Start",
              "app_resolved_by": "Signature",
              "qualifier": "New",
              "log_version": "1"
            }
          },
          "data_stream": {
            "namespace": "default",
            "type": "logs",
            "dataset": "sophos.xg"
          },
          "@version": "1",
          "host": {
            "name": "fwgate01.firewal.local"
          },
          "event": {
            "agent_id_status": "auth_metadata_missing",
            "severity": 6,
            "ingested": "2023-09-29T09:54:23Z",
            "code": "00001",
            "timezone": "+02:00",
            "kind": "event",
            "module": "sophos",
            "action": "allowed",
            "category": [
              "network"
            ],
            "dataset": "sophos.xg",
            "outcome": "success"
          }
        },
        "_ingest": {
          "timestamp": "2023-09-29T09:55:01.098419772Z"
        }
      }
    }
  ]
}

elasticmachine commented 1 year ago

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

P1llus commented 1 year ago

@StefanSa I have a feeling this issue is still that it cannot send it to the correct pipeline.

Version 2.11 is fairly old and I believe the reason you are seeing the UTM packages parsed is because that version still had the old local processing for UTM, from 3.0 the UTM was completely rewritten as it was an old experimental integration.

So the reason it might work for you for UTM is because the local agent is still doing the processing there.

P1llus commented 1 year ago

It could be a variety of reasons, but I believe Logstash should be able to determine where it should go using only data_stream: true.

derelict commented 1 year ago

i have the exact same problem ... but i'm not sure if it's a securityonion configuration issue ... or the integration itself

StefanSa commented 1 year ago

Hi @derelict As a temporary solution i took filebeat last version. This works without problems.

derelict commented 1 year ago

Did you just replace the filebeat binary or the whole elasticagent ? Which Version is working for you ? in my case (just replacing the binary) does not work.

StefanSa commented 1 year ago

Hi @derelict I have installed a separate filebeat (latest version) instance, so not with the elastic agent. This can be on the so server itself or ideally, on a separate ingest server like mine.

/etc/filebeat/filebeat.yml

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["172.16.34.36:9200"]
  allow_older_versions: true

  # Protocol - either `http` (default) or `https`.
  protocol: "https"
  ssl.enabled: true
  ssl.verification_mode: none

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "so_elastic"
  password: "secret"

  indices:
    - index: "logs-%{[event.dataset]}-default"
      when.has_fields: ['event.dataset']

/etc/filebeat/modules.d/sophos.yml

# Module: sophos
# Docs: https://www.elastic.co/guide/en/beats/filebeat/8.10/filebeat-module-sophos.html

- module: sophos
  xg:
    enabled: true

    # Set which input to use between tcp, udp (default) or file.
    var.input: udp

    # The interface to listen to syslog traffic. Defaults to
    # localhost. Set to 0.0.0.0 to bind to all available interfaces.
    var.syslog_host: 0.0.0.0

    # The port to listen for syslog traffic. Defaults to 9004.
    var.syslog_port: 9005

    # firewall default hostname
    var.default_host_name: firewall.test.local

    # known firewalls
    var.known_devices:
      - serial_number: "12345678"
        hostname: "fwgate01.test.local"
      - serial_number: "87654321"
        hostname: "fwgate02.test.local"

  utm:
    enabled: true

    # Set which input to use between udp (default), tcp or file.
    var.input: tcp
    var.syslog_host: 0.0.0.0
    var.syslog_port: 9012

    # Set paths for the log files when file input is used.
    # var.paths:

    # Toggle output of non-ECS fields (default true).
    var.rsa_fields: true

    # Set custom timezone offset.
    # "local" (default) for system timezone.
    # "+02:00" for GMT+02:00
    # var.tz_offset: local

Before you start filebeat with systemctl, you have to transvert the pipelines to the so server. filebeat setup --pipelines Now the data should be transferred from the firewalls via filebeat to the so server.

derelict commented 1 year ago

Hi @derelict I have installed a separate filebeat (latest version) instance, so not with the elastic agent. This can be on the so server itself or ideally, on a separate ingest server like mine.

Ok. Cool. I will give it a try then. Thank you very much.

jamiehynds commented 9 months ago

Closing as the newer version of the Sophos integration (v3.8.1) should address this issue. @derelict @StefanSa if you're still having issues, please feel free to re-open and we can investigate.

elastic / integrations

[sophos][xg] Failed to parse field #8022