Closed BrianMer closed 2 months ago
philhagen
commented
2 months ago This is helpful - thanks! I can see this is being passed via a live source. Is this being shipped via syslog (UDP/5514 or TCP/5514), relp (TCP/5516), or filebeat (TCP/5504)?
BrianMer
commented
2 months ago I forgot to mention that this sample was send via syslog TCP/5514 (which was not open in the firewall, btw).
BrianMer
commented
2 months ago I also tested via Filebeat outputing to Logstash TCP/5044, although the fields that I mentioned are still not parsed, it seems like some are via Filebeat:
{
"_index": "logstash-2024.04",
"_id": "vp-AGY8Bz6W6pw6KB7vG",
"_version": 1,
"_score": 0,
"_source": {
"ecs": {
"version": "8.0.0"
},
"message": "Started dnf makecache.",
"@timestamp": "2024-04-26T08:21:40.000Z",
"sof-elk.processing_time": 0.00254822,
"host": {
"architecture": "x86_64",
"mac": [
"BC-24-11-12-C4-4C",
"BC-24-11-46-A4-61",
"BC-24-11-5D-2C-32",
"BC-24-11-A3-6B-1D"
],
"name": "relai-log-srv",
"hostname": "relai-log-srv",
"id": "d71e290da28048c0bc100945fc378362",
"containerized": false,
"os": {
"family": "redhat",
"kernel": "4.18.0-513.24.1.el8_9.x86_64",
"codename": "Green Obsidian",
"name": "Rocky Linux",
"type": "linux",
"platform": "rocky",
"version": "8.9 (Green Obsidian)"
},
"ip": [
"10.3.0.196",
"fe80::be24:11ff:fea3:6b1d",
"192.168.151.2",
"fe80::be24:11ff:fe12:c44c",
"192.168.152.1",
"fe80::be24:11ff:fe46:a461",
"192.168.149.2",
"fe80::be24:11ff:fe5d:2c32"
]
},
"event": {
"severity": 6,
"original": "Started dnf makecache."
},
"syslog": {
"facility": 3,
"priority": 30,
"severity_label": "Informational",
"facility_label": "system"
},
"log": {
"source": {
"address": "192.168.151.1:59392"
}
},
"@version": "1",
"hostname": "vm2-rocky8-8-sonde",
"tags": [
"process_archive",
"filebeat",
"beats_input_codec_plain_applied"
],
"input": {
"type": "syslog"
},
"agent": {
"id": "156e0c6e-7748-4b13-9ad9-dc0a4a2dd575",
"ephemeral_id": "4c718a07-0856-49dc-90e6-9503f73b7d54",
"type": "filebeat",
"name": "relai-LOG-SRV",
"version": "8.13.2"
},
"process": {
"program": "systemd",
"pid": 1
}
},
"fields": {
"agent.version.keyword": [
"8.13.2"
],
"host.architecture.keyword": [
"x86_64"
],
"syslog.severity_label.keyword": [
"Informational"
],
"host.name.keyword": [
"relai-log-srv"
],
"host.hostname": [
"relai-log-srv"
],
"process.pid": [
1
],
"host.mac": [
"BC-24-11-12-C4-4C",
"BC-24-11-46-A4-61",
"BC-24-11-5D-2C-32",
"BC-24-11-A3-6B-1D"
],
"hostname": [
"vm2-rocky8-8-sonde"
],
"ecs.version.keyword": [
"8.0.0"
],
"host.os.version": [
"8.9 (Green Obsidian)"
],
"host.os.name": [
"Rocky Linux"
],
"agent.name": [
"relai-LOG-SRV"
],
"host.id.keyword": [
"d71e290da28048c0bc100945fc378362"
],
"host.name": [
"relai-log-srv"
],
"host.os.version.keyword": [
"8.9 (Green Obsidian)"
],
"event.severity": [
6
],
"event.original": [
"Started dnf makecache."
],
"host.os.type": [
"linux"
],
"agent.id.keyword": [
"156e0c6e-7748-4b13-9ad9-dc0a4a2dd575"
],
"input.type": [
"syslog"
],
"tags": [
"process_archive",
"filebeat",
"beats_input_codec_plain_applied"
],
"host.architecture": [
"x86_64"
],
"process.program.keyword": [
"systemd"
],
"agent.id": [
"156e0c6e-7748-4b13-9ad9-dc0a4a2dd575"
],
"ecs.version": [
"8.0.0"
],
"host.containerized": [
false
],
"log.source.address": [
"192.168.151.1:59392"
],
"sof-elk.processing_time": [
0.00254822
],
"message.keyword": [
"Started dnf makecache."
],
"host.hostname.keyword": [
"relai-log-srv"
],
"agent.version": [
"8.13.2"
],
"host.os.family": [
"redhat"
],
"hostname.keyword": [
"vm2-rocky8-8-sonde"
],
"input.type.keyword": [
"syslog"
],
"process.program": [
"systemd"
],
"tags.keyword": [
"process_archive",
"filebeat",
"beats_input_codec_plain_applied"
],
"syslog.facility": [
3
],
"host.ip": [
"10.3.0.196",
"fe80::be24:11ff:fea3:6b1d",
"192.168.151.2",
"fe80::be24:11ff:fe12:c44c",
"192.168.152.1",
"fe80::be24:11ff:fe46:a461",
"192.168.149.2",
"fe80::be24:11ff:fe5d:2c32"
],
"agent.type": [
"filebeat"
],
"host.os.kernel.keyword": [
"4.18.0-513.24.1.el8_9.x86_64"
],
"host.os.kernel": [
"4.18.0-513.24.1.el8_9.x86_64"
],
"@version": [
"1"
],
"host.os.name.keyword": [
"Rocky Linux"
],
"host.id": [
"d71e290da28048c0bc100945fc378362"
],
"agent.type.keyword": [
"filebeat"
],
"agent.ephemeral_id.keyword": [
"4c718a07-0856-49dc-90e6-9503f73b7d54"
],
"host.os.codename.keyword": [
"Green Obsidian"
],
"host.mac.keyword": [
"BC-24-11-12-C4-4C",
"BC-24-11-46-A4-61",
"BC-24-11-5D-2C-32",
"BC-24-11-A3-6B-1D"
],
"agent.name.keyword": [
"relai-LOG-SRV"
],
"syslog.priority": [
30
],
"host.os.codename": [
"Green Obsidian"
],
"message": [
"Started dnf makecache."
],
"host.os.family.keyword": [
"redhat"
],
"@timestamp": [
"2024-04-26T08:21:40.000Z"
],
"host.os.type.keyword": [
"linux"
],
"syslog.severity_label": [
"Informational"
],
"host.os.platform": [
"rocky"
],
"host.os.platform.keyword": [
"rocky"
],
"event.original.keyword": [
"Started dnf makecache."
],
"agent.ephemeral_id": [
"4c718a07-0856-49dc-90e6-9503f73b7d54"
],
"syslog.facility_label": [
"system"
],
"log.source.address.keyword": [
"192.168.151.1:59392"
],
"syslog.facility_label.keyword": [
"system"
]
}
}Thanks - I'll test this via the syslog ingest pipeline. Depending on how filebeat is configured (codecs, etc), it can sometimes do some of the initial parsing on its own. Since that is highly variable and subject to admin preference, we don't test that particular approach but I'll see what I can do.
philhagen
commented
2 months ago ok! this should be ready for validation. if you can update with sudo sof-elk_update.sh, then observe the incoming message, it should be fixed. let me know!
BrianMer
commented
2 months ago OK this is fixed for Syslog (TCP/5514) (although there still some _grokparsefailure errors in tags):
{
"_index": "logstash-2024.04",
"_id": "YEseKY8BeRsBtd3RjC45",
"_version": 1,
"_score": 0,
"_ignored": [
"message.keyword",
"event.original.keyword"
],
"_source": {
"sof-elk.processing_time": 0.00335312,
"message": "{\"log.level\":\"info\",\"@timestamp\":\"2024-04-29T11:10:17.235+0200\",\"log.logger\":\"syslog\",\"log.origin\":{\"function\":\"github.com/elastic/beats/v7/filebeat/input/syslog.(*Input).Run\",\"file.name\":\"syslog/input.go\",\"file.line\":147},\"message\":\"Starting Syslog input\",\"service.name\":\"filebeat\",\"protocol\":\"tcp\",\"ecs.version\":\"1.6.0\"}\n",
"log": {
"syslog": {
"priority": 30,
"facility": {
"code": 3,
"name": "system"
},
"severity": {
"code": 6,
"name": "informational"
},
"procid": 1056,
"appname": "filebeat",
"hostname": "relai-log-srv"
}
},
"tags": [
"process_live",
"syslog",
"_grokparsefailure"
],
"host": {
"ip": "192.168.152.1",
"hostname": "relai-LOG-SRV"
},
"service": {
"type": "system"
},
"@version": "1",
"event": {
"original": "<30>Apr 29 11:10:17 relai-LOG-SRV filebeat[1056]: {\"log.level\":\"info\",\"@timestamp\":\"2024-04-29T11:10:17.235+0200\",\"log.logger\":\"syslog\",\"log.origin\":{\"function\":\"github.com/elastic/beats/v7/filebeat/input/syslog.(*Input).Run\",\"file.name\":\"syslog/input.go\",\"file.line\":147},\"message\":\"Starting Syslog input\",\"service.name\":\"filebeat\",\"protocol\":\"tcp\",\"ecs.version\":\"1.6.0\"}\n"
},
"path": "syslog from relai-LOG-SRV",
"type": "syslog",
"@timestamp": "2024-04-29T11:10:17.000Z",
"process": {
"pid": 1056,
"name": "filebeat"
}
},
"fields": {
"service.type.keyword": [
"system"
],
"log.syslog.hostname": [
"relai-log-srv"
],
"log.syslog.appname": [
"filebeat"
],
"tags.keyword": [
"process_live",
"syslog",
"_grokparsefailure"
],
"host.hostname": [
"relai-LOG-SRV"
],
"process.pid": [
1056
],
"type": [
"syslog"
],
"service.type": [
"system"
],
"host.ip": [
"192.168.152.1"
],
"path": [
"syslog from relai-LOG-SRV"
],
"log.syslog.facility.name": [
"system"
],
"type.keyword": [
"syslog"
],
"@version": [
"1"
],
"log.syslog.severity.name": [
"informational"
],
"log.syslog.priority": [
30
],
"log.syslog.appname.keyword": [
"filebeat"
],
"event.original": [
"<30>Apr 29 11:10:17 relai-LOG-SRV filebeat[1056]: {\"log.level\":\"info\",\"@timestamp\":\"2024-04-29T11:10:17.235+0200\",\"log.logger\":\"syslog\",\"log.origin\":{\"function\":\"github.com/elastic/beats/v7/filebeat/input/syslog.(*Input).Run\",\"file.name\":\"syslog/input.go\",\"file.line\":147},\"message\":\"Starting Syslog input\",\"service.name\":\"filebeat\",\"protocol\":\"tcp\",\"ecs.version\":\"1.6.0\"}\n"
],
"log.syslog.severity.code": [
6
],
"log.syslog.procid": [
1056
],
"message": [
"{\"log.level\":\"info\",\"@timestamp\":\"2024-04-29T11:10:17.235+0200\",\"log.logger\":\"syslog\",\"log.origin\":{\"function\":\"github.com/elastic/beats/v7/filebeat/input/syslog.(*Input).Run\",\"file.name\":\"syslog/input.go\",\"file.line\":147},\"message\":\"Starting Syslog input\",\"service.name\":\"filebeat\",\"protocol\":\"tcp\",\"ecs.version\":\"1.6.0\"}\n"
],
"tags": [
"process_live",
"syslog",
"_grokparsefailure"
],
"process.name": [
"filebeat"
],
"log.syslog.hostname.keyword": [
"relai-log-srv"
],
"@timestamp": [
"2024-04-29T11:10:17.000Z"
],
"process.name.keyword": [
"filebeat"
],
"log.syslog.facility.name.keyword": [
"system"
],
"sof-elk.processing_time": [
0.00335312
],
"host.hostname.keyword": [
"relai-LOG-SRV"
],
"log.syslog.severity.name.keyword": [
"informational"
],
"log.syslog.facility.code": [
3
]
},
"ignored_field_values": {
"event.original.keyword": [
"<30>Apr 29 11:10:17 relai-LOG-SRV filebeat[1056]: {\"log.level\":\"info\",\"@timestamp\":\"2024-04-29T11:10:17.235+0200\",\"log.logger\":\"syslog\",\"log.origin\":{\"function\":\"github.com/elastic/beats/v7/filebeat/input/syslog.(*Input).Run\",\"file.name\":\"syslog/input.go\",\"file.line\":147},\"message\":\"Starting Syslog input\",\"service.name\":\"filebeat\",\"protocol\":\"tcp\",\"ecs.version\":\"1.6.0\"}\n"
],
"message.keyword": [
"{\"log.level\":\"info\",\"@timestamp\":\"2024-04-29T11:10:17.235+0200\",\"log.logger\":\"syslog\",\"log.origin\":{\"function\":\"github.com/elastic/beats/v7/filebeat/input/syslog.(*Input).Run\",\"file.name\":\"syslog/input.go\",\"file.line\":147},\"message\":\"Starting Syslog input\",\"service.name\":\"filebeat\",\"protocol\":\"tcp\",\"ecs.version\":\"1.6.0\"}\n"
]
}
}But it seems like this is not the case with Filebeat (TCP/5044). The Filebeat's config that I use is just a relay, without any pre-processing or module. I just added that into the default config file:
filebeat.inputs:
- type: syslog
enabled: true
format: auto
protocol.tcp:
host: "192.168.151.2:2514"
output.logstash:
hosts: ["192.168.152.3:5044"]And here is the result:
{
"_index": "logstash-2024.04",
"_id": "_EsuKY8BeRsBtd3RIy6Q",
"_version": 1,
"_score": 0,
"_source": {
"sof-elk.processing_time": 0.00101399,
"input": {
"type": "syslog"
},
"message": "action 'action-1-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2102.0-15.el8 try https://www.rsyslog.com/e/2359 ]",
"log": {
"source": {
"address": "192.168.151.1:37548"
}
},
"tags": [
"process_archive",
"filebeat",
"beats_input_codec_plain_applied"
],
"host": {
"architecture": "x86_64",
"containerized": false,
"ip": [
"10.3.0.196",
"fe80::be24:11ff:fea3:6b1d",
"192.168.151.2",
"fe80::be24:11ff:fe12:c44c",
"192.168.152.1",
"fe80::be24:11ff:fe46:a461",
"192.168.149.2",
"fe80::be24:11ff:fe5d:2c32"
],
"name": "relai-log-srv",
"mac": [
"BC-24-11-12-C4-4C",
"BC-24-11-46-A4-61",
"BC-24-11-5D-2C-32",
"BC-24-11-A3-6B-1D"
],
"id": "d71e290da28048c0bc100945fc378362",
"hostname": "relai-log-srv",
"os": {
"family": "redhat",
"kernel": "4.18.0-513.24.1.el8_9.x86_64",
"codename": "Green Obsidian",
"name": "Rocky Linux",
"type": "linux",
"platform": "rocky",
"version": "8.9 (Green Obsidian)"
}
},
"@version": "1",
"event": {
"severity": 6,
"original": "action 'action-1-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2102.0-15.el8 try https://www.rsyslog.com/e/2359 ]"
},
"hostname": "collecteur-SRV-LOG",
"syslog": {
"priority": 46,
"facility": 5,
"facility_label": "syslogd",
"severity_label": "Informational"
},
"ecs": {
"version": "8.0.0"
},
"agent": {
"ephemeral_id": "5301d166-d97a-4844-92fe-4945a4bb44f1",
"id": "156e0c6e-7748-4b13-9ad9-dc0a4a2dd575",
"type": "filebeat",
"name": "relai-LOG-SRV",
"version": "8.13.2"
},
"@timestamp": "2024-04-29T09:27:14.000Z",
"process": {
"pid": 1045,
"program": "rsyslogd"
}
},
"fields": {
"agent.version.keyword": [
"8.13.2"
],
"host.architecture.keyword": [
"x86_64"
],
"syslog.severity_label.keyword": [
"Informational"
],
"host.name.keyword": [
"relai-log-srv"
],
"host.hostname": [
"relai-log-srv"
],
"process.pid": [
1045
],
"host.mac": [
"BC-24-11-12-C4-4C",
"BC-24-11-46-A4-61",
"BC-24-11-5D-2C-32",
"BC-24-11-A3-6B-1D"
],
"hostname": [
"collecteur-SRV-LOG"
],
"ecs.version.keyword": [
"8.0.0"
],
"host.os.version": [
"8.9 (Green Obsidian)"
],
"host.os.name": [
"Rocky Linux"
],
"agent.name": [
"relai-LOG-SRV"
],
"host.id.keyword": [
"d71e290da28048c0bc100945fc378362"
],
"host.name": [
"relai-log-srv"
],
"host.os.version.keyword": [
"8.9 (Green Obsidian)"
],
"event.severity": [
6
],
"event.original": [
"action 'action-1-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2102.0-15.el8 try https://www.rsyslog.com/e/2359 ]"
],
"host.os.type": [
"linux"
],
"agent.id.keyword": [
"156e0c6e-7748-4b13-9ad9-dc0a4a2dd575"
],
"input.type": [
"syslog"
],
"tags": [
"process_archive",
"filebeat",
"beats_input_codec_plain_applied"
],
"host.architecture": [
"x86_64"
],
"process.program.keyword": [
"rsyslogd"
],
"agent.id": [
"156e0c6e-7748-4b13-9ad9-dc0a4a2dd575"
],
"ecs.version": [
"8.0.0"
],
"host.containerized": [
false
],
"log.source.address": [
"192.168.151.1:37548"
],
"sof-elk.processing_time": [
0.00101399
],
"message.keyword": [
"action 'action-1-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2102.0-15.el8 try https://www.rsyslog.com/e/2359 ]"
],
"host.hostname.keyword": [
"relai-log-srv"
],
"agent.version": [
"8.13.2"
],
"host.os.family": [
"redhat"
],
"hostname.keyword": [
"collecteur-SRV-LOG"
],
"input.type.keyword": [
"syslog"
],
"process.program": [
"rsyslogd"
],
"tags.keyword": [
"process_archive",
"filebeat",
"beats_input_codec_plain_applied"
],
"syslog.facility": [
5
],
"host.ip": [
"10.3.0.196",
"fe80::be24:11ff:fea3:6b1d",
"192.168.151.2",
"fe80::be24:11ff:fe12:c44c",
"192.168.152.1",
"fe80::be24:11ff:fe46:a461",
"192.168.149.2",
"fe80::be24:11ff:fe5d:2c32"
],
"agent.type": [
"filebeat"
],
"host.os.kernel.keyword": [
"4.18.0-513.24.1.el8_9.x86_64"
],
"host.os.kernel": [
"4.18.0-513.24.1.el8_9.x86_64"
],
"@version": [
"1"
],
"host.os.name.keyword": [
"Rocky Linux"
],
"host.id": [
"d71e290da28048c0bc100945fc378362"
],
"agent.type.keyword": [
"filebeat"
],
"agent.ephemeral_id.keyword": [
"5301d166-d97a-4844-92fe-4945a4bb44f1"
],
"host.os.codename.keyword": [
"Green Obsidian"
],
"host.mac.keyword": [
"BC-24-11-12-C4-4C",
"BC-24-11-46-A4-61",
"BC-24-11-5D-2C-32",
"BC-24-11-A3-6B-1D"
],
"agent.name.keyword": [
"relai-LOG-SRV"
],
"syslog.priority": [
46
],
"host.os.codename": [
"Green Obsidian"
],
"message": [
"action 'action-1-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2102.0-15.el8 try https://www.rsyslog.com/e/2359 ]"
],
"host.os.family.keyword": [
"redhat"
],
"@timestamp": [
"2024-04-29T09:27:14.000Z"
],
"host.os.type.keyword": [
"linux"
],
"syslog.severity_label": [
"Informational"
],
"host.os.platform": [
"rocky"
],
"host.os.platform.keyword": [
"rocky"
],
"event.original.keyword": [
"action 'action-1-builtin:omfwd' resumed (module 'builtin:omfwd') [v8.2102.0-15.el8 try https://www.rsyslog.com/e/2359 ]"
],
"agent.ephemeral_id": [
"5301d166-d97a-4844-92fe-4945a4bb44f1"
],
"syslog.facility_label": [
"syslogd"
],
"log.source.address.keyword": [
"192.168.151.1:37548"
],
"syslog.facility_label.keyword": [
"syslogd"
]
}
}interesting. filebeat is definitely doing some parsing of its own, though. there is no type field, and the series of syslog.* fields is not present in the native syslog-received record so I'm not sure where those are being assigned. The lack of the type field is particularly limiting, since that's one of the main conditions checked for parsing in the our configuration.
I'll have to dig in more with this and figure out what is going on - it will probably result in documentation on the supported filebeat shipper configurations. if the native syslog shipper's records are properly parsed, I'm inclined to close this issue - please let me know if you think this is in a state to do that. I'll create a new one for future attention to shipping the syslog messages from a remote filebeat instance in just a moment.
BrianMer
commented
2 months ago OK Phil, I agree with closing this issue. As long as the native Syslog is working, this is fine.
Thanks!
philhagen
commented
2 months ago thanks for confirming! I'll reassess the filebeat part with the above mentioned issue.
Hi Phil,
I'm currently testing the VM SOF-ELK - ECS with some Syslog architecture, and it seems like this is not parsing properly.
e.g. : The hostname, PID, and process name is present, but not in the right field (
host.hostnameinstead oflog.syslog.hostname.keyword,process.pidinstead oflog.syslog.procid, andprocess.nameinstead oflog.syslog.appname.keyword).There are also
_grokparsefailurethat I can not find the source.Here is a logstash document as an example :
Hope this is clear enough!