Open kittawei1123 opened 5 years ago
I wish output can be :
sq_num:1
timestamp: Mon Aug 12 01:19:18 2019
message: Rogue AP: a4:d9:31:55:dc:ac detected on Base Radio MAC: f4:db:e6:96:ff:a0 Interface no: 1(802.11a) Channel: 64 RSSI: -71 SNR: 21 Classification: unclassified, State: Alert, RuleClassified : N, Severity Score: 0, RuleName: N.A. ,Classified AP MAC: 00:00:00:00:00:00 ,Classified RSSI: 0
I use this for regex test, it's ok, but failed to parse it when apply it in kinesis agent..
Tailer Progress: Tailer has parsed 0 records (624 bytes), transformed 0 records, skipped 0 records, and has successfully sent 0 records to destination.
^(\\d+)\\s{2,}(\\w{3} \\w{3} \\d+ \\d{2}:\\d{2}:\\d{2} \\d{4})\\s{2,}(.*)
If you turn on debugging, log4j, you should see if checkpoints are being reused. Maybe not what you intended.
Issue 118: "Initialization logs can be found in /tmp/aws-kinesis-agent.20190712081053.initlog”. You should have a similar file to review ….
Will Martin
On Aug 12, 2019, at 10:13 PM, kittawei1123 notifications@github.com wrote:
I use this for regex test, it's ok, but failed to apply it in kinesis agent.. ^(\d+)\s{2,}(\w{3} \w{3} \d+ \d{2}:\d{2}:\d{2} \d{4})\s{2,}(.*)
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/awslabs/amazon-kinesis-agent/issues/180?email_source=notifications&email_token=ALCZMGBLXAB7JJZOEDG3WN3QEIKDVA5CNFSM4IK7NQAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4EKTWQ#issuecomment-520661466, or mute the thread https://github.com/notifications/unsubscribe-auth/ALCZMGB4Y2D2ULCAT6UMUA3QEIKDVANCNFSM4IK7NQAA.
2019-08-13 04:11:03.070+0000 elk-l1 (FileTailer[kinesis:ec2_log:/var/log/wlc.log]) com.amazon.kinesis.streaming.agent.processing.processors.LogToJSONDataConverter [DEBUG] Getting exception while parsing record: [0 Mon Aug 12 01:19:18 2019 Rogue AP: a4:d9:31:55:dc:ac detected on Base Radio MAC: f4:db:e6:96:ec:e0 Interface no: 1(802.11a) Channel: 44 RSSI: -85 SNR: 10 Classification: unclassified, State: Alert, RuleClassified : N, Severity Score: 0, RuleName: N.A. ,Classified AP MAC: 00:00:00:00:00:00 ,Classified RSSI: 0], record will be skipped com.amazon.kinesis.streaming.agent.processing.exceptions.LogParsingException: Invalid log entry given the entry pattern at com.amazon.kinesis.streaming.agent.processing.parsers.SysLogParser.parseLogRecord(Unknown Source) at com.amazon.kinesis.streaming.agent.processing.processors.LogToJSONDataConverter.convert(Unknown Source) at com.amazon.kinesis.streaming.agent.processing.processors.AgentDataConverterChain.convert(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.AbstractParser.convertData(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.AbstractParser.buildRecord(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.AbstractParser.readRecordFromCurrentBuffer(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.AbstractParser.readRecord(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.processRecordsInCurrentFile(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.processRecords(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.runOnce(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.run(Unknown Source) at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60) at com.google.common.util.concurrent.Callables$3.run(Callables.java:95) at java.lang.Thread.run(Thread.java:748) 2019-08-13 04:11:03.070+0000 elk-l1 (FileTailer[kinesis:ec2_log:/var/log/wlc.log]) com.amazon.kinesis.streaming.agent.tailing.KinesisParser [WARN] 1 record parsed but skipped for processing and delivering
Ok. Your input record has 1 not a minimum of 2
Using a regex construction tool is not considered the best approach to learning regex, but this one will let you break it down, a group at a time….
Good luck. Stick with PCRE, its closest to Java’s regex.
https://regex101.com/ https://regex101.com/. Do not put PII or site identifying information into the tool, right? That includes your MAC address
Regards
Will Martin
On Aug 13, 2019, at 12:12 AM, kittawei1123 notifications@github.com wrote:
2019-08-13 04:11:03.070+0000 elk-l1 (FileTailer[kinesis:ec2_log:/var/log/wlc.log]) com.amazon.kinesis.streaming.agent.processing.processors.LogToJSONDataConverter [DEBUG] Getting exception while parsing record: [0 Mon Aug 12 01:19:18 2019 Rogue AP: a4:d9:31:55:dc:ac detected on Base Radio MAC: f4:db:e6:96:ec:e0 Interface no: 1(802.11a) Channel: 44 RSSI: -85 SNR: 10 Classification: unclassified, State: Alert, RuleClassified : N, Severity Score: 0, RuleName: N.A. ,Classified AP MAC: 00:00:00:00:00:00 ,Classified RSSI: 0], record will be skipped com.amazon.kinesis.streaming.agent.processing.exceptions.LogParsingException: Invalid log entry given the entry pattern at com.amazon.kinesis.streaming.agent.processing.parsers.SysLogParser.parseLogRecord(Unknown Source) at com.amazon.kinesis.streaming.agent.processing.processors.LogToJSONDataConverter.convert(Unknown Source) at com.amazon.kinesis.streaming.agent.processing.processors.AgentDataConverterChain.convert(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.AbstractParser.convertData(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.AbstractParser.buildRecord(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.AbstractParser.readRecordFromCurrentBuffer(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.AbstractParser.readRecord(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.processRecordsInCurrentFile(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.processRecords(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.runOnce(Unknown Source) at com.amazon.kinesis.streaming.agent.tailing.FileTailer.run(Unknown Source) at com.google.common.util.concurrent.AbstractExecutionThreadService$1$2.run(AbstractExecutionThreadService.java:60) at com.google.common.util.concurrent.Callables$3.run(Callables.java:95) at java.lang.Thread.run(Thread.java:748) 2019-08-13 04:11:03.070+0000 elk-l1 (FileTailer[kinesis:ec2_log:/var/log/wlc.log]) com.amazon.kinesis.streaming.agent.tailing.KinesisParser [WARN] 1 record parsed but skipped for processing and delivering
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/awslabs/amazon-kinesis-agent/issues/180?email_source=notifications&email_token=ALCZMGDPA42O6YMU6STBKOTQEIYBFA5CNFSM4IK7NQAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4EPQXY#issuecomment-520681567, or mute the thread https://github.com/notifications/unsubscribe-auth/ALCZMGELEJLJIDDVJ3BH4ZDQEIYBFANCNFSM4IK7NQAA.
thanks Will Martin, I did use the regx101 , and it's correct。
"matchPattern": "^(\\d+)\\s{1,}(\\w{3} \\w{3} \\d+ \\d{2}:\\d{2}:\\d{2} \\d{4})\\s{2,}(.*)",
"customFieldNames": [ "seqnum", "timestamp", "message" ]
fixed, sorry it's not space, it's tab "matchPattern": "^(\d+)\t{1,}(\w{3} \w{3} \d+ \d{2}:\d{2}:\d{2} \d{4})\t{1,}(.*)", thanks Martin, you're the BEST.
my json file is ok, kinesis can start up successfully, but write my log writing to the file , it will report error: com.amazon.kinesis.streaming.agent.Agent [ERROR] FATAL: Thread FileTailer[kinesis:ec2_log:/var/log/wlc.log] threw an unrecoverable error. Aborting application java.lang.IllegalStateException here is my log example: 1 Mon Aug 12 01:19:18 2019 Rogue AP: a4:d9:31:55:dc:ac detected on Base Radio MAC: f4:db:e6:96:ff:a0 Interface no: 1(802.11a) Channel: 64 RSSI: -71 SNR: 21 Classification: unclassified, State: Alert, RuleClassified : N, Severity Score: 0, RuleName: N.A. ,Classified AP MAC: 00:00:00:00:00:00 ,Classified RSSI: 0 the problem is how to write matchPattern correctly, sorry, it's not regex, I am confused, and don't know how to match a bunch of strings with my keyword "message". thanks.