Support for TCP Syslog

[Alan-Carr]

I believe that this application only supports UDP Syslog it would be useful if it also supported TCP

[trparky]

I've implemented preliminary and very experimental code to receive syslog data via TCP. You're welcome to try it but be warned, it's still very experimental and may be buggy. I have no way to test the new code since everything I have uses UDP for syslog data.

Below is a compiled version that supports TCP. Note, UDP is still the preferred method for receiving syslog data.

[trparky]

Here's an updated ZIP file, I forgot to start the server in the prior compiled version. Free SysLog.zip

[trparky]

I setup a rudimentary test environment in which I simulated sending data to the TCP server and fixed a lot of bugs. I have a new binary for you to test. However, you will need to enable the TCP server in the Settings menu to turn the TCP server on. I'm sure you will find the item in the Settings menu, it's rather self-explanatory. Free SysLog.zip

[Alan-Carr]

Hi

I have carried out some basic tests with your latest Version 3.1 Build 3 (Debug Build)

I noticed initially when I hadn't set enable syslog server that a TCP port was open from your application to a 172.x address maybe this is a result of some of your debug code ?

I am getting syslog data over TCP into your application however the data we are monitoring contains structured data as specified in rfc 5424

[Alan-Carr]

One additional thing I have noticed it seem difficult to drag the application to an additional monitor screen

[trparky]

this is a result of some of your debug code ?

I am getting syslog data over TCP into your application however the data we are monitoring contains structured data as specified in rfc 5424

Yeah, I fixed a bug regarding that. See... https://github.com/trparky/Free-SysLog/commit/8c1c2f7b396fdfe6ef4fb7b3c3593c97701e99dd

[trparky]

One additional thing I have noticed it seem difficult to drag the application to an additional monitor screen

Yes, I've noticed that as well. I have a fix in place.

[Alan-Carr]

Thanks for the update, seems like great progress

Do you think you maybe able to incorporate the structured data ?

[trparky]

Thanks for the update, seems like great progress

Do you think you maybe able to incorporate the structured data ?

Excuse me? I'm not sure I understand.

Here's a new program binary to test. Free SysLog.zip

[Alan-Carr]

With ref to the structured data see https://datatracker.ietf.org/doc/html/rfc5424#section-6 and https://datatracker.ietf.org/doc/html/rfc5424#section-6.5

[trparky]

OK, it took a lot of time to do it but it's done. I think. Free SysLog.zip

[Alan-Carr]

Hi

Thanks for the update and its great to see such progress. The data we are monitoring contains structured data within the Log message incl for example a timestamp which amongst others I'm expecting to see in separate columns.

We are also seeing on some occasions where multiple messages are shown in a single line eg the Log column shows more than one message.

Copying log text doesn't appear to work as expected

[trparky]

We are also seeing on some occasions where multiple messages are shown in a single line eg the Log column shows more than one message.

I've noticed that as well with some of my networking equipment. As for why, I'm not sure. All the code is doing is literally taking in what it's being fed, parsing it, displaying it; nothing more, nothing less. What I'm saying is that if there are duplicate log entries, it's the system that's sending it that's duplicating the data.

Copying log text doesn't appear to work as expected

OK, I'll work on that bug later. Right now, we're working with alpha code, not even beta level code here. In other words, expect bugs.

[Alan-Carr]

Its not duplicate messages its multiple different messages on the same line. Bearing in mind that the log message content also contains timestamp with milliseconds resolution and additional data which should be in columns

[trparky]

Are you saying that the line breaks aren't being handled properly? As for timestamps, the program is generating them on the fly as the log comes in. Now, I could make additional changes to use the timestamp that the log comes in with instead of generating them in the program itself.

[Alan-Carr]

I think you should keep your timestamp and show the timestamp that is part of the additional structured data in the log messages along with an other additional columns.

[Alan-Carr]

keep yours as received time

[Alan-Carr]

you might get multiple separate messages at the same timestamp second

[trparky]

I'll continue to work on this on the weekend when I have more free time to do this.

[trparky]

I have, however, fixed the clipboard functionality since that was easy to fix. Free SysLog.zip

[trparky]

Alright, I have a lot of changes in store for you. Hopefully they live up to your expectations. Free SysLog.zip

[Alan-Carr]

Had a quick look data below shows in log field -: Object reference not set to an instance of an object. -- at Free_SysLog.SyslogParser.SyslogParser.ProcessIncomingLog(String strLogText, String strSourceIP) in C:\Users\trpar\OneDrive\My Visual Studio Projects\Projects\Free SysLog\Free SysLog\Support Code\Namespace Code\Syslog Parser.vb:line 176

[trparky]

I'll look into it when I get back in front of my computer.

[trparky]

I put in some additional checks to hopefully solve that issue.

Free SysLog.zip

[trparky]

OK, I was able to reproduce the bug that you mentioned on my system but I'm still trying to track down why the bug is happening. I put some additional debug code into the program so as to capture data so hopefully the next time it happens, I'll have additional data to work with.

[Alan-Carr]

Using your latest -:

Object reference not set to an instance of an object. -- at Free_SysLog.SyslogParser.SyslogParser.ConvertLineFeeds(String strInput) in C:\Users\trpar\OneDrive\My Visual Studio Projects\Projects\Free SysLog\Free SysLog\Support Code\Namespace Code\Syslog Parser.vb:line 122 at Free_SysLog.SyslogParser.SyslogParser.ProcessIncomingLog(String strRawLogText, String strSourceIP) in C:\Users\trpar\OneDrive\My Visual Studio Projects\Projects\Free SysLog\Free SysLog\Support Code\Namespace Code\Syslog Parser.vb:line 183

[trparky]

I refactored a lot of code and quite possibly fixed the bug in the process. I think. Free SysLog.zip

[trparky]

Nevermind the previous file, here's another; I refactored more code for easier maintenance. Free SysLog.zip

[trparky]

AH HA! Found it! I found the bug!

The bug has to do with how my regexp pattern used to parse syslog entries fails to parse logs if the remote app (or binary) that sent the log contains a space. I've tweaked the pattern to fix it. Free SysLog.zip

[Alan-Carr]

Using your latest, I'm now getting a column for the timestamp sent from the syslog device.

I'm seeing the syslog data in the log field however I'm still seeing vb error messages

Object reference not set to an instance of an object. -- at Free_SysLog.SyslogParser.SyslogParser.ConvertLineFeeds(String strInput) in C:\Users\trpar\OneDrive\My Visual Studio Projects\Projects\Free SysLog\Free SysLog\Support Code\Namespace Code\Syslog Parser.vb:line 125 at Free_SysLog.SyslogParser.SyslogParser.ProcessIncomingLog(String strRawLogText, String strSourceIP) in C:\Users\trpar\OneDrive\My Visual Studio Projects\Projects\Free SysLog\Free SysLog\Support Code\Namespace Code\Syslog Parser.vb:line 175

[trparky]

Damn, and here I thought I finally solved the issue. I guess I have more looking into this.

[trparky]

I have to wonder if it has something to do with multiline log entries.

[trparky]

Alright, I added some additional tweaks to the code. Free SysLog.zip

[trparky]

More tweaks to the RegEx pattern. God I hate RegEx! Free SysLog.zip

[trparky]

I'm closing this issue on account of the fact that I've not heard from you in over three days. I'm going to say that you're no longer interested in this program anymore. Feel free to re-open the issue or create a new issue if problems persist.

[Alan-Carr]

@trparky I am still very much interested in your application and will test the latest

[Alan-Carr]

@trparky I wasn't sure which version to test, I took the latest from GitHub

Unknown timestamp format. -- at Free_SysLog.SyslogParser.SyslogParser.ParseTimestamp(String timestamp) in C:\Users\trpar\OneDrive\My Visual Studio Projects\Projects\Free SysLog\Free SysLog\Support Code\Namespace Code\Syslog Parser.vb:line 119 at Free_SysLog.SyslogParser.SyslogParser.AddToLogList(String strTimeStampFromServer, String strSourceIP, String strHostname, String strRemoteProcess, String strLogText, Boolean boolIgnored, Boolean boolAlerted, ValueTuple`2 priority, String strRawLogText) in C:\Users\trpar\OneDrive\My Visual Studio Projects\Projects\Free SysLog\Free SysLog\Support Code\Namespace Code\Syslog Parser.vb:line 224 at Free_SysLog.SyslogParser.SyslogParser.ProcessIncomingLog(String strRawLogText, String strSourceIP) in C:\Users\trpar\OneDrive\My Visual Studio Projects\Projects\Free SysLog\Free SysLog\Support Code\Namespace Code\Syslog Parser.vb:line 193

[Alan-Carr]

@trparky Screen shot showing server time -:

Time from log is in Milliseconds and it also seems that there is an hour offset from what is displayed compared with incoming

2024-09-16T09:58:03.290Z

[Alan-Carr]

@trparky We get multiple messages in a single log.

From what I understand the "1269" below shows the length of the message.

1269 <134>1 2024-09-16T09:49:01.360Z

[trparky]

OK, can you give me an example of a full log entry so that I can find out how the program parses the incoming log?

[trparky]

We get multiple messages in a single log.

That's weird. Because if I understand the syslog standard correctly, each log entry should be stand on its own. If there are multiple events, there should be multiple log entries.

[trparky]

Free SysLog.zip OK, I included changes to the ParseTimestamp() function as per commit 98220d4.

[Alan-Carr]

With your latest -: Unknown timestamp format. -- at Free_SysLog.SyslogParser.SyslogParser.ParseTimestamp(String timestamp) in C:\Users\trpar\OneDrive\My Visual Studio Projects\Projects\Free SysLog\Free SysLog\Support Code\Namespace Code\Syslog Parser.vb:line 137 at Free_SysLog.SyslogParser.SyslogParser.AddToLogList(String strTimeStampFromServer, String strSourceIP, String strHostname, String strRemoteProcess, String strLogText, Boolean boolIgnored, Boolean boolAlerted, ValueTuple`2 priority, String strRawLogText) in C:\Users\trpar\OneDrive\My Visual Studio Projects\Projects\Free SysLog\Free SysLog\Support Code\Namespace Code\Syslog Parser.vb:line 239 at Free_SysLog.SyslogParser.SyslogParser.ProcessIncomingLog(String strRawLogText, String strSourceIP) in C:\Users\trpar\OneDrive\My Visual Studio Projects\Projects\Free SysLog\Free SysLog\Support Code\Namespace Code\Syslog Parser.vb:line 208

I note that the Server Timestamp shows only secs not msecs

[trparky]

I threw the timestamp "2024-09-16T09:58:03.290Z" through my parser function and it returned 9/16/2024 9:58:03 AM. So, that's good. Right?

[trparky]

If there's another timestamp that's failing, I need an example of it to test with.

[Alan-Carr]

Regarding your comment about the timestamp

"2024-09-16T09:58:03.290Z" through my parser function and it returned 9/16/2024 9:58:03 AM. So, that's good. Right?

The time is 09e:58.03.290 eg its in HH:MM:ss.SSS eg The time stamp ends in 290 Milliseconds eg 03 seconds and 290 Milliseconds

[trparky]

Regarding your comment about the timestamp

"2024-09-16T09:58:03.290Z" through my parser function and it returned 9/16/2024 9:58:03 AM. So, that's good. Right?

The time is 09e:58.03.290 eg its in HH:MM:ss.SSS eg The time stamp ends in 290 Milliseconds eg 03 seconds and 290 Milliseconds

OK, so again... Is the function parsing it is correctly?

[trparky]

I just added some additional error handling of timestamps as per commit https://github.com/trparky/Free-SysLog/commit/9d104ce1c2fabd6732c967a2b8cadc872a871d52. Free SysLog.zip

[Alan-Carr]

With regards to the screenshot it seems that we are getting an extra message in the log "unable to parse timestamp" which is associated with the previous message.

I note that the server time which comes from the message is only showing HH MM SS not ms

Furthermore I think the server time should only be seen in the server time column and no longer in the message but guess its no big deal if in both however the server time column needs to show the ms.

Bearing in mind you can get multiple separate messages on the same ms.

[trparky]

Oh no, that helps! I now have something that I can use to develop a parser for the timestamp.