Remove the facility number on TCP

[paminhoff]

We are having issues parsing events been sent via TCP because the scripts adds facility <30> to every event. How do we remove these? Can't find a configuration item.

[paminhoff]

Actually this is wrong, the issue seems to be there is no CR or NULL value between entries which is causing logs that are cued up to be sent as one large blob. This happens for CEF, JSON and KEY_VALUE.

This does not happen with CEF on version 2.0.1. JSON on this version is not usable.

[paminhoff]

Figured out the problem, because the new message format functions (CEF, JSON, KEYVALUE) in the siem.py are missing a the new line character, they are getting sent in as one blob, and the parser used by the collector (Rapid7) is unable to identify any breaks per message, the parser in just processed the entire message as one blob unable to identify any fields and could not get any useful data, after reviewing the code for version V2.0.1 and 2.1.0 I found that the dev removed the + u"\n" at the end of each message in V2.1.0. To resolve this I modified line 91 and added line 92 to add the *+ u"\n" back to the _write_jsonformats function and now my SIEM ingestion and parsing works properly and I can query logs. My sample

>82 def write_json_format(results):
>83     """Write JSON format data.
>84     Arguments:
>85         results {list}: data
>86     """
>87     for i in results:
>88         i = remove_null_values(i)
>89         update_cef_keys(i)
>90         name_mapping.update_fields(log, i)
>91        d = json.dumps(i, ensure_ascii=False).strip()
>92         SIEM_LOGGER.info(d + u"\n")
> 

Note I only modified JSON, because Rapid7 IDR parses JSON formats better than CEF, if your collector or SIEM ingests CEF or KEYVALUE you should be able to modify the _write_keyvalueformat or _write_cefformat and achieve the same results.

[andrewthad]

The issue author marked this as closed since they modified this application and started using a vendored version of it. However, my team just ran into this same problem. So, this hasn't ever been corrected upstream. Sending json over a tcp socket is typically done with newline-delimited json:

{"id":1}
{"id":2}

The sophos integration does this instead:

<30>{"id":1}<30>{"id":2}

This format is not commonly supported by log ingest applications.

[wildborn]

Hello @paminhoff can you please help me to resolve as per your above comments it wasn't helpful. I'm still receiving the logs along with the facility number.

Please let me know what else is required to be done for achieving log on syslog server without facility number.

[paminhoff]

Hello @paminhoff can you please help me to resolve as per your above comments it wasn't helpful. I'm still receiving the logs along with the facility number.

Please let me know what else is required to be done for achieving log on syslog server without facility number.

I just added this, nothing else:

91 d = json.dumps(i, ensure_ascii=False).strip() 92 SIEM_LOGGER.info(d + u"\n")

maybe you are running a different version of the python script than I had a12 months ago.

[wildborn]

Thanks for your reply @paminhoff but I've been using the latest version and made same changes in these very lines which results me in multiple errors.

Would request you to share your running script and for transparency please share the running Sophos folder so that I can use the same for my environment too.

[paminhoff]

As I said, I only had to update the 2 lines and it worked for me , if this did not work for you not sure how to help, if my updates did not work for you, start new because the script I have as base is probably a different version, not sure which one I used, I am not the maintainer of this repo. You can also open a new issue with the maintainers.

Last option, now that ChatGPT v4 is available, and you have a ChatGPT 4 (not 3.5) account feed it the original script and ask it to solve for your issue it will get you really close if not solve it.

Regards, Paul Hoff

On Sep 24, 2023, at 4:26 AM, Lakshay @.***> wrote:

that

[wildborn]

Totally understandable for the repos and versions but if you can share me the script which you are using would help me solve it fully. As I've raised a new case 2 weeks ago but the maintainers reply often so please share me your running script.

Tried GPT too which leads to multiple sophos bad repos only.

I appreaciate for your response and time and would request if its feasible to share your working script.