[Feature Request] <Can you add logging to JSON format, please?>

[rusbomber]

Hi! Thanks for this great project! Can you add JSON to log formats? It can be useful for sending logs to SIEM/Elasticsearch etc for machine parsing, while still beeng human readable.

Using Elastic Common Schema and/or Common Event Format fields' names when possible would be great.

For example: {"ECS.Base.Timestamp":"2023-05-23T13:06:01.704054+03:00", "ECS.Host.Hostname":"firewall", "ECS.Process.Name":"opensnitch", "ECS.Event.Action":"always-deny-incoming-avahi-daemon", "CEFX.DeviceInboundInterface":"eth1", "CEFX.DeviceOutboundInterface":"", "ECS.Source.MAC":"01:00:5e:00:00:fb", "ECS.Destination.MAC":"b6:0b:e5:ed:e3:3c", "MAC_PROTO":"08:00", "ECS.Source.Address":"192.168.3.195", "ECS.Destination.Address":"224.0.0.251", "LEN":"145", "TOS":"0x00", "PREC":"0x00", "TTL":"255", "ECS.Event.ID":"14262", "ip_flags":"", "ECS.Network.Transport":"UDP", "ECS.Source.Port":"5353", "ECS.Destination.Port":"5353", "UDP-LEN":"125"}

Summary: Please, add logging in JSON format

[gustavo-iniguez-goya]

Hi @rusbomber ,

I've played a little bit with this, and with logstash + input plugin udp I can send the events in json format just as we send them to the GUI:

    udp {
        port => 3333
        codec => json {
            target => "[document]"
        }
    }

You can then use the document to filter by fields of the event.

I don't know if this is too bizarre for ELK experienced users/admins, but could it be a start?

[gustavo-iniguez-goya]

Added initial support. Adding new formats should be easy.

[rusbomber]

cat ./default-config.json

{
    "Server":
    {
        "Address":"unix:///tmp/osui.sock",
        "LogFile":"/var/log/opensnitchd.log",
        "Loggers": [
            {
                "Name": "syslog",
                "Server": "127.0.0.1:514",
                "Protocol": "udp",
                "Format": "json",
                "Tag": "opensnitchd"
            }
        ]
    },
    "DefaultAction": "allow",
    "DefaultDuration": "once",
    "InterceptUnknown": false,
    "ProcMonitorMethod": "proc",
    "LogLevel": 2,
    "Firewall": "iptables",
    "Stats": {
        "MaxEvents": 150,
        "MaxStats": 25,
        "Workers": 6
    }
}

then restart opensnitchd, but it fails: systemctl status opensnitchd ● opensnitchd.service - Application firewall OpenSnitch Loaded: loaded (/etc/systemd/system/opensnitchd.service; enabled; vendor preset: enabled) Active: activating (auto-restart) (Result: exit-code) since Mon 2023-05-29 17:50:53 MSK; 25s ago Docs: https://github.com/evilsocket/opensnitch/wiki Process: 20979 ExecStartPre=/bin/mkdir -p /etc/opensnitchd/rules (code=exited, status=0/SUCCESS) Process: 20980 ExecStart=/usr/local/bin/opensnitchd -rules-path /etc/opensnitchd/rules (code=exited, status=2) Main PID: 20980 (code=exited, status=2)

may 29 17:50:53 devel systemd[1]: opensnitchd.service: Main process exited, code=exited, status=2/INVALIDARGUMENT may 29 17:50:53 devel systemd[1]: opensnitchd.service: Failed with result 'exit-code'

when I run /usr/local/bin/opensnitchd -rules-path /etc/opensnitchd/rules then [2023-05-29 14:53:11] IMP Starting opensnitch-daemon v1.6.0 [2023-05-29 14:53:11] INF Loading rules from /etc/opensnitchd/rules ... OK: libnetfiler_queue supports nfq_get_uid OK: libnetfiler_queue supports nfq_get_uid found /lib/x86_64-linux-gnu/libc.so.6 panic: runtime error: index out of range [1] with length 1

[gustavo-iniguez-goya]

Instead of syslog use "remote_syslog" and use "rfc5424" as format:

    "Server": {
        "Address": "unix:///tmp/osui.sock",
        "LogFile": "/var/log/opensnitchd.log",
        "Loggers": [{
            "Name": "remote_syslog",
            "Server": "127.0.0.1:514",
            "Protocol": "udp",
            "Format": "rfc5424",
            "Tag": "opensnitchd"
        }]

I havent't tested sending logs to syslog in json format.

What I've tested is sending logs directly to logstash:

input {
    tcp {
        port => 3333
        codec => json_lines {
            target => "[document]"
        }
}

with this config:

        "Loggers": [{
            "Name": "remote_syslog",
            "Server": "127.0.0.1:3333",
            "Protocol": "tcp",
            "Format": "json",
            "Tag": "opensnitchd"
        }]

[rusbomber]

Remote_syslog works:

        "Loggers": [
            {
                "Name": "remote_syslog",
                "Server": "192.168.2.129:49000",
                "Protocol": "tcp",
                "Workers": 5,
                "Format": "json",
                "Tag": "opensnitch"
            }
        ]

I get in my SIEM something like

<29>1 2023-05-29T18:40:49+03:00 devel opensnitch 25520 TCPOUT - [SRC="192.168.122.135" SPT="34706" DST="10.63.2.129" DSTHOST="" DPT="443" PROTO="tcp" PID="13541" UID="1000" PATH="/usr/lib/firefox/firefox" CMDLINE="[/usr/lib/firefox/firefox]" CWD="/home/admini" ARG1="allow" ARG2="allow-until-restart-simple-usrlibfirefoxfirefox"] Local syslog: ``` "Loggers": [ { "Name": "syslog", "Server": "127.0.0.1:514", "Protocol": "udp", "Workers": 5, "Format": "json", "Tag": "opensnitch" } ] ``` Doesn't see anything in /var/log/syslog: grep opensnitch /var/log/syslog May 29 18:35:11 devel opensnitchd[25520]: [2023-05-29 15:35:11] IMP Starting opensnitch-daemon v1.6.0 May 29 18:35:11 devel opensnitchd[25520]: [2023-05-29 15:35:11] INF Loading rules from /etc/opensnitchd/rules ... May 29 18:35:11 devel opensnitchd[25520]: found /lib/x86_64-linux-gnu/libc.so.6

[gustavo-iniguez-goya]

The "syslog" logger doesn't support logging to IP addresses. Use "remote_syslog" with "rfc5424". I could send messages as json, but it looks like the standard is the rfc5424 or rfc3164 format (with structured data as KEY=VALUE format).

By the way, what's your elastic stack? logstash+elastic+kibana?

[rusbomber]

The "syslog" logger doesn't support logging to IP addresses. Use "remote_syslog" with "rfc5424". I could send messages as json, but it looks like the standard is the rfc5424 or rfc3164 format (with structured data as KEY=VALUE format).

Yep, now I'm getting logs in key-value format with proprietary SIEM collector. I want to get them in JSON and not to parse them with something resource heavy

By the way, what's your elastic stack? logstash+elastic+kibana?

No ELK stack right now, maybe later

[gustavo-iniguez-goya]

Yep, now I'm getting logs in key-value format with proprietary SIEM collector. I want to get them in JSON and not to parse them with something resource heavy

I need all the details of your SIEM and what input options it offers, to see if any of the logging options we offer is enough, or what we can add or modify. Otherwise I'll close this issue.

[rusbomber]

Well, I don't understand why you need any details about SIEM I use for testing purposes to make logging in such a common format like JSON. This SIEM is pretty much work in progress and is not production ready yet. I just have contacts with developers and they were kind enough to let me have a look at it. As of now, they have Syslog, SNMP, SQL, FTP, WMI, SFTP, SSH, xFlow, HTTP collectors. My scenario was to test syslog collector with several event sources, like auditd, firewalls (for example, opensnitch), IDS etc. Syslog collector by itself parse CEF and JSON pretty well already, so I want opensnitch to be able to send events in JSON.

Anyway, I'm not 100% shure if I'm gonna use it in future work. My next choice for testing would be probably Wazuh, which is built around ELK stack.

[gustavo-iniguez-goya]

Well, I don't understand why you need any details about SIEM I use for testing purposes to make logging in such a common format like JSON.

Because I need to understand how it ingest logs, saying "I want to get them in JSON" does not help. The RFC5424, RFC3164 or CEF clearly specifies in what format you must send the data byte by byte.

What does the SIEM expect? a RFC5424/RFC3164 message followed by a json document instead of the standard structured data (KEY=VALUE)? or nothing at all, just the json document?

Elastic: https://www.elastic.co/guide/en/elasticsearch/plugins/current/ingest.html

Grafana: https://grafana.com/blog/2022/06/23/how-to-send-logs-to-grafana-loki-with-the-opentelemetry-collector-using-fluent-forward-and-filelog-receivers/

All of them have clear and concise tutorials and howtos on how to prepare and send the logs to them.

[rusbomber]

Understood. Just the JSON document got parsed OK in SIEM, I can reproduce it by sending the string from my first message here:

echo '{"ECS.Base.Timestamp":"2023-05-23T13:06:01.704054+03:00", "ECS.Host.Hostname":"firewall", "ECS.Process.Name":"opensnitch", "ECS.Event.Action":"always-deny-incoming-avahi-daemon", "CEFX.DeviceInboundInterface":"eth1", "CEFX.DeviceOutboundInterface":"", "ECS.Source.MAC":"01:00:5e:00:00:fb", "ECS.Destination.MAC":"b6:0b:e5:ed:e3:3c", "MAC_PROTO":"08:00", "ECS.Source.Address":"192.168.3.195", "ECS.Destination.Address":"224.0.0.251", "LEN":"145", "TOS":"0x00", "PREC":"0x00", "TTL":"255", "ECS.Event.ID":"14262", "ip_flags":"", "ECS.Network.Transport":"UDP", "ECS.Source.Port":"5353", "ECS.Destination.Port":"5353", "UDP-LEN":"125"}' > /dev/tcp/192.168.2.129/49000