Using sc4s alternate destination config to forward syslog to a thirdparty SIEM do not retain the source IP after being being received by a SIEM

aalisher-tmx commented 3 weeks ago

Was the issue replicated by support? Yes. Case #3475996 What is the sc4s version ? 3.16.0 Which operating system (including its version) are you using for hosting SC4S? Rhel 8 Which runtime (Docker, Podman, Docker Swarm, BYOE, MicroK8s) are you using for SC4S? podman Is there a pcap available? If so, would you prefer to attach it to this issue or send it to Splunk support? Yes, will send seperately once the issue is assigned Is the issue related to the environment of the customer or Software related issue? Software related Is it related to Data loss, please explain ? Protocol? Hardware specs?

Last chance index/Fallback index?

Is the issue related to local customization?

Do we have all the default indexes created?

Describe the bug A clear and concise description of what the bug is. All our syslog devices are ingesting syslog to Splunk Cloud via sc4s app using HEC. No issues. All good. Also, using alternate destinations config (https://splunk.github.io/splunk-connect-for-syslog/main/destinations/#example-5-cisco-asa-send-to-a-third-party-siem), we want to forward to the third-party SIEM (QRADAR) appliance. I followed the documentation and logs are showing in qradar, but with the following issue:

The source IP is being overwritten with the sc4s server instead of the actual source IP after being received by a SIEM from sc4s. - client IP ----> sc4s (syslog-ng) ---> SIEM (source is coming up as sc4s server instead of client IP).
looks like the syslog protocol is unable to maintain the original sender's address when forwarding syslog messages. This is because the sender's address is taken from the received TCP packet.

env_file:

SC4S_DEST_SYSLOG_QRADARSIEM_HOST=x.x.x.x SC4S_DEST_SYSLOG_QRADARSIEM_PORT=514 SC4S_DEST_SYSLOG_QRADARSIEM_MODE=SELECT

set to #yes for ietf frames

SC4S_DEST_SYSLOG_QRADARSIEM_IETF=yes

To Reproduce Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

aalisher-tmx commented 1 week ago

Hi @rjha-splunk any update on this? Thanks

rjha-splunk commented 6 days ago

Hi @aalisher-tmx unfortunately syslog-ng under the hood retains the last relays ip as IP of the source, in this case it is SC4S https://syslog-ng.github.io/admin-guide/110_Template_and_rewrite/000_Customize_message_format/004_Macros_of_syslog-ng#sourceip

I can still send this information using templates and reset the soft MACRO host with source ip here

Example env_config: SC4S_DEST_SYSLOG_TEST_HOST=x.x.x.x SC4S_DEST_SYSLOG_TEST_PORT=514 SC4S_DEST_SYSLOG_TEST_MODE=SELECT SC4S_DEST_SYSLOG_TEST_TRANSPORT=UDP SC4S_SOURCE_STORE_RAWMSG=yes

Example conf file:

block parser sc4s-lp-cisco_asa_d_syslog_test(){
    channel {
        rewrite {
        set('$SOURCEIP', value('HOST'));
    };
 };

};

application sc4s-lp-cisco_asa_d_syslog_test[sc4s-lp-dest-select-d_syslog_test] {
    filter {
        'cisco' eq "${fields.sc4s_vendor}" and 'asa' eq "${fields.sc4s_product}";
    };

    parser {
        sc4s-lp-cisco_asa_d_syslog_test();
        syslog-parser(template("$MESSAGE"));
    };
};

aalisher-tmx commented 6 days ago

No, it didn't work @rjha-splunk SIEM is looking into the header info and calculating the source ip as 10.152.210.206 (which is the sc4s server instead of the actual host sending the packet).

As per your config, host is being changed to IP in the sample message content shown below (see 10.152.208.98), but we don't want that

Sample Packet Header: 18:21:57.494911 IP (tos 0x0, ttl 64, id 61068, offset 0, flags [DF], proto TCP (6), length 159) 10.152.210.206.41025 > 10.152.1.129.514: Flags [P.], cksum 0xe9fc (incorrect -> 0x470d), seq 2382849015:2382849122, ack 1794184352, win 58, options [nop,nop,TS val 1786126440 ecr 2148056678], length 107

Sample Message: 1.2024-06-24T22:21:57.494+00:00.10.152.208.98.-.6736.-.-....0].info:.start.of.service.(unbound.1.6.6)..

rjha-splunk commented 6 days ago

We need to look for a solution and possible enhancement, we will keep you posted

rjha-splunk commented 5 days ago

I investigated and checked, right now i have following for the case:

1) we cant make it possible for TCP, UDP we can try to implement that using spoof-source and python parsers but it will be only possible after syslog-ng 4.8.0 release

2) Right now using parsers Sourceip information can be added to the header.


application sc4s-lp-cisco_asa_d_syslog_test[sc4s-lp-dest-select-d_syslog_test] {
    filter {
        'cisco' eq "${fields.sc4s_vendor}" and 'asa' eq "${fields.sc4s_product}";
    };

    parser {
        syslog-parser(template("SourceIP=${SOURCEIP} ${MSG}\n"));
    };
};

rjha-splunk commented 4 days ago

Closing this issue as per above comment, SC4S today cant spoof tcp packets intial origin source.

aalisher-tmx commented 4 days ago

2. Right now using parsers Sourceip information can be added to the header.

application sc4s-lp-cisco_asa_d_syslog_test[sc4s-lp-dest-select-d_syslog_test] {
    filter {
        'cisco' eq "${fields.sc4s_vendor}" and 'asa' eq "${fields.sc4s_product}";
    };

    parser {
        syslog-parser(template("SourceIP=${SOURCEIP} ${MSG}\n"));
    };
};

was this tested? above config doesn't update the header source ip.

rjha-splunk commented 4 days ago

Yes it is tested with ASA sample, it will just add sourceip to the message , i will share the screenshot tomorrow

aalisher-tmx commented 4 days ago

Oh, adding sourceip to the message not to the header. This doesn't resolve our issue. Can you please let us know if there are any timelines to get my issue resolved? Thanks

aalisher-tmx commented 4 days ago

documentation also needs to be updated appropriately as this is not working as expected - https://splunk.github.io/splunk-connect-for-syslog/main/destinations/#example-5-cisco-asa-send-to-a-third-party-siem

rjha-splunk commented 4 days ago

As mentioned in my previous comment, right now syslog-ng under the hood doesn't provide this capability, we are working with them to get a work around.

For now alternate destinations can be explored using edge processor( splunk)

SC4S can be used to send data to third party but it will be written as source because Syslog does not include this information. We do embed it into otlp as shown but the source is lost as soon as it hops

aalisher-tmx commented 3 days ago

Thanks @rjha-splunk

splunk / splunk-connect-for-syslog

Using sc4s alternate destination config to forward syslog to a thirdparty SIEM do not retain the source IP after being being received by a SIEM #2488

env_file:

set to #yes for ietf frames