search

F5Networks / f5-telemetry-streaming

F5 BIG-IP Telemetry Streaming
Apache License 2.0
53 stars 24 forks source link

Telemetry adds the "OriginalRawData" field #217

Open SankoZaha opened 2 years ago

SankoZaha commented 2 years ago

Environment

Summary

When sending a log to the consumer (Splunk), a field named "OriginalRawData" is added by Telemetry Streaming. This field contains the "entire log" in Syslog format, which is not ideal because it consumes twice the bandwidth. There doesn't seem to be any way to remove this field at this time.

Steps To Reproduce

Steps to reproduce the behavior:

  1. Submit the following declaration:

    {
"class": "Telemetry",
"System_Polling": {
    "class": "Telemetry_System",
    "trace": false,
    "systemPoller": {
        "interval": 60
    }
},
"Listener_TS": {
    "class": "Telemetry_Listener",
    "port": 6514
},
"Splunk_Peer": {
    "class": "Telemetry_Consumer",
    "trace": false,
    "allowSelfSignedCert": true,
    "type": "Splunk",
    "host": "example.domain.fr",
    "protocol": "https",
    "port": 8088,
    "passphrase": {
        "cipherText": "XXXXXXXXXX"
    }
}
}

  2. Capture the Telemetry Streaming output to verify the addition of the "OriginalRawData" field:

    {
"time": 1658944099000,
"host": "example.domain.fr",
"source": "f5.telemetry",
"sourcetype": "f5:telemetry:json",
"event": {
"hostname": "example.domain.fr",
"management_ip_address": "XXX.XXX.XXX.XXX",
"management_ip_address_2": "",
"http_class_name": "/Common/asm_policy_Test_v1.0.0",
"web_application_name": "/Common/asm_policy_Test_v1.0.0",
"policy_name": "/Common/asm_policy_Test_v1.0.0",
"policy_apply_date": "2022-07-20 14:48:03",
"violations": "Illegal request length,Illegal URL length,Illegal file type",
"support_id": "5632789474866890585",
"request_status": "blocked",
"response_code": "0",
"ip_client": "XXX.XXX.XXX.XXX",
"route_domain": "0",
"method": "GET",
"protocol": "HTTPS",
"query_string": "",
"x_forwarded_for_header_value": "N/A",
"sig_ids": "",
"sig_names": "",
"date_time": "2022-07-27T17:48:18.000Z",
"severity": "Critical",
"attack_type": "Buffer Overflow,Forceful Browsing",
"geo_location": "N/A",
"ip_address_intelligence": "N/A",
"username": "N/A",
"session_id": "44f154d1f965ec02",
"src_port": "49241",
"dest_port": "8080",
"dest_ip": "XXX.XXX.XXX.XXX",
"sub_violations": "",
"virus_name": "N/A",
"violation_rating": "3",
"websocket_direction": "N/A",
"websocket_message_type": "N/A",
"device_id": "N/A",
"staged_sig_ids": "",
"staged_sig_names": "",
"threat_campaign_names": "",
"staged_threat_campaign_names": "",
"blocking_exception_reason": "N/A",
"captcha_result": "not_received",
"microservice": "",
"vs_name": "/Common/VS_Test",
"uri": "/webgoat/css/animate.css",
"fragment": "",
"request": "GET /WebGoat/css/animate.css HTTP/1.1\\r\\nHost: XXX.XXX.XXX.XXX:8080\\r\\nConnection: keep-alive\\r\\nsec-ch-ua: %22 Not A;Brand%22;v=%2299%22, %22Chromium%22;v=%22102%22, %22Microsoft Edge%22;v=%22102%22\\r\\nDNT: 1\\r\\nsec-ch-ua-mobile: ?0\\r\\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.148 Safari/537.36 Edg/102.0.1245.56\\r\\nsec-ch-ua-platform: %22Windows%22\\r\\nAccept: text/css,*/*;q=0.1\\r\\nSec-Fetch-Site: same-origin\\r\\nSec-Fetch-Mode: no-cors\\r\\nSec-Fetch-Dest: style\\r\\nReferer: https://XXX.XXX.XXX.XXX:8080/WebGoat/login\\r\\nAccept-Encoding: gzip, deflate, br\\r\\nAccept-Language: fr,fr-FR;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6\\r\\nCookie: JSESSIONID=XXXXXXXXXXX; TS017b6ae6=XXXXXXXXXXX",
"telemetryEventCategory": "ASM",
"originalRawData": "<130>Jul 27 19:48:19 example.domain.fr ASM:unit_hostname=\"example.domain.fr\",management_ip_address=\"XXX.XXX.XXX.XXX\",management_ip_address_2=\"\",http_class_name=\"/Common/asm_policy_Test_v1.0.0\",web_application_name=\"/Common/asm_policy_Test_v1.0.0\",policy_name=\"/Common/asm_policy_Test_v1.0.0\",policy_apply_date=\"2022-07-20 14:48:03\",violations=\"Illegal request length,Illegal URL length,Illegal file type\",support_id=\"5632789474866890585\",request_status=\"blocked\",response_code=\"0\",ip_client=\"XXX.XXX.XXX.XXX\",route_domain=\"0\",method=\"GET\",protocol=\"HTTPS\",query_string=\"\",x_forwarded_for_header_value=\"N/A\",sig_ids=\"\",sig_names=\"\",date_time=\"2022-07-27 19:48:18\",severity=\"Critical\",attack_type=\"Buffer Overflow,Forceful Browsing\",geo_location=\"N/A\",ip_address_intelligence=\"N/A\",username=\"N/A\",session_id=\"44f154d1f965ec02\",src_port=\"49241\",dest_port=\"8080\",dest_ip=\"XXX.XXX.XXX.XXX\",sub_violations=\"\",virus_name=\"N/A\",violation_rating=\"3\",websocket_direction=\"N/A\",websocket_message_type=\"N/A\",device_id=\"N/A\",staged_sig_ids=\"\",staged_sig_names=\"\",threat_campaign_names=\"\",staged_threat_campaign_names=\"\",blocking_exception_reason=\"N/A\",captcha_result=\"not_received\",microservice=\"\",vs_name=\"/Common/VS_Test\",uri=\"/webgoat/css/animate.css\",fragment=\"\",request=\"GET /WebGoat/css/animate.css HTTP/1.1\\r\\nHost: XXX.XXX.XXX.XXX:8080\\r\\nConnection: keep-alive\\r\\nsec-ch-ua: %22 Not A;Brand%22;v=%2299%22, %22Chromium%22;v=%22102%22, %22Microsoft Edge%22;v=%22102%22\\r\\nDNT: 1\\r\\nsec-ch-ua-mobile: ?0\\r\\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.148 Safari/537.36 Edg/102.0.1245.56\\r\\nsec-ch-ua-platform: %22Windows%22\\r\\nAccept: text/css,*/*;q=0.1\\r\\nSec-Fetch-Site: same-origin\\r\\nSec-Fetch-Mode: no-cors\\r\\nSec-Fetch-Dest: style\\r\\nReferer: https://XXX.XXX.XXX.XXX:8080/WebGoat/login\\r\\nAccept-Encoding: gzip, deflate, br\\r\\nAccept-Language: fr,fr-FR;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6\\r\\nCookie: JSESSIONID=XXXXXXXXXXX; TS017b6ae6=XXXXXXXXXXX\r\n<130>Jul 27 19:48:19 example.domain.fr ASM:unit_hostname=\"example.domain.fr\",management_ip_address=\"XXX.XXX.XXX.XXX\",management_ip_address_2=\"\",http_class_name=\"/Common/asm_policy_Test_v1.0.0\",web_application_name=\"/Common/asm_policy_Test_v1.0.0\",policy_name=\"/Common/asm_policy_Test_v1.0.0\",policy_apply_date=\"2022-07-20 14:48:03\",violations=\"Illegal request length,Illegal URL length,Illegal file type\",support_id=\"5632789474866897492\",request_status=\"blocked\",response_code=\"0\",ip_client=\"XXX.XXX.XXX.XXX\",route_domain=\"0\",method=\"GET\",protocol=\"HTTPS\",query_string=\"\",x_forwarded_for_header_value=\"N/A\",sig_ids=\"\",sig_names=\"\",date_time=\"2022-07-27 19:48:18\",severity=\"Critical\",attack_type=\"Buffer Overflow,Forceful Browsing\",geo_location=\"N/A\",ip_address_intelligence=\"N/A\",username=\"N/A\",session_id=\"44f154d1f965ec02\",src_port=\"49244\",dest_port=\"8080\",dest_ip=\"XXX.XXX.XXX.XXX\",sub_violations=\"\",virus_name=\"N/A\",violation_rating=\"3\",websocket_direction=\"N/A\",websocket_message_type=\"N/A\",device_id=\"N/A\",staged_sig_ids=\"\",staged_sig_names=\"\",threat_campaign_names=\"\",staged_threat_campaign_names=\"\",blocking_exception_reason=\"N/A\",captcha_result=\"not_received\",microservice=\"\",vs_name=\"/Common/VS_Test\",uri=\"/favicon.ico\",fragment=\"\",request=\"GET /favicon.ico HTTP/1.1\\r\\nHost: XXX.XXX.XXX.XXX:8080\\r\\nConnection: keep-alive\\r\\nsec-ch-ua: %22 Not A;Brand%22;v=%2299%22, %22Chromium%22;v=%22102%22, %22Microsoft Edge%22;v=%22102%22\\r\\nDNT: 1\\r\\nsec-ch-ua-mobile: ?0\\r\\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.148 Safari/537.36 Edg/102.0.1245.56\\r\\nsec-ch-ua-platform: %22Windows%22\\r\\nAccept: image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8\\r\\nSec-Fetch-Site: same-origin\\r\\nSec-Fetch-Mode: no-cors\\r\\nSec-Fetch-Dest: image\\r\\nReferer: https://XXX.XXX.XXX.XXX:8080/WebGoat/login\\r\\nAccept-Encoding: gzip, deflate, br\\r\\nAccept-Language: fr,fr-FR;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6\\r\\nCookie: TS019131d9026=XXXXXXXXXXX",
"tenant": "Common"
}
}

Expected Behavior

Be able to send TS events without the OriginalRawData field.

Actual Behavior

As explained, at each event sent, the "OiginalRawData" field is added.

dstokesf5 commented 2 years ago

Thank you for your feedback. This sounds like some debug data being sent unintentionally. I have added this issue to our internal product backlog as TS-698.

pixelbrainz commented 1 year ago

@SankoZaha and any others that are experiencing this still with Splunk on the latest TS code: Another option while this is still getting sorted is to move your consumer type from Splunk to Generic_HTTP (this does not send the 'originalRawData' in the payload). The tuning unfortunately does not end here. This will also require some extra work with iRule logging and/or traffic profiles/JMESpath depending on your requirements. See the following guides.

The upside here is with the generic consumer you really open the door to how customized you want to get with your logging. The downside is that if you want something that will just do it all like the Splunk consumer then this is not the consumer for you.

All else, we will just have to hurry up and wait to find out what Telemetry is being replaced with.

https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/declarations.html#multiple

https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/schema-reference.html