F5Networks / f5-telemetry-streaming

F5 BIG-IP Telemetry Streaming
Apache License 2.0
56 stars 23 forks source link

Unable to send ASM logs to a HTTP endpoint via telemetry streaming #280

Open harshnasitcrest opened 2 months ago

harshnasitcrest commented 2 months ago

Environment

Summary

I'm able to generate ASM logs deployed in our on-premise setup but unable to send it via telemetry streaming to a HTTP endpoint (consumer type is Generic_HTTP).

Steps To Reproduce

Steps to reproduce the behavior:

  1. Submit the following declaration:

    {
    "class": "Telemetry",
     "My_ASM_Listener": {
        "class": "Telemetry_Listener",
        "port": 6514,  
        "trace": true
        "match": "ASM",
        "actions": [
            {
                "setTag": {
                    "application": "`ASM`"
                        },
                "enable": true
                }
            ]
    
    },
    "My_Consumer": {
        "class": "Telemetry_Consumer",
        "type": "Generic_HTTP",
        "host": "10.50.9.132",
        "protocol": "http",
        "port": 5151,
        "path": "/post",
        "headers": [
            {"name": "Authorization", "value": "12345689"},
            {"name": “ID1", "value": "ABC"},
            {"name": "ID2", "value": "XYZ"}
    
        ],
        "actions": [
            {
                "JMESPath": {},
                "expression": "{ logs: [@] }"
            }
        ]
    }
    }
  2. On submitting above declaration, we're getting 200 response code.

  3. Verified that ASM logs are generated in the backend at /var/log/asmdata1/request_log and shown on the F5 UI at: Security --> Events logs --> Application --> Requests.

  4. Log level was set to debug but unable to see API failure logs. Following are the contents of logs at /var/log/restnoded/restnoded.log:

    Screenshot 1:

    Screenshot 2024-09-11 at 12 33 50 PM

    Screenshot 2: unnamed

    1. Verified that enough resources are provided for this setup to work.

Expected Behavior

pgouband commented 2 months ago

Hi @harshnasitcrest,

Have you tried to apply the following and fine tune memory management?

https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/troubleshooting.html#why-is-my-big-ip-experiencing-occasional-high-cpu-usage-and-slower-performance

https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/memory-monitor.html?highlight=beta

pgouband commented 2 months ago

Hi @harshnasitcrest,

Can you try to disable system poller and test it? If the issue occurs even with system poller disabled, you should increase the amount of memory for host system.

harshnasitcrest commented 2 months ago

Referring to https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/telemetry-system.html#system-poller , I disabled system poller using following declaration:

"My_System_Minimal": {
     "class": "Telemetry_System",
     "systemPoller": {
         "enable": false
     }
 }

I increased memory and CPU both after disabling system poller.

Somehow, I don't see memory and tomcat restart errors now, but I still don't see any logs regarding failures in making API call to destination.

Does the last log saying Applying restrictions to incoming data causing the trouble? Any other steps I can try? Is it usually this cumbersome to configure and send ASM events to HTTP server?

petrov-serg commented 1 month ago

what is your full declaration? (you can mask/remove secrets from it)

harshnasitcrest commented 1 month ago

what is your full declaration? (you can mask/remove secrets from it)

@petrov-serg Here's the full declaration that I posted using POST request to https://<ip>/mgmt/shared/telemetry/declareAPI:

{
    "class": "Telemetry",
    "My_ASM_Listener": {
        "class": "Telemetry_Listener",
        "port": 6514,
        "trace": true,
        "match": "ASM",
        "actions": [
            {
                "setTag": {
                    "application": "`ASM`"
                },
                "enable": true
            }
        ]
    },
    "My_System_Minimal": {
        "class": "Telemetry_System",
        "systemPoller": {
            "enable": false
        }
    },
    "My_Consumer": {
        "class": "Telemetry_Consumer",
        "type": "Generic_HTTP",
        "host": "10.50.9.132",
        "protocol": "http",
        "port": 5151,
        "path": "/post",
        "headers": [
            {
                "name": "Authorization",
                "value": "12345689"
            },
            {
                "name": "ID1",
                "value": "ABC"
            },
            {
                "name": "ID2",
                "value": "XYZ"
            }
        ],
        "actions": [
            {
                "JMESPath": {},
                "expression": "{ logs: [@] }"
            }
        ]
    }
}