Data of type F5Telemetry_virtualServers (and same for F5Telemetry_AVR) was dropped: The number of custom fields xxx is above the limit of 500 fields per data type

danielcayer commented 1 year ago

Environment

Telemetry Streaming Version:
BIG-IP Version:

Summary

The F5Telemetry_virtualServers table in Microsoft Sentinel is not populating even when utilizing the workaround proposed in article K42151385 (https://support.f5.com/csp/article/K42151385). All corresponding messages for tables F5Telemetry_virtualServers and also F5Telemetry_AVR from my BIG-IP using TS result in errors which can be seen in the Azure portal. I am under impression this issue should have been resolved since the 1.24.0 release of TS (as per final comment in issue number 156 - https://github.com/F5Networks/f5-telemetry-streaming/issues/156) This issue is easily reproducible. The impact is low since the alternate workaround is sufficient for the time being (as documented here: https://support.f5.com/csp/article/K42151385).

Steps To Reproduce

Steps to reproduce the behavior:

Submit the following declaration:

curl -sku username:password --location --request POST 'https://Management_IP_Address/mgmt/shared/telemetry/declare' \
--header 'Content-Type: application/json' \
--data-raw '{
"class": "Telemetry",
"controls": {
    "class": "Controls",
    "logLevel": "debug",
    "debug": true
},
"My_Listener": {
"class": "Telemetry_Listener",
"port": 6514
},
"Poller": {
"class": "Telemetry_System_Poller",
"interval": 60,
"enable": true,
"trace": false,
"allowSelfSignedCert": false,
"host": "localhost",
"port": 8100,
"protocol": "http",
"actions": [
{
"enable": true,
"includeData": {},
"locations": {
"system": true,
"virtualServers": true,
"httpProfiles": true,
"clientSslProfiles": true,
"serverSslProfiles": true
}
}
]
},
"Pull_Consumer": {
"class": "Telemetry_Pull_Consumer",
"type": "default",
"systemPoller": [
"Poller"
]
},

"Azure_Consumer": {
"class": "Telemetry_Consumer",
"type": "Azure_Log_Analytics",
"workspaceId": "xxxxxxxxxxxxxxxxxxxxx",
"passphrase": {
"cipherText": " xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx="
},
"format": "propertyBased"
},
"schemaVersion": "1.12.0"
}'

Observe the following error response: In the Azure portal browse to Microsoft Sentinel -> [click to select your workspace] -> Genera/Logs and run the "operation" query.
```
OperationStatus: Failed
```

Detail: Data of type F5Telemetry_virtualServers was dropped: The number of custom fields 501 is above the limit of 500 fields per data type. See https://aka.ms/AA593as to find instructions for removing unnecessary custom fields for this type.

OperationCategory: Ingestion OperationKey: Custom log ingestion CorrelationId: 5f384089-cba0-41ae-9846-ddb60b571b64 Type: Operation



### Expected Behavior
All LTM and AVR logs should be logged in Azure Log Analytics and Sentinel without exceeding the 500 fields limit.

### Actual Behavior
The Virtual server stats are not logged in Azure.
The following error is reported in Azure log: 
Data of type F5Telemetry_virtualServers was dropped: The number of custom fields 501 is above the limit of 500 fields per data type. See https://aka.ms/AA593as to find instructions for removing unnecessary custom fields for this type.

joernott commented 1 year ago

The same happens with elasticsearch as well:

Fri, 18 Nov 2022 12:55:16 GMT - finest: [telemetry.ElasticSearch.f5telemetry_default::Elasticsearch_cluster] response body: {"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse"}],"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] has been exceeded while adding new fields [1001]"}},"status":400}

I edited the elasticsearch index mapping (see https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-settings-limit.html) and set "index.mapping.total_fields.limit": "10000", as even 2k was not enough.

It currently creates 3177 fields on my test ltm which is basically empty

joernott commented 1 year ago

I am currently working with the data modification feature (see https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/data-modification.html) to limit the output to the relevant fields. Maybe, that is also a way for you to reduce the number of fields.

joernott commented 1 year ago

Using the following actions, I have reduced the number of fields from 3177 to 674 for 16 pools with 1 member each. There are still a lot of superfluous infos in there, so it should be possible to condense it even more.

    "My_System": {
        "class": "Telemetry_System",
        "systemPoller": {
            "interval": 60,
            "actions": [
                {
                    "includeData": {},
                    "locations": {
                        "pools": {
                            ".*": {
                                "activeMemberCnt": true,
                                "availabilityState": true,
                                "enabledState": true,
                                "members": {
                                    ".*": {
                                        "availabilityState": true,
                                        "enabledState": true,
                                        "monitorStatus": true,
                                        "serverside": true
                                    }
                                },
                                "serverside": true,
                                "status": true,
                                "totRequests": true
                            }
                        },
                        "system": {
                            "connectionsPerformance": {
                                ".*": true
                            },
                            "cpu": true,
                            "memory": true,
                            "networkInterfaces": true,
                            "swap": true,
                            "throughputPerformance": true,
                            "tmmCpu": true,
                            "tmmMemory": true,
                            "tmmTraffic": true
                        },
                        "virtualservers": {
                            ".*": {
                                "availabilityState": true,
                                "clientSide": true,
                                "totRequests": true
                            }
                        }
                    }
                }
            ]
        }
    },

DahlPatric commented 8 months ago

Helped me to solve 500 limit.


    "actions": [
          {
            "includeData": {},
            "locations": {
              "telemetryServiceInfo": true,
              "system": true,
              "virtualServers": {
                ".*": {
                  "availabilityState": true,
                  "clientside": true,
                  "status.statusReason": true,
                  "totRequests": true,
                  "name": true
                }
              },
              "pools": true
            }
          }
        ]
      }

F5Networks / f5-telemetry-streaming