elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
92 stars 4.92k forks source link

Azure Eventhub Input fails to decode invalid JSON sent via Azure Diagnostic Settings #40641

Closed ckauf closed 1 month ago

ckauf commented 2 months ago

For confirmed bugs, please report:

Flexible Azure Database for PostgreSQL-Server sends invalid JSON like this:

{
  "records": [
    {
      "AppImage": "orcas/postgres_standalone_16_u18:38.1.240825",
      "AppType": "PostgreSQL",
      "AppVersion": "breadthpg16_2024-08-06-07-22-43",
      "Region": "westeurope",
      "category": "PostgreSQLLogs",
      "location": "westeurope",
      "operationName": "LogEvent",
      "properties": [
        227 B blob data
      ],
      "resourceId": "/SUBSCRIPTIONS/01234567-89AB-CDEF-0123-456789ABCDEF/RESOURCEGROUPS/RG-OBS/PROVIDERS/MICROSOFT.DBFORPOSTGRESQL/FLEXIBLESERVERS/OBS-PG-TEST",
      "time": "2024-08-27T14:26:06.446Z",
      "ServerType": "PostgreSQL",
      "LogicalServerName": "psql-pg-test",
      "ServerVersion": "breadthpg16_2024-08-06-07-22-43",
      "ServerLocation": "prod:westeurope",
      "ReplicaRole": "Primary",
      "OriginalPrimaryServerName": "psql-pg-test"
    },
    {
      "AppImage": "orcas/postgres_standalone_16_u18:38.1.240825",
      "AppType": "PostgreSQL",
      "AppVersion": "breadthpg16_2024-08-06-07-22-43",
      "Region": "westeurope",
      "category": "PostgreSQLLogs",
      "location": "westeurope",
      "operationName": "LogEvent",
      "properties": {
        "timestamp": "2024-08-27 14:26:06.447 UTC",
        "processId": 430,
        "errorLevel": "LOG",
        "sqlerrcode": "00000",
        "message": "2024-08-27 14:26:06 UTC-66cde1fe.1ae-LOG:  background worker \"is_db_alive\" (PID 443) exited with exit code 1"
      },
      "resourceId": "/SUBSCRIPTIONS/01234567-89AB-CDEF-0123-456789ABCDEF/RESOURCEGROUPS/RG-OBS/PROVIDERS/MICROSOFT.DBFORPOSTGRESQL/FLEXIBLESERVERS/OBS-PG-TEST",
      "time": "2024-08-27T14:26:06.447Z",
      "ServerType": "PostgreSQL",
      "LogicalServerName": "psql-pg-test",
      "ServerVersion": "breadthpg16_2024-08-06-07-22-43",
      "ServerLocation": "prod:westeurope",
      "ReplicaRole": "Primary",
      "OriginalPrimaryServerName": "psql-pg-test"
    },
    {
      "AppImage": "orcas/postgres_standalone_16_u18:38.1.240825",
      "AppType": "PostgreSQL",
      "AppVersion": "breadthpg16_2024-08-06-07-22-43",
      "Region": "westeurope",
      "category": "PostgreSQLLogs",
      "location": "westeurope",
      "operationName": "LogEvent",
      "properties": [
        218 B blob data
      ],
      "resourceId": "/SUBSCRIPTIONS/01234567-89AB-CDEF-0123-456789ABCDEF/RESOURCEGROUPS/RG-OBS/PROVIDERS/MICROSOFT.DBFORPOSTGRESQL/FLEXIBLESERVERS/OBS-PG-TEST",
      "time": "2024-08-27T14:26:08.629Z",
      "ServerType": "PostgreSQL",
      "LogicalServerName": "psql-pg-test",
      "ServerVersion": "breadthpg16_2024-08-06-07-22-43",
      "ServerLocation": "prod:westeurope",
      "ReplicaRole": "Primary",
      "OriginalPrimaryServerName": "psql-pg-test"
    }
  ]
}

The first and third event in the records list do contain a properties array, which contains raw text not encapsulated in ":

"properties": [
  218 B blob data
],

When the Azure Eventhub input tries to decode this JSON message it fails due to the syntax error in the JSON and subsequently is not able to extract the single events stored in the records list. While this issue is clearly caused by the Azure Service, it would be great if the Eventhub input could handle this kind of malformed JSON.

elasticmachine commented 2 months ago

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)