elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
12.14k stars 4.91k forks source link

Reading from JSON File Gives an error allthough json Syntax is correct #36688

Closed addelovein closed 11 months ago

addelovein commented 11 months ago

I Have created my own module. But one log event creates the following error:

json error from filebeat

{"log.level":"warn","@timestamp":"2023-09-27T11:34:26.544+0200","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":446},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.September, 27, 9, 30, 40, 428000000, time.UTC), Meta:{\"pipeline\":\"filebeat-8.10.2-p2bsql-log-pipeline\"}, Fields:{\"agent\":{\"ephemeral_id\":\"af7ce066-43ac-43c4-9546-6595b7e57607\",\"id\":\"e48b9230-d4c8-4cc0-bb49-3d38fd60a4e4\",\"name\":\"LinuxSQL\",\"type\":\"filebeat\",\"version\":\"8.10.2\"},\"category\":\"2\",\"client_app_name\":\"Core Microsoft SqlClient Data Provider\",\"client_hostname\":\"PERWIN\",\"container\":{\"id\":\"p2b.json\"},\"database_id\":\"11\",\"database_name\":\"perf_pricing\",\"destination\":\"0x00000002\",\"ecs\":{\"version\":\"1.12.0\"},\"error_number\":\"207\",\"event\":{\"dataset\":\"p2bsql.log\",\"module\":\"p2bsql\",\"timezone\":\"+02:00\"},\"fileset\":{\"name\":\"log\"},\"host\":{\"architecture\":\"x86_64\",\"containerized\":false,\"hostname\":\"linuxsql\",\"id\":\"3bd8a7728f15490b822bc60c6102cedb\",\"ip\":[\"192.168.0.13\",\"xc\",\"172.18.0.1\",\"172.17.0.1\"],\"mac\":[\"02-42-18-ED-09-DE\",\"02-42-E6-93-0F-52\",\"26-FF-47-7F-9E-FC\"],\"name\":\"linuxsql\",\"os\":{\"codename\":\"jammy\",\"family\":\"debian\",\"kernel\":\"5.15.0-84-generic\",\"name\":\"Ubuntu\",\"platform\":\"ubuntu\",\"type\":\"linux\",\"version\":\"22.04.3 LTS (Jammy Jellyfish)\"}},\"input\":{\"type\":\"filestream\"},\"is_intercepted\":\"false\",\"log\":{\"file\":{\"device_id\":2065,\"inode\":131112,\"path\":\"/var/opt/mssql/log/p2b.json\"},\"offset\":113572},\"message\":\"Invalid column name 'SubscriptionReference'.\",\"service\":{\"type\":\"p2bsql\"},\"session_id\":\"74\",\"severity\":\"16\",\"sql_text\":\"(@__status_0 nvarchar(4000))SELECT [s].[Id], [s].[BankEntity], [s].[x], [s].[CreatedDate], [s].[EndDate], [s].[ExpireDate], [s].[ExternalId], [s].[IsPackage], [s].[ModifiedDate], [s].[PackageAllocationOrderType], [s].[ProductId], [s].[SourceSystemId], [s].[StartDate], [s].[Status], [s].[SubscriptionCycle], [s].[SubscriptionId], [s].[SubscriptionReference], [s].[x], [s].[Version]\\nFROM [SubscriptionData] AS [s]\\nWHERE [s].[Status] = @__status_0\",\"state\":\"1\",\"tags\":[\"test\"],\"user_defined\":\"false\",\"username\":\"sa\"}, Private:(*input_logfile.updateOp)(0xc00100fec0), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {\"type\":\"x_content_parse_exception\",\"reason\":\"[1:9] Unrecognized token 'Invalid': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 9]\",\"caused_by\":{\"type\":\"json_parse_exception\",\"reason\":\"Unrecognized token 'Invalid': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\\n at [Source: (org.elasticsearch.common.io.stream.ByteBufferStreamInput); line: 1, column: 9]\"}}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"}

So basically it sees the Quoted Word: Invalid as a token???

EXACT ROW IN LOGFILE

{"@timestamp": "2023-09-27T09:30:43.984Z", "error_number": "207", "severity": "16", "state": "1", "user_defined": "false", "category": "2", "destination": "0x00000002", "is_intercepted": "false", "message": "Invalid column name 'SubscriptionReference'.", "username": "sa", "sql_text": "(@__status_0 nvarchar(4000))SELECT [s].[Id], [s].[x], [s].[xx], [s].[CreatedDate], [s].[EndDate], [s].[ExpireDate], [s].[ExternalId], [s].[IsPackage], [s].[ModifiedDate], [s].[PackageAllocationOrderType], [s].[ProductId], [s].[SourceSystemId], [s].[StartDate], [s].[Status], [s].[xxx], [s].[xx], [s].[xxx], [s].[xxxxx], [s].[Version]\nFROM [SubscriptionData] AS [s]\nWHERE [s].[Status] = @__status_0", "session_id": "74", "database_name": "xxxxx", "database_id": "11", "client_hostname": "xx", "client_app_name": "Core Microsoft SqlClient Data Provider"}

Module config

type: filestream
id: xsql
paths:
  {{ range $i, $path := .paths }}
- {{$path}}
  {{ end }}
#exclude_files: [".gz$"]
#multiline.pattern: '^\d\d'
#multiline.negate: true
#multiline.match: after

processors:
  - add_locale: ~
  - add_fields:
      target: ''
      fields:
        ecs.version: 1.12.0
  - if:
      regexp:
        message: "^{"
    then:
      - decode_json_fields:
          fields: [ "message" ]
          target: ""
          process_array: false
          max_depth: 1
          overwrite_keys: true
          add_error_key: true
      - rename:
          fields:
            - from: _json.request.body
              to: _request
          ignore_missing: true
      - move_fields:
          from: "sql"
          to: ""

Module Pipeline

processors:
  - json:
      field: message
botelastic[bot] commented 11 months ago

This issue doesn't have a Team:<team> label.