[darktrace] object mapping for [darktrace.model_breach_alert.acknowledged] tried to parse field [acknowledged] as object, but found a concrete value.

Integration Name

Darktrace [darktrace]

Dataset Name

darktrace.model_breach_alert

Integration Version

1.19.0

Agent Version

8.14.3

Agent Output Type

elasticsearch

Elasticsearch Version

8.14.3

OS Version and Architecture

Ubuntu 22.04 LTS

Software/API Version

DarkTrace v6.1

Error Message

[elastic_agent.filebeat][debug] Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Meta:null, Fields:null, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}, EncodedEvent:(*elasticsearch.encodedEvent)(0xc006280e00)} (status=400): {"type":"document_parsing_exception","reason":"[1:1393] object mapping for [darktrace.model_breach_alert.acknowledged] tried to parse field [acknowledged] as object, but found a concrete value"}, dropping event!

Event Original

{ "acknowledged": false, "breachUrl": "https://someurl.com/#modelbreach/25196", "commentCount": 0, "creationTime": 1731070657000, "device": { "did": 7281, "firstSeen": 1623686103000, "ip": "192.168.1.1", "lastSeen": 1731072276000, "macaddress": "57:d5:10:r3:13:fb", "os": "FreeBSD", "ossource": "TCP", "sid": 76, "typelabel": "Laptop", "typename": "laptop", "vendor": "Some Vendor" }, "devicepercentscore": 48, "devicescore": 0.476, "did": 7281, "model": { "actions": { "alert": true, "antigena": {}, "breach": true, "model": true, "setPriority": false, "setTag": false, "setType": false }, "active": true, "activeTimes": { "devices": {}, "tags": {}, "type": "exclusions", "version": 2 }, "autoSuppress": true, "autoUpdatable": true, "autoUpdate": true, "behaviour": "decreasing", "category": "Informational", "compliance": false, "created": { "by": "System" }, "defeats": [ { "arguments": { "value": "some.hostname" }, "comparator": "matches", "defeatID": 1, "filtertype": "Connection hostname" }, { "arguments": { "value": "some.connection" }, "comparator": "matches", "defeatID": 2, "filtertype": "Connection hostname" }, { "arguments": { "value": "another.connection" }, "comparator": "matches", "defeatID": 3, "filtertype": "Connection hostname" } ], "delay": 0, "description": "A device is moving large volumes of data (1GiB+) out of the network within a short period of time. \n\nAction: Investigate if the external data transfer is a legitimate business activity or a loss of corporate data.", "edited": { "by": "System" }, "interval": 0, "logic": { "data": [ 620291 ], "type": "componentList", "version": 1 }, "mitre": { "tactics": [ "exfiltration" ], "techniques": [ "T1041", "T1567.002" ] }, "modified": "2023-12-05 22:39:00", "name": "Anomalous Connection::Uncommon 1 GiB Outbound", "phid": 609255, "pid": 600644, "priority": 2, "sequenced": false, "sharedEndpoints": false, "tags": [ "AP: Egress", "OT Engineer" ], "throttle": 86400, "uuid": "1f00371b-7681-42d5-5867-c744441898dc", "version": 29 }, "pbid": 25196, "pbscore": 0.634, "percentscore": 48, "score": 0.476, "time": 1731070643000, "triggeredComponents": [ { "cbid": 34058, "chid": 630381, "cid": 620291, "interval": 3600, "logic": { "data": { "left": { "left": "A", "operator": "AND", "right": { "left": "B", "operator": "AND", "right": "C" } }, "operator": "OR", "right": { "left": { "left": "A", "operator": "AND", "right": { "left": "B", "operator": "AND", "right": "D" } }, "operator": "OR", "right": { "left": "A", "operator": "AND", "right": { "left": "B", "operator": "AND", "right": "E" } } } }, "version": "v0.1" }, "metric": { "label": "Model", "mlid": 234, "name": "dtmodelbreach" }, "size": 1, "threshold": 0, "time": 1731070642000, "triggeredFilters": [ { "arguments": { "value": "Device / Anomaly Indicators / 1 GiB Outbound" }, "cfid": 254461, "comparatorType": "matches", "filterType": "Message", "id": "A", "trigger": { "value": "Device / Anomaly Indicators / 1 GiB Outbound" } }, { "arguments": { "value": 95 }, "cfid": 254462, "comparatorType": ">", "filterType": "New or uncommon occurrence", "id": "B", "trigger": { "value": "100" } }, { "arguments": { "value": 10 }, "cfid": 254464, "comparatorType": ">", "filterType": "Rare external IP", "id": "D", "trigger": { "value": "100" } }, { "arguments": {}, "cfid": 254466, "comparatorType": "display", "filterType": "Connection hostname", "id": "d1", "trigger": { "value": "another.connection.hostname" } }, { "arguments": {}, "cfid": 254467, "comparatorType": "display", "filterType": "Destination IP", "id": "d2", "trigger": { "value": "1.1.1.1" } }, { "arguments": {}, "cfid": 254468, "comparatorType": "display", "filterType": "ASN", "id": "d3", "trigger": { "value": "SOME ASN" } }, { "arguments": {}, "cfid": 254469, "comparatorType": "display", "filterType": "Destination port", "id": "d4", "trigger": { "value": "80" } } ] } ] }

What did you do?

Standard, OOTB config for the DT integration, no changes made to ingest pipeline, templates etc.

What did you see?

What did you expect to see?

The event get indexed as expected.

Anything else?

Event.original contains "acknowledged": false, only.

The subsequent json payload that gets extracted within the ingest pipeline contains "json.acknowledged: false"

Initial workaround was to clone the Ingest pipeline and remove the parts of the Ingest Pipeline shown in attached image DarkTrace_1 and DarkTrace_2.,

I then cloned the Index template 'logs-darktrace.model_breach_alert', replaced the component template with the modified version and then cloned and removed the mappings for the darktrace.model_breach_alert.is_acknowledged field. This was so I could work out what was going on. I've since updated it, so it tries to parse time and username but doesn't fail and removes the json, but this should give you an idea of what's up.

Currently working with DT v6.1, suspect this may be fixed in DT 6.2; but the changelog for v1.19 of this version does not say anything about DarkTrace versions.

We've also got the same problem with the ai analyst alerts, will this need a secondary ticket or can you pick this up from just this?

elastic / integrations