chris-counteractive
commented
4 years ago Madhu - thanks for the question! Elasticsearch is failing to index the ModifiedProperties field because it's an object and elasticsearch expects a string. We discussed this and pushed a work-around in issue #15, implemented as a processor in your config file. Unfortunately there's another inconvenient bug where these important processors in the config file get shadowed by another default processor declaration (see issue #9). Luckily the fix is straightforward.
My guess is that your config file (o365beat.yml) still has an un-commented section that looks like this:
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~If so, you'll have to delete that or merge those two processors into the "real" processors section, which should look something like this (note the ModifiedProperties processor at the end of the list):
processors:
- dissect:
field: ClientIP
tokenizer: '[%{clientip}]:%{clientport}'
when:
contains:
ClientIP: '['
- convert:
fields:
- {from: Id, to: 'event.id', type: string} # ecs core
- {from: RecordType, to: 'event.code', type: string} # ecs extended
- {from: Operation, to: 'event.action', type: string} # ecs core
- {from: OrganizationId, to: 'cloud.account.id', type: string} # ecs extended
- {from: Workload, to: 'event.category', type: string} # ecs core
- {from: ResultStatus, to: 'event.outcome', type: string} # ecs extended
- {from: UserId, to: 'user.id', type: string} # ecs core
- {from: ClientIP, to: 'client.ip', type: ip} # ecs core
- {from: 'dissect.clientip', to: 'client.ip', type: ip} # ecs core
# the following fields use the challenging array-of-name-value-pairs format
# converting them to strings fixes issues in elastic, eases non-script parsing
# easier to rehydrate into arrays from strings than vice versa:
- {from: Parameters, type: string} # no ecs mapping
- {from: ExtendedProperties, type: string} # no ecs mapping
- {from: ModifiedProperties, type: string} # no ecs mapping
ignore_missing: true
fail_on_error: false
mode: copy # defaultThat should ship the ModifiedProperties field as a string, and elasticsearch will be happy. If this fixes your error, please let me know so I can close this issue, otherwise we can explore other reasons this might be happening. You can also reach me at chris@counteractive.net if you want to share more details about your config outside a public issue.
HI,
When I start o365beat I below error and I am not able view events in Kibana discover. Can you please let me know the reason for this issue and help me with solution?
2019-12-03T12:05:01.328Z WARN elasticsearch/client.go:535 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x0, ext:63710376320, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"Actor":[]interface {}{common.MapStr{"ID":"Device Registration Service", "Type":1}, common.MapStr{"ID":"01cb2876-7ebd-4aa4-9cc9-d28bd4d359a9", "Type":2}, common.MapStr{"ID":"ServicePrincipal_d7922889-6244-40de-9aa2-0c60ab7e7ae9", "Type":2}, common.MapStr{"ID":"d7922889-6244-40de-9aa2-0c60ab7e7ae9", "Type":2}, common.MapStr{"ID":"ServicePrincipal", "Type":2}}, "ActorContextId":"2ab446d6-6d34-4e72-a643-1634ef0758fd", "ActorIpAddress":"", "AzureActiveDirectoryEventType":1, "ClientIP":"", "CreationTime":"2019-11-26T14:45:20", "ExtendedProperties":[]interface {}{common.MapStr{"Name":"resultType", "Value":"Success"}, common.MapStr{"Name":"auditEventCategory", "Value":"Other"}, common.MapStr{"Name":"nCloud", "Value":""}, common.MapStr{"Name":"actorContextId", "Value":"2ab446d6-6d34-4e72-a643-1634ef0758fd"}, common.MapStr{"Name":"actorObjectId", "Value":"d7922889-6244-40de-9aa2-0c60ab7e7ae9"}, common.MapStr{"Name":"actorObjectClass", "Value":"ServicePrincipal"}, common.MapStr{"Name":"teamName", "Value":"MSODS."}, common.MapStr{"Name":"targetContextId", "Value":"2ab446d6-6d34-4e72-a643-1634ef0758fd"}, common.MapStr{"Name":"targetObjectId", "Value":"7d88f04c-05db-457a-b71e-d570a9d27bfe"}, common.MapStr{"Name":"extendedAuditEventCategory", "Value":"Device"}, common.MapStr{"Name":"targetName", "Value":"LAPTOP-G1RJLVQA"}, common.MapStr{"Name":"targetIncludedUpdatedProperties", "Value":"[]"}, common.MapStr{"Name":"correlationId", "Value":"fe1c3106-30b0-462a-a85e-eeb44b657e6d"}, common.MapStr{"Name":"version", "Value":"2"}, common.MapStr{"Name":"additionalDetails", "Value":"{}"}, common.MapStr{"Name":"env_ver", "Value":"2.1"}, common.MapStr{"Name":"env_name", "Value":"#Ifx.AuditSchema#IfxMsods.AuditCommonEvent"}, common.MapStr{"Name":"env_time", "Value":"2019-11-26T14:45:20.6592929Z"}, common.MapStr{"Name":"env_epoch", "Value":"7CQ0J"}, common.MapStr{"Name":"env_seqNum", "Value":"57710097"}, common.MapStr{"Name":"env_popSample", "Value":"0"}, common.MapStr{"Name":"env_iKey", "Value":"ikey"}, common.MapStr{"Name":"env_flags", "Value":"257"}, common.MapStr{"Name":"env_cv", "Value":"##01431934-caf7-42e8-ab93-ccdacdedcfab_00000000-0000-0000-0000-000000000000_01431934-caf7-42e8-ab93-ccdacdedcfab"}, common.MapStr{"Name":"env_os", "Value":""}, common.MapStr{"Name":"env_osVer", "Value":""}, common.MapStr{"Name":"env_appId", "Value":"restdirectoryservice"}, common.MapStr{"Name":"env_appVer", "Value":"1.0.11379.0"}, common.MapStr{"Name":"env_cloud_ver", "Value":"1.0"}, common.MapStr{"Name":"env_cloud_name", "Value":"MSO-BL2"}, common.MapStr{"Name":"env_cloud_role", "Value":"restdirectoryservice"}, common.MapStr{"Name":"env_cloud_roleVer", "Value":"1.0.11379.0"}, common.MapStr{"Name":"env_cloud_roleInstance", "Value":"BL2RDSR575"}, common.MapStr{"Name":"env_cloud_environment", "Value":"PROD"}, common.MapStr{"Name":"env_cloud_deploymentUnit", "Value":"R5"}}, "Id":"7a8bd47b-6492-4510-ab7a-0e2744499923", "ModifiedProperties":[]interface {}{common.MapStr{"Name":"Included Updated Properties", "NewValue":"", "OldValue":""}}, "ObjectId":"Not Available", "Operation":"Update device.", "OrganizationId":"2ab446d6-6d34-4e72-a643-1634ef0758fd", "RecordType":8, "ResultStatus":"Success", "SupportTicketId":"", "Target":[]interface {}{common.MapStr{"ID":"Device_7d88f04c-05db-457a-b71e-d570a9d27bfe", "Type":2}, common.MapStr{"ID":"7d88f04c-05db-457a-b71e-d570a9d27bfe", "Type":2}, common.MapStr{"ID":"Device", "Type":2}, common.MapStr{"ID":"LAPTOP-G1RJLVQA", "Type":1}}, "TargetContextId":"2ab446d6-6d34-4e72-a643-1634ef0758fd", "UserId":"ServicePrincipal_d7922889-6244-40de-9aa2-0c60ab7e7ae9", "UserKey":"Not Available", "UserType":4, "Version":1, "Workload":"AzureActiveDirectory", "agent":common.MapStr{"ephemeral_id":"dba1b10f-549e-422e-a6a8-ed123b4fb8b9", "hostname":"ip-1-0-0-171.ap-south-1.compute.internal", "id":"2b1c5393-9866-4e26-85f2-184ae5c17acf", "type":"o365beat", "version":"1.4.3"}, "cloud":common.MapStr{"account":common.MapStr{"id":"544851249924"}, "availability_zone":"ap-south-1c", "image":common.MapStr{"id":"ami-0ce933e2ae91880d3"}, "instance":common.MapStr{"id":"i-08a56a13d34ef21d3"}, "machine":common.MapStr{"type":"t3a.medium"}, "provider":"aws", "region":"ap-south-1"}, "ecs":common.MapStr{"version":"1.1.0"}, "host":common.MapStr{"architecture":"x86_64", "containerized":false, "hostname":"ip-1-0-0-171.ap-south-1.compute.internal", "id":"ec2eb2a972ed3b680c44e709588d1e20", "name":"ip-1-0-0-171.ap-south-1.compute.internal", "os":common.MapStr{"codename":"Karoo", "family":"redhat", "kernel":"4.14.152-127.182.amzn2.x86_64", "name":"Amazon Linux", "platform":"amzn", "version":"2"}}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x0} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [ModifiedProperties] of type [keyword] in document with id 'u82my24Bg0YgRrVHQvjp'. Preview of field's value: '{OldValue=, NewValue=, Name=Included Updated Properties}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:1643"}}
Many Thanks, Madhu