[Netskope] Dropping event due to parsing error

[ShubhangiCrest02]

Error msg in logs

"log.level":"warn","@timestamp":"2022-04-08T10:22:34.247-0400","log.logger":"elasticsearch","log.origin":{"file.name":"elasticsearch/client.go","file.line":428},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.April, 8, 10, 20, 8, 945537222, time.Local), Meta:{\"raw_index\":\"logs-netskope.events-dev\"}, Fields:{\"agent\":{\"ephemeral_id\":\"212e9e66-2e4b-4a67-bfa8-9eeb70f81fc3\",\"id\":\"c812cc89-068e-4cb0-8ace-f186f60f156c\",\"name\":\"SECCTEPR01\",\"type\":\"filebeat\",\"version\":\"8.1.1\"},\"data_stream\":{\"dataset\":\"netskope.events\",\"namespace\":\"dev\",\"type\":\"logs\"},\"ecs\":{\"version\":\"8.0.0\"},\"elastic_agent\":{\"id\":\"c812cc89-068e-4cb0-8ace-f186f60f156c\",\"snapshot\":false,\"version\":\"8.1.1\"},\"event\":{\"dataset\":\"netskope.events\"},\"input\":{\"type\":\"tcp\"},\"log\":{\"source\":{\"address\":\"172.20.0.5:38754\"}},\"message\":\"{\\\"netskope\\\": {\\\"events\\\": {\\\"event_type\\\": \\\"page\\\", \\\"insertion\\\": {\\\"timestamp\\\": 1649182275}, \\\"access_method\\\": \\\"GRE\\\", \\\"app\\\": {\\\"category\\\": \\\"Streaming \\u0026 Downloadable Video\\\"}, \\\"category\\\": {\\\"name\\\": \\\"Streaming \\u0026 Downloadable Video\\\"}, \\\"cci\\\": \\\"0\\\", \\\"ccl\\\": \\\"unknown\\\", \\\"connection\\\": {\\\"id\\\": \\\"0\\\"}, \\\"count\\\": 1, \\\"device\\\": {\\\"type\\\": \\\"Other\\\"}, \\\"domain\\\": \\\"usea-1.api.microsoftstream.com\\\", \\\"organization_unit\\\": \\\"stores.rlseafood.com/MWS/Accounts\\\", \\\"page\\\": \\\"usea-1.api.microsoftstream.com\\\", \\\"site\\\": \\\"microsoftstream\\\", \\\"traffic\\\": {\\\"type\\\": \\\"Web\\\"}, \\\"type\\\": \\\"connection\\\", \\\"user\\\": {\\\"generated\\\": \\\"yes\\\", \\\"ip\\\": \\\"10.207.94.xx\\\"}, \\\"url\\\": \\\"usea-1.api.microsoftstream.com\\\", \\\"is_bypass_traffic\\\": \\\"yes\\\", \\\"transaction\\\": {\\\"id\\\": \\\"0\\\"}}}, \\\"event\\\": {\\\"id\\\": \\\"004bad0deade8dd33fafb916\\\"}, \\\"destination\\\": {\\\"geo\\\": {\\\"country_iso_code\\\": \\\"US\\\", \\\"location\\\": {\\\"lat\\\": 37.9273, \\\"lon\\\": -76.8545}, \\\"city_name\\\": \\\"Tappahannock\\\", \\\"region_name\\\": \\\"Virginia\\\", \\\"timezone\\\": \\\"America/New_York\\\", \\\"postal_code\\\": \\\"22560\\\"}, \\\"address\\\": \\\"40.76.22.xxx\\\", \\\"ip\\\": \\\"40.76.22.xxx\\\", \\\"port\\\": 443}, \\\"user_agent\\\": {\\\"os\\\": {\\\"name\\\": \\\"Windows NT 10.0\\\"}}, \\\"source\\\": {\\\"geo\\\": {\\\"country_iso_code\\\": \\\"US\\\", \\\"location\\\": {\\\"lat\\\": 28.9025, \\\"lon\\\": -81.248}, \\\"city_name\\\": \\\"Deltona\\\", \\\"region_name\\\": \\\"Florida\\\", \\\"timezone\\\": \\\"America/New_York\\\", \\\"postal_code\\\": \\\"32725\\\"}, \\\"address\\\": \\\"8.43.64.xxx\\\", \\\"ip\\\": \\\"8.43.64.xxx\\\"}, \\\"@timestamp\\\": \\\"2022-04-05T18:11:10.000Z\\\", \\\"user\\\": {\\\"email\\\": {\\\"1\\\": \\\"xx00640m@stores.xxxfood.com\\\", \\\"2\\\": \\\"xx100640M@stores.xxxxfood.com\\\", \\\"3\\\": \\\"xx100640M@stores.xxxxfood.com\\\"}}}\",\"tags\":[\"forwarded\",\"netskope-events\"]}, Private:interface {}(nil), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse field [netskope.events.user.generated] of type [boolean] in document with id 'X6qOCYABwqEQGRWoLDf6'. Preview of field's value: 'yes'\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Failed to parse value [yes] as only [true] or [false] are allowed.\"}}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"

---

Initial Analysis: As per our analysis for dropping event due to parsing error. Found out that it is related to pipeline for field "netskope.events.user.generated" which is currently accepting value True or False But as per the logs in logs.zip user is getting Value as yes or no resulting into parsing error causing event dropping. Possible solution for this is we need to recreate the pipeline again which will accept both values True or False and Yes or No. So, considering it as a future enhancement.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[andrewkroh]

Crest is working on this issue.