mapper_parsing_exception, TCP flag parsing problem

[dexter-sec]

🐛 Summary

after install and build, everything running good except this error,

logstash_1 | [2022-09-01T03:04:53,615][WARN ][logstash.outputs.opensearch] Could not index event to OpenSearch. {:status=>400, :action=>["index", {:_id=>"220901-ZO7C1nd7-47Z7BUKiOEaOA", :_index=>"arkime_sessions3-220901", :routing=>nil}, {"network"=>{"packets"=>62, "type"=>"ipv4", "iana_number"=>["6"], "bytes"=>2804, "vlan"=>{"id"=>["1005"]}, "transport"=>["tcp"], "community_id"=>"1:ZfAjPAlzM8PYZFb+9nQ9dynDUUs=", "direction"=>"outbound"}, "protocol"=>["tcp"], "ipProtocol"=>6, "host"=>{"name"=>"IDS01"}, "tcpflags"=>{"rst"=>1000000000000000000000000, "syn-ack"=>1, "syn"=>1, "ack"=>1}, "destination"=>{"packets"=>52, "as"=>{"full"=>"AS42473 ANEXIA Internetdienstleistungs GmbH"}, "bytes"=>2120, "geo"=>{"location"=>{"lat"=>1.2929, "lon"=>103.8547}, "timezone"=>"Asia/Singapore", "country_code3"=>"SG", "country_name"=>"Singapore", "ip"=>"188.172.203.42", "continent_code"=>"AS", "postal_code"=>"18", "country_iso_code"=>"SG", "longitude"=>103.8547, "city_name"=>"Singapore", "country_code2"=>"SG", "latitude"=>1.2929}, "ip"=>"188.172.203.42", "mac"=>["00:00:5e:00:01:05"], "oui"=>"ICANN, IANA Department", "port"=>5938}, "srcOui"=>["Qbic Technology Co., Ltd"], "firstPacket"=>1662001268934, "dstOui"=>["ICANN, IANA Department"],"server"=>{"bytes"=>0}, "@timestamp"=>2022-09-01T03:01:08.934Z, "timestamp"=>1662001268934, "client"=>{"bytes"=>37}, "rootId"=>"CM00QEG59T273hhWa", "log"=>{"file"=>{"path"=>"conn.log"}}, "related"=>{"ip"=>["10.0.5.11", "188.172.203.42"], "mac"=>["c8:f5:d6:20:06:06", "00:00:5e:00:01:05"], "oui"=>["Qbic Technology Co., Ltd", "ICANN, IANA Department"]}, "totDataBytes"=>37, "event"=>{"risk_score"=>70.0, "severity"=>70, "category"=>["network"], "start"=>"1662001268934", "provider"=>"zeek", "risk_score_norm"=>70.0, "end"=>"1662001269015", "dataset"=>"conn", "ingested"=>2022-09-01T03:04:53.417Z, "severity_tags"=>["Connection aborted (responder)", "Outbound traffic"], "hash"=>"ZO7C1nd7-47Z7BUKiOEaOA", "id"=>["CM00QEG59T273hhWa"], "kind"=>"event", "duration"=>80664000}, "length"=>81, "input"=>{}, "agent"=>{"name"=>"IDS01", "hostname"=>"filebeat"}, "@version"=>"1", "node"=>"IDS01", "zeek"=>{"uid"=>"CM00QEG59T273hhWa", "conn"=>{"history"=>"ShADTrrrrrrrrrrrrrrrrrrrrrrrrr", "resp_pkts"=>"52", "local_resp"=>"F", "vlan"=>"1005", "conn_state"=>"RSTR", "orig_bytes"=>"37", "orig_ip_bytes"=>"684", "resp_ip_bytes"=>"2120", "orig_pkts"=>"10", "duration"=>"0.080664", "missed_bytes"=>"0", "local_orig"=>"T", "resp_bytes"=>"0", "conn_state_description"=>"Established, responder aborted"}, "ts"=>"2022-09-01T03:01:08.934Z"}, "tags"=>["internal_source", "external_destination"], "lastPacket"=>1662001269015, "ecs"=>{"version"=>"1.6.0"}, "source"=>{"packets"=>10, "bytes"=>684, "ip"=>"10.0.5.11", "mac"=>["c8:f5:d6:20:06:06"], "oui"=>"Qbic Technology Co., Ltd", "port"=>43366}}], :response=>{"index"=>{"_index"=>"arkime_sessions3-220901", "_id"=>"220901-ZO7C1nd7-47Z7BUKiOEaOA", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [tcpflags.rst] of type [long] in document with id '220901-ZO7C1nd7-47Z7BUKiOEaOA'. Preview of field's value: '1000000000000000000000000'", "caused_by"=>{"type"=>"input_coercion_exception", "reason"=>"Numeric value (1000000000000000000000000) out of range of long (-9223372036854775808 - 9223372036854775807)\n at [Source: (byte[])\"{\"network\":{\"packets\":62,\"type\":\"ipv4\",\"iana_number\":[\"6\"],\"bytes\":2804,\"vlan\":{\"id\":[\"1005\"]},\"transport\":[\"tcp\"],\"community_id\":\"1:ZfAjPAlzM8PYZFb+9nQ9dynDUUs=\",\"direction\":\"outbound\"},\"protocol\":[\"tcp\"],\"ipProtocol\":6,\"host\":{\"name\":\"IDS01\"},\"tcpflags\":{\"rst\":1000000000000000000000000,\"syn-ack\":1,\"syn\":1,\"ack\":1},\"destination\":{\"packets\":52,\"as\":{\"full\":\"AS42473 ANEXIA Internetdienstleistungs GmbH\"},\"bytes\":2120,\"geo\":{\"location\":{\"lat\":1.2929,\"lon\":103.8547},\"timezone\":\"Asia/Singapore\",\"coun\"[truncated 1859 bytes]; line: 1, column: 289]"}}}}}

i have no idea, why [tcpflags.rst] set and parse '1000000000000000000000000' how do i change [tcpflags.rst] type and any idea handle this error? plz help me :(

[mmguero]

Interesting... the conn.log history value is "history"=>"ShADTrrrrrrrrrrrrrrrrrrrrrrrrr" which causes the calculation for the tcpflags value to overflow. I will cap it, expect a commit for the next release.

[mmguero]

Closed as fixed in upcoming v6.3.0 release