search

kosho / flight-track

Logstash config and Kibana dashboard to visualize ADS-B signals. Your own receiver with dump1090 is required.
48 stars 7 forks source link

Not parsing correctly with dump1090-fa #6

Open donslice opened 6 years ago

donslice commented 6 years ago

I've gotten this set up, pointing to my dump1090-fa instance. Unfortunately, it does not have data.json, but does have aircraft.json. The included logstash parser does not parse this correctly. Was wondering if I could get some help?

aircraft.json output attached.

aircraft.txt

donslice commented 6 years ago

Also, some parsing errors occur with "altitude":"ground", as "ground" is not an integer.

[2018-04-26T23:15:36,651][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"flight-track-2018", :_type=>"doc", :_routing=>nil}, #], :response=>{"index"=>{"_index"=>"flight-track-2018", "_type"=>"doc", "_id"=>"e8E9BGMBEEpYgM88nMFH", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [aircraft.altitude]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"For input string: \"ground\""}}}}}

donslice commented 6 years ago

Ok... After messing with it some more, I changed the codec from "json" to "json_lines" and the data began parsing mostly correct. I still have some warns, though.

Example:

[2018-04-27T04:12:44,004][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (String)"}"; line: 1, column: 0]), at [Source: (String)"}"; line: 1, column: 2]>, :data=>"}"}, at [Source: (String)" ]"; line: 1, column: 4]>, :data=>" ]"}, [2018-04-27T04:12:44,004][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected close marker ']': expected '}' (for root starting at [Source: (String)" ]"; line: 1, column: 0]), [2018-04-27T04:12:43,995][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>" \"aircraft\" : ["}

SomKen commented 5 years ago

@donslice Sorry for the @ then a delete...

https://github.com/SomKen/flight-track

I've created a fork to fix the issues with this. It's a total hack, due to using the exec input in logstash to run curl and pipe into jq, but it got the job done.

If someone who knows how to configure logstash wants to make a PR on my fork, I'll be more than happy to merge. Based on @kosho 's creds, I'm sure he'll be more than able to fix. (notice me senpai)