Invalid JSON output files

[dcj]

I'm using/running dump1090-fa I have a Python app that reads each aircraft.json file, the json.load function is triggered by an inotify watch, this works almost always, but occasionally the json.load throws an exception, presumably due to malformed JSON in the file. Here is a portion of the stack trace:

File "/usr/lib/python2.7/json/__init__.py", line 291, in load
**kw)
File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 380, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting , delimiter: line 138 column 4 (char 32767)

I am going to work around this by catching the exception and moving on. I will also try to capture an example file for additional debugging purposes...

[mutability]

Do you have an example of a malformed file? Maybe load the contents yourself and use json.loads on that, so you have the exact content that caused the failure

[dcj]

I have modified my code to both catch the exception, copy the offending file somewhere, and continue on, so I hope that next time this happens, I can provide a example malformed file. But this could take days. Stay tuned...

[dcj]

OK, got 11 invalid JSON files while I was out to dinner. The times:

Dec 31 02:59:42 
Dec 31 03:06:44 
Dec 31 03:39:38 
Dec 31 03:39:39 
Dec 31 03:50:23 
Dec 31 03:50:24 
Dec 31 03:50:25 
Dec 31 03:54:25 
Dec 31 03:55:08 
Dec 31 03:55:09 
Dec 31 03:55:10 

Here is one of those files:

aircraft.json.2018.12.31.03.55.10.txt

[mutability]

Hmm interesting. There is exactly 32k of good data followed by a few bytes of garbage and an early EOF. Looking into it. Given the 32k thing it's probably a bug in resizing the buffer used when 32k is hit.

edit: ok, I think I see the bug, it affects cases when there's between 32760-32767 bytes of aircraft data and then there's not quite enough space to fit the final line into the existing buffer

[mutability]

See if that commit helps?