WARNING:root:malformed node or string for json's file elements that take lists as values

[kf89]

Hello!I have a json file and each line of it seems like the folowing line

{"ip":"74.63.161.124","timestamp":"2015-10-16T14:00:13-04:00","data":{},"error":"EOF","error_component":"banner","location":[17.3753, 78.4744]}

The program exports that is an invalid file and the reason of that is that it does not accept the feature "location":[17.3753, 78.4744] which contains a list. Particularly, it exports:

WARNING:root:malformed node or string: <_ast.BinOp object at 0x0000029A2FFF7E80>

Moreover, when I do not validate the file, I cannot tranfer to ES the lines that "correspond" to these warnings.

It would be a good idea to be fixed as it's a valid json file.The error is because of json.loads command but I couldn't fix it.

[xros]

@kf89 Your JSON data is invalid. jsonpyes takes each piece of your data as a string from the very first beginning.

I wrote these codes to help you understand the valid data format for jsonpyes and explain why in Python.

>>> import json
>>> a = {"ip":"74.63.161.124","timestamp":"2015-10-16T14:00:13-04:00","data":{},"error":"EOF","error_component":"banner","location":[17.3753, 78.4744]}
>>> json.loads(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: expected string or buffer

>>> json.dumps(a)
'{"timestamp": "2015-10-16T14:00:13-04:00", "ip": "74.63.161.124", "error_component": "banner", "location": [17.3753, 78.4744], "error": "EOF", "data": {}}'

>>> json.loads(json.dumps(a))
>>> {u'ip': u'74.63.161.124', u'timestamp': u'2015-10-16T14:00:13-04:00', u'error_component': u'banner', u'location': [17.3753, 78.4744], u'error': u'EOF', u'data': {}}

Now with this tweak, you can export your data using jsonpyes correctly. All you have to do is

Add a sample ' at the beginning and at the end of each line of your data.

Such as '{"timestamp": "2015-10-16T14:00:13-04:00", "ip": "74.63.161.124", "error_component": "banner", "location": [17.3753, 78.4744], "error": "EOF", "data": {}}'