JSON log message with "message":{"some-json":"value"} not indexed

[mrdavidlaing]

Works:

{"logger":"JSONPARSETEST","message":"A string"}
{"logger":"JSONPARSETEST","message":"A string", "message-data":{"key":"value"}}

Fails:

{"logger":"JSONPARSETEST","message":{"key":"value"}}

With related errors like:

[2014-02-06 13:49:49,039][DEBUG][action.bulk              ] [Grey King] [logstash-2014.02.06][0] failed to execute bulk item (index) index {[logstash-2014.02.06][json][mct8rvy9Tqy_qfqWWPVE9A], source[{"@version":"1","@timestamp":"2014-02-06T13:49:46.643+00:00","@type":"json","@environment":"DEV","@source.service":"AutoRequoteService","@source.host":"LON-WS01186","@source.path":"C:\\Dev\\logsearch-shipper.NET\\sample-data\\AutoRequoteService.log","@source.offset":"387250","@message":"{\"logger\":\"JSONPARSETEST\",\"message\":{\"key\":\"value\"}}","logger":"JSONPARSETEST","message":{"key":"value"}}]}

[dpb587]

Our logstash template requires that message is a string. I believe the behavior is working as expected and whatever is generating those logs needs to change so that message is a string. In my opinion, I don't think it makes sense to have message be an object anyway.

[mrdavidlaing]

Unfortunately changing the log format is going to be very difficult.

What options do we have to fix this on the logsearch side?

[dpb587]

I think we'll need to add conditional behavior in our logstash config which will rename message in our json filter if it detects it is a hash instead of a string. What would you prefer the field be renamed to, keeping in mind it won't be specific to the current logger: message_obj, message_, something else?

It's not easy (possible?) to change the mapping in elasticsearch to accommodate an object and a string for the same field; and logstash doesn't have a built-in type check for its conditionals. I think a ruby filter will be the best solution.

[mrdavidlaing]

Ok. Let's go for message_json

----- Reply message ----- From: "Danny Berger" notifications@github.com To: "cityindex/logsearch" logsearch@noreply.github.com Cc: "David Laing" david.laing@cityindex.co.uk Subject: [logsearch] JSON log message with "message":{"some-json":"value"} not indexed (#330) Date: Thu, Feb 6, 2014 16:11

I think we'll need to add conditional behavior in our logstash config which will rename message in our json filter if it detects it is a hash instead of a string. What would you prefer the field be renamed to, keeping in mind it won't be specific to the current logger: messageobj, message, something else?

It's not easy (possible?) to change the mapping in elasticsearch to accommodate an object and a string for the same field; and logstash doesn't have a built-in type check for its conditionals. I think a ruby filter will be the best solution.

— Reply to this email directly or view it on GitHub.

This email is intended only for the use of the individual or entity to which it is addressed and may contain information that is confidential, subject to copyright or which constitutes a trade secret. You should not copy it for any purpose or disclose its contents to any person. City Index Ltd has taken reasonable precautions to minimise the risk of transmitting software viruses but we advise you to carry out your own virus checks on any attachment to this message. We cannot accept any liability for any loss or damage resulting from any software viruses. If you receive this email in error, please telephone our postmaster on +44 20 7550 8500 or email postmaster@cityindex.co.uk and delete all copies on your system.

City Index Limited is authorised and regulated by the Financial Conduct Authority. FCA Register Number: 113942. Registered Office: Park House, 16 Finsbury Circus, London, EC2M 7EB. Registered in England and Wales, number: 1761813.

[dpb587]

Not to be disagreeable, but to me message_json suggests that it will be a string of JSON data, not a traversable object. Thoughts?

[dpb587]

message_data

[dpb587]

Merged message_data via 5875be5853