Getting rid of XML CDATA tags in message log

[NicolSpies]

Hi, I have a question,

• My message payloads is in JSON.
• I use a custom formatter to convert the JSON to XML and set the XML as new message payload.
• In the message log the XML payload is wrapped in CDATA tags.

How do I prevent the log4j logger to wrap the XML payload in the CDATA opening and closing tags <![CDATA[ and ]]>

[kamilfb]

The message log can be either in plain format or encoded to Base64. With plain, if it detects you've got some XML there, it wraps it because having XML within another XML is problematic and can cause problems. If you choose the encoded version, you won't see the CDATA tags. What's the reason you want to get rid of the CDATA tags?

[NicolSpies]

Hi Kamil, this is the normal message from the log with the data (payload) as received. I need to convert the payload from JSON to XML to be able to process the data as a total XML string.

<MqttMessage id="92" timestamp="1470901829770" topic="Oasis/Franchise/RSA/Gauteng/Pretoria/Central/Oos-Moot/plant1/1417474/filter">{"Vol":21.003,"sp":"2016-08-11T09:50:21Z","dp":{"1":60,"3":0,"2":0,"5":0,"4":0,"6":0},"st":"2016-08-11T09:46:30Z"}</MqttMessage>

This I do and in the formatter and the payload is displayed correctly: <payload><type>flow2</type><name>Oos-Moot</name><cfg3><monthly><monthly>17697.647</monthly><time>2016-08-11T10:00:00Z</time><deltp>0</deltp></monthly><daily><daily>427.996</daily><time>2016-08-11T10:00:00Z</time><deltp>0</deltp></daily><weekly><time>2016-08-11T10:00:00Z</time><weekly>6112.24</weekly><deltp>0</deltp></weekly><hourly><hourly>67.286</hourly><time>null</time><deltp>0</deltp></hourly></cfg3><customer>Oasis</customer><time>2016-08-11T10:30:02Z</time></payload>

but in the message log it looks as follows with the CDATA tags added. This cause a problem as it tells my extarnal XML post processor to ignore (not parse) the data within the tags even though it is proper XML. So if I manually remove the tags the whole XML string is valid and parses as required.

<MqttMessage id="104" timestamp="1470904204800" topic="Oasis/Franchise/RSA/Gauteng/Pretoria/Central/Oos-Moot/flow2/13587724/get"><![CDATA[<type>flow2</type><name>Oos-Moot</name><cfg3><monthly><monthly>17697.647</monthly><time>2016-08-11T10:00:00Z</time><deltp>0</deltp></monthly><daily><daily>427.996</daily><time>2016-08-11T10:00:00Z</time><deltp>0</deltp></daily><weekly><time>2016-08-11T10:00:00Z</time><weekly>6112.24</weekly><deltp>0</deltp></weekly><hourly><hourly>67.286</hourly><time>null</time><deltp>0</deltp></hourly></cfg3><customer>Oasis</customer><time>2016-08-11T10:30:02Z</time></payload>]]></MqttMessage>

PS. The logger appends the log.messages file with each message as a separate xml "document". This results that the file cannot be parsed as an xml file directly as no tag exists. So when I add and on top and at the end, it becomes a valid xml file. It would also be convenient if the logger can add this tags and add the xml logs in between to generate a valid xml messages log.

It appears to be related to the log4j configuration. This describes the similar problem from someone else https://bz.apache.org/bugzilla/show_bug.cgi?id=49471

[kamilfb]

Hi @NicolSpies. Writing a separate XML document per message is as designed. This is to allow for more efficient message replay or streaming of the log file. It's more efficient, because you only need to read individual lines to parse/process them, as oppose to a large XML file all in one go. This makes a huge difference with log files that contain thousands or millions of messages, or are quite big on the disk.

The CDATA tags are again for efficiency but also for additional protection - in case your XML is not 100% valid or you put some text with opening or closing XML tags that would invalidate the whole document.

Rather than using the auditing functionality, I'd recommend expanding your onMessage to simply append to a file of your choice with the format that you need.

[NicolSpies]

Hi @kamilfb. This was my original plan but I could not get the onMessage to trigger when a message is received. I accept your explanation and have studied the operation of the log4j logger and the reason for the way it is implemented and used.

I would very much like to use the onMessage (proper) way of implementing what I need.