CPI: Certain characters (e.g. 'ä', 'ö') are mangled with DeepPlain2XMLConverter

kepatron commented 3 years ago

It seems that at least some characters (ä, Ä, ö, Ö) get mangled in DeepPlain2XMLConverter -conversion on CPI.

The scenario is from UTF-8 to UTF-8, so there should be no other sources of charset corruption.

The conversion script is simply;

import com.sap.gateway.ip.core.customdev.util.Message
import com.equalize.cpi.converter.FormatConversionBean

def Message processData(Message message) {
    def fcb = new FormatConversionBean(message.exchange, message.getProperties())
    def output = fcb.convert()

    message.setBody(output)
    return message
}

GitHub_FCB_Issue_00

"Yrjö Kyrölä" converts into "Yrj�� Kyr��l��"

Might be related to issue "Encoding of umlaut character corrupted during XML to JSON conversion #10"

kepatron commented 3 years ago

Please find attached an example iFlow that reproduces the issue as follows:

Format Conversion Bean Example.zip

GitHub_FCB_Issue_01

Message body:

K|äöåü|ÄÖÅÜ

engswee commented 3 years ago

Hi @kepatron, thanks for providing all the details. I managed to reproduce the issue and found out where was the bug. It wasn't particularly obvious when working locally as my environment is set to use UTF-8 for the encoding. But somehow, on the CPI tenant it uses a different default file encoding, and that caused the issue. I've updated the particular code so that is uses specific encoding, rather that relies on the platform default encoding.

The fix is available in release 2.2.1 (https://github.com/engswee/equalize-cpi-converter/releases/tag/2.2.1), so if you use that, you should not need a workaround on the IFlow.

engswee / equalize-cpi-converter

CPI: Certain characters (e.g. 'ä', 'ö') are mangled with DeepPlain2XMLConverter #8