Difficulty exporting/importing text using non-Latin alphabets

[dwatteau]

See for example this report:
https://zenhax.com/viewtopic.php?t=14391&p=62976

Using ScummTR with non-Latin alphabets is quite inconvenient.

For example, to extract the text from a Japanese SCUMM game:

$ scummtr -b -g loomtowns -of jpn-tmp-bin.txt
$ LC_ALL=C tr '\000' '\n' jpn-tmp-bin.txt | iconv -c -f SHIFT-JIS -t UTF-8 > jpn.txt

But the binary output is not convenient, and we even need to call iconv with -c (probably because of the internal SCUMM escape sequences such as \255\001 etc.).

Using scummtr -H (for hexadecimal output) is maybe a bit better, if you then use a script which can interpret the hexadecimal SHIT-JIS codes. Still, it's not very convenient.

Possible solutions (from the easiest ones to the ones requiring more effort):

• document an acceptable way of dealing with non-Latin alphabets, as is;
• find a way to output a raw SHIFT-IJS files right from ScummTR while keeping internal SCUMM escape sequences (i.e. a mix between -b and the pseudo ASCII-US table for v1/v2 games);
• add a flag which processes everything as UTF-8.

[dwatteau]

The second solution has been implemented in PR #33.