Add terminal encoding option

[deuill]

By default, VTE initializes a UTF8 terminal (which is specifically instantiated here as such via the VTE_ENCODING constant). However, not all applications are UTF8-aware, and some rely on hacks or alternative encodings in order to display certain characters (cough cough Nethack).

This commit adds the required hooks and documentation for allowing us to change the terminal encoding scheme to something other than UTF8, in an effort to cope with these broken applications.

The option character t was the closest thing to 'terminal encoding' I could find.

[deuill]

Noted that I've tested this on my computer, and it seems to work as well as one might hope, but I have no idea how to cross-compile for the Kindle.

This will allow us, however, to get Nethack working with some sweet TTY font tiles (it already works OK with DECgraphics, IBMgraphics need ISO encodings, I believe).

[bfabiszewski]

Thanks for the contribution! Everything seems ok. Let's check if it works on kindle. I built a version with your patch. Could you please test if it works as expected? kterm-kindle-2.4.zip

[deuill]

@bfabiszewski thanks! The build itself works, but changing encodings fails with this error. I'll test some more and see if this is due to missing locales or anything like that.

[bfabiszewski]

When browsing source, remember that on Kindle we use Vte-28.2 (newer versions require GTK+3). Your error leads to this line. Probably libiconv on Kindle can't handle requested encodings. Did you try to do the conversion between these encodings on command line, using Kindle's iconv utility?

[deuill]

Indeed, and it appears that iconv on the Kindle fails with the same error:

iconv: conversion to `ISO-8859-1` is not supported

Running locale -a gives me a single result:

POSIX

Which leads me to believe that glibc hasn't been set up with locales other than POSIX (not even C?). I'll look into how to enable additional locales (but this is probably starting to get outside of the realm of what kterm should do).

[bfabiszewski]

Which Kindle model do you have? What iconv -l lists on you Kindle? On my old Touch it gives pretty long list, including ISO_8859-1. I didn't try your patch on my device yet.

[deuill]

I've got the 3rd Gen Paperwhite, running on glibc 2.20. Running iconv -l gives the full list, but I believe iconv get derives its transliteration rules from installed locales. It's been a while since I've touched this kind of stuff, but localedef is installed so if anything, additional locales can be installed if need be.

[bfabiszewski]

I tested your build on my Kindle. It runs fine on KT. No error with ISO-8859-1 encoding. And UTF-8 encoded files now look scrambled, so it seems to work.

Locales on my system are:

C
de_DE.utf8
en_GB.utf8
en_US.utf8
es_ES.utf8
fr_FR.utf8
it_IT.utf8
ja_JP.utf8
POSIX
pt_BR.utf8
ru_RU.utf8
sl_SI.utf8
sq_AL.utf8
tr_TR.utf8
zh_CN.utf8
zh_HK.utf8
zh_TW.utf8

I am going to merge it now and I hope you will find a way to use it on your device. Let me know if you make any progress. 👍

[deuill]

Thanks @bfabiszewski! I'm sure I'll find a way to fix. Onwards!

[mergen3107]

Hi! Thank you for kterm! Can you please tell me how to change encodings properly? I tried in command line: kterm -t cp1251 but it says "(kterm:30384): Vte-0.0-WARNING **: Unable to convert characters from UTF-8 to cp1251." I also tried to change encoding line in the /bin/kterm.conf, no luck.

The problem is that Cyrillic characters are shown as squares in kterm (e.g. in ls command). I have Kindle PW3, firmware 5.9.7

Thank you!

[deuill]

You'll only be able to use encodings that have been built into glibc -- the full list can be had by running iconv -l. You can install additional locales using localedef, which I seem to remember is available, but you might need to find a guide on how to do that for the Kindle (probably against a Debian package for the locale and ARM architecture).

[mergen3107]

Thank you! :)

[mergen3107]

I still can't make kterm to use any other encoding. Here is what I did: 1) iconv -l gives a long list of encodings (iconv -l.txt), among which there is CP1251. I tried using encoding = CP1251 in the kterm/bin/kterm.conf, and launching kterm.sh. It still gives me the same error as above.

2) Moreover, if I leave the encoding paramater as a default UTF-8, it now says:

(kterm:6569): Vte-0.0-WARNING : Unable to convert characters from UTF-8 to Monospace. Monospace is the font family name written in the same conf file. If I replace Monospace by another font name (e.g. Bookerly), it is used in kterm, but terminal still says similar error. If I comment out the font selection, then it says (kterm:6626): Vte-0.0-WARNING : Unable to convert characters from UTF-8 to /mnt/us/extensions/kterm/bin.

3) locale -a command shows me (locale -a.txt) all available locales, among which there is ru_RU.utf8 . So it looks like I don't need to install new locales, since I wanted to see in kterm path names with Cyrillic characters.

4) locale shows that current locale is en_US.utf8 (locale.txt)

Is there anything I can do else?

[bfabiszewski]

I can't help you. I only tested character conversion once on old Kindle Touch and with very simple test case. Never really used this feature. Also remember that in order to properly display characters you also need properly encoded font.

Regarding your strange issues with config file I now discovered looking into source code that encoding must actually be quoted in the config