UTF8 Issue in Ubuntu 16.10

[MaxPerl]

Dear Mr Karasiak, Thank you for your work. The prima GUI toolkit is incredible fast! I noticed an (I think) issue with UTF8 on Linux. Perhaps I dont understand the usage of Prima at the moment not good enough, but in the following basic code the umlauts are shown wrongly:

use utf8;

use Prima;
use Prima::Application;

my $mw = Prima::MainWindow->new(text => "Halölä");
run Prima;

I am on Linux and the system encoding is utf8.

Thanks in advance, Max

[dk]

Hi,

It looks like that the problem lies elsewhere. Could you try the example with these adjustments possibly?

`` use utf8; use Encode;

use Prima; use Prima::Application;

my $text = "Halölä";

if ( Encode::is_utf8($text)) { print "is utf8 already\n"; } else { print "not utf8, converting\n"; $text = Encode::decode( 'utf-8', $text); } my $mw = Prima::MainWindow->new(text => $text);

run Prima;

[MaxPerl]

Hello, Thank you very much for your fast answer! Unfortunately the hack doesn't work for me. In the terminal I got the output "ist utf8 already", but in the window title the umlauts are wrong.. Best wishes, Max

[dk]

Hi,

Indeed I tried it on Ubuntu 16 and I could reproduce the problem ( I couldn't on my non-Ubuntu setup ).

I think I fixed it now, please try and see if it works for you

Regards, Dmitry

[MaxPerl]

Dear Dimitry, Thank you for your commit. I just cloned the actual github code and tried the fix. Unfortunately the problem is still there (the signs changed, but still the umlauts are not shown correctely..) Thanks for your struggle, Max

[dk]

Hi Max,

This becomes even more interesting! I'd like to ask you to post some debug information then.

First we'll need the X properties dump for Prima. I'll ask you to run the example in one window, and in another, run this command: "LANG=C xprop" . This command will turn your pointer into a crosshair cursor, and you'll need to point it to the Prima window and click on it. Then it will dump lots of information; I'll need only these four:

WM_NAME(UTF8_STRING) = "Hal\366l\344" WM_ICON_NAME(UTF8_STRING) = "Hal\366l\344" _NET_WM_ICON_NAME(UTF8_STRING) = "Hal\303\266l\303\244" _NET_WM_NAME(UTF8_STRING) = "Hal\303\266l\303\244"

Next I'll need to ask for your perl version (from "perl -v"), and "echo $LANG" .

[dk]

(and in addition to that, just to be sure, I'd like to ask you this too):

LANG=C xprop -f WM_NAME 8s | grep WM_NAME

(this will not try to decode UTF8)

[MaxPerl]

Hello Dimity, Here is the output of xprop:

_NET_WM_ICON_NAME(UTF8_STRING) = "Halölä" _NET_WM_NAME(UTF8_STRING) = "Halölä" WM_NAME(UTF8_STRING) = "Halölä" WM_ICON_NAME(UTF8_STRING) = "Halölä"

[MaxPerl]

perl -v:

This is perl 5, version 22, subversion 2 (v5.22.2) built for x86_64-linux-gnu-thread-multi (with 67 registered patches, see perl -V for more detail)

Copyright 1987-2015, Larry Wall

Perl may be copied only under the terms of either the Artistic License or the GNU General Public License, which may be found in the Perl 5 source kit.

Complete documentation for Perl, including FAQ lists, should be found on this system using "man perl" or "perldoc perl". If you have access to the Internet, point your browser at http://www.perl.org/, the Perl Home Page.

[MaxPerl]

echo $LANG:

de_DE.UTF-8

[MaxPerl]

And last LANG=C xprop -f WM_NAME 8s | grep WM_NAME

_NET_WM_NAME(UTF8_STRING) = "Halölä" WM_NAME(UTF8_STRING) = "Hal\303\266l\303\244"

[MaxPerl]

Everything is created with your adjustment code above. In the terminal I geht as said the output "is utf8 already)..

I find it great, how fast you answer and try help me. This is really wonderful...

[dk]

I'm a bit lost, because everything looks quite correct. Could you tell what is the window manager you're running on ubuntu - is that the default one, or you have replaced it to something else? Also, could you possibly post the screenshot of the wrong umlauts?

[dk]

... and I also have another idea: could you run this command:

xterm -title 'Halölä'

and try the same trick with xprop on it? This might not actually display the correct text in the xterm title, but if it does, I wonder what the corresponding WM_NAME looks like.

[MaxPerl]

Enclosed you find the screenshot of the wrong umlauts

The xterm -title does not work. Title does not change with this option...

I use Ubuntu GNOME. I think my window manager is X. There is an option in the gdm with "GNOME under wayland". But also under wayland I have the same issue. I keep trying the xterm thing..

[MaxPerl]

I have the suspicion that it is a font problem. I have added a label with "Halölä". First only "Hall" was shown. With the gnome tweak tool I changed the font (from Canterell to something different). Now in the label the whole text is shown (although it is not the font I selected!). Unfortunately now I cannot change any more to the issue :-) I don't know how I can undo the gnome tweak tool setting ;-)

So Perhaps it correlates to https://ubuntuforums.org/showthread.php?t=2322753 ??

The title is still wrong. Changing font doesn't work here.

Update: This is very curious with the label: I found "Reset to the defaults" in GNOME tweak tools. But even if I reseted, the label shows the umlauts correctly (but the font is very thin, but perhaps this is normal with my resolution?)

[MaxPerl]

Update: I tested on another machine with Mageia 5.1 First with the CPAN release: The umlauts in the title are shown incorrect. The umlauts in the label was not shown (i.e. "Hall" instead of "Halölä"). I additional had the problem, that the label was only visible, after resizing the window (but this is another problem),

Then I tested the Github patch: Now the umlauts in the title are correct!!! Yeah! The umlauts in the label are still nit visible, same as in the CPAN release..

PS.: When I tried examples/label.pl from the CPAN I got the error "DGifOpen: undefined symbol" (perhaps later I will open an own bug report).

[dk]

Hi Max,

Here's what I did. I have a freshly installed Ubuntu 16, with updates and all, and I changed the default locale to de_DE.UTF-8 . Then I run Prima from the latest github snapshot on it. And it all works just fine. Here's the screenshot:

It seems like there's something goes on in your installation that I cannot reproduce. I wonder if it would be possible to duplicate the issue so I could try to reproduce it.

[MaxPerl]

Did you also tried a label with umlauts? As said in my Mageia the patch worked for the title, too..

My ubuntu was preinstalled from the manufacture. When Mageia 6 is out I want to switch to Mageia. I saw that there are some custom packages from tuxedo, e.g. libdrm-stuff libdrm2 libegl-mesa libgbm1 libgl1-mesa-stuff libglapi-mesa libllvm libosmesa6 libwayland-egl1-mesa mesa-vdpau-drivers

It was an automatical installation over network. But the manufacture offers here an adjustment script, too But after a first scan, I don't see anything relevant for our problem...

[dk]

I'm actually was a bit confused by "label", what is the label you're referring to? I'd try ti out if I understand what it is

[MaxPerl]

Sorry, I didn't explain it. I just insert the following for testing of a label. And there the umlauts are not shown under Mageia (under ubuntu they are after changing the fonts): Perhaps it is a new bug?

my $label = Prima::Label->create(
        text => "Halölä Peter, daß ist ja schön ÖÖÖÖÖÖÖÖÖÖÖÖÖÖ ÄÄÄÄÄÄÄÄÄÄÄÄÄä",
        owner => $mw,
        alignment => ta::Center,
        );

[dk]

Hmm no, the text inside Prima window is managed strictly by Prima itself, and I don't think is a subject to OS bugs. This functionality was introduced long time ago, and I can't see any bugs with it. It might looks like a bad font actually (or bad fontconfig setup, but I doubt it).

Could you run one of standard Prima examples, f.ex. example/fontdlg.pl and see if the font you're using is okay. Or examples/keys.pl and try to change fonts and press umlauted keys, to see what happens?

[dk]

(and, if you install additionally Text::Bidi, there's another example in examples/bidi.pl that is actually a showcase for all things unicode - so if it doesn't work, that's some serious trouble)

[MaxPerl]

Dear Dimitry,

The problem with the label occurs only under Mageia. Under Ubuntu switching font solved the problem (see above). After reseting the old settings the problem did not reappear once again.

Unfortunately under Mageia the examples does not work because of undefined symbol: DGifOpen (see the new issue).

I am tonight again on my laptop and can test it again.

Best wishes, Max

[MaxPerl]

PS. Enclosed you find a screenshot under Mageia with th label and the wrong umlauts (actual github version of Prima)

[dk]

Yep, that looks very much like a bad font. You can try this:

• run examples/keys.pl
• go to menu, choose "set font"
• double click on the AaBbCc text sample and enter text with umlauts
• try to select a good font and see if it worls

[dk]

I just tried Mageia 5.1 and I can reproduce the error. Indeed Prima is picking up a suboptimal luxi-sans font, that has more problems than just umlauts. However I managed to easily fix it by installing libgtk+2.0-devel package and recompiling Prima - and after that all was just fine.

[MaxPerl]

Yeah, installing libgtk-devel solved the problem. Perhaps one could do some hint on CPAN at the FAQ or something similar. For Prima beginners as me this would be very helpful. Or would it be possible to use another font by default which doesn't have so much problems? Installing libgtk2-devel is really a huge improvement! I don't know whether we should close the bug now. On my ubuntu machine the tilte is still wrong, but on a fresh ubuntu installation, there is no problem. So I think it is a problem with my installation. Furthermore I don't know how I can help to find where the mistake in my setup is. But if the problem only affects a single installation, then this is not really a problem I think...

[dk]

Hi Max,

Good idea, I think I'll write a couple of lines about GTK. It is optional for Prima, but I see how not having it raises issues, and the look will be different under standard environments.

As for the default font in non-GTK version, that's an issue, because there is no such thing as default font in X. All old-school X applications, Prima included, require manual font configuration ( which is not really relevant for this issue, because we're discussing the automatic selection ). So Prima does the best job trying to look for some generic usable font (i.e., non-serif, scalable, with good resolution etc). And Luxi is actually just that type of a font, except there's something wrong with umlauts (and more).

So I'd also like to keep the issue a bit, because I want to take a closer look at Luxi, whether it is the font itself that sucks (in that case I'll probably handicap it), or it is Prima that does something bad with it.

BTW good to know that it works on your new Ubuntu. If you could find out how I can reproduce the problem, please let me know - because so far I tried Ubuntu 12, 14, and two different 16s, and everything is just fine.

Regards, Dmitry

[dk]

I think I found the problem. Before utf8, one would set a locale like de_DE.ISO-8859-1, and the corresponding font locale was too, iso8859-1. But for de_DE.UTF-8 locale, there are no utf-8 fonts, but there are iso10646-1 ones. This patch will detect that and will use them by default.

So I believe this is fixed too

[MaxPerl]

Unfortunately on my setup the umlauts in window title are still wrong. Sorry to say that...

[dk]

Sorry I wasn't specific. I addressed the "label" issue for setup without gtk

[MaxPerl]

ah, wonderful...

[dk]

Hi Max,

I have an idea about umlauts not showing in title on your Ubuntu. If xterm doesn't work , could you possibly find any application with umlauts in title, and try the xprop trick on it? I'd be interested in what is the full output would be, but mostly of course in WM_NAME.

Dmitry

[MaxPerl]

Hi Dimitry, It is obviously a problem with my system setup, not with Prima. This is good :-) In xpdf a title with umlauts is wrongly displayed, too. I opened a file with the name "HalöläßßßÄÄÖÖÜÜ.pdf" and here is the relevant output of xprop:

WM_ICON_NAME(STRING) = "Xpdf: ./HalöläÃ\302\237Ã\302\237Ã\302\237Ã\302\204Ã\302\204Ã\302\226Ã\302\226Ã\302\234Ã\302\234.pdf" WM_NAME(STRING) = "Xpdf: ./HalöläÃ\302\237Ã\302\237Ã\302\237Ã\302\204Ã\302\204Ã\302\226Ã\302\226Ã\302\234Ã\302\234.pdf"

[MaxPerl]

with Halölä.pdf opened with xpdf the full output is:

_NET_WM_STATE(ATOM) = WM_STATE(WM_STATE): window state: Normal icon window: 0x0 _NET_WM_DESKTOP(CARDINAL) = 0 _NET_FRAME_EXTENTS(CARDINAL) = 0, 0, 37, 0 _NET_WM_ALLOWED_ACTIONS(ATOM) = _NET_WM_ACTION_MOVE, _NET_WM_ACTION_RESIZE, _NET_WM_ACTION_FULLSCREEN, _NET_WM_ACTION_MINIMIZE, _NET_WM_ACTION_SHADE, _NET_WM_ACTION_MAXIMIZE_HORZ, _NET_WM_ACTION_MAXIMIZE_VERT, _NET_WM_ACTION_CHANGE_DESKTOP, _NET_WM_ACTION_CLOSE, _NET_WM_ACTION_ABOVE, _NET_WM_ACTION_BELOW _MOTIF_WM_MESSAGES(ATOM) = _MOTIF_WM_OFFSET WM_PROTOCOLS(ATOM): protocols _MOTIF_WM_MESSAGES, WM_DELETE_WINDOW WM_LOCALE_NAME(STRING) = "C" WM_CLASS(STRING) = "win", "Xpdf" WM_HINTS(WM_HINTS): Client accepts input or input focus: True Initial state is Normal State. WM_NORMAL_HINTS(WM_SIZE_HINTS): program specified location: 0, 0 program specified size: 782 by 1036 program specified minimum size: 100 by 100 program specified base size: 0 by 0 window gravity: NorthWest WM_CLIENT_MACHINE(STRING) = "maximilian-N24-25BU" WM_ICON_NAME(STRING) = "Xpdf: ./Halölä.pdf" WM_NAME(STRING) = "Xpdf: ./Halölä.pdf"

[dk]

It might be because your default system locale is not UTF8 (I think it is in /etc/default/locale on ubuntu), while your own locale is. But it seems that this problem is resolved then?

[dk]

If the issue is resolved, I'm closing the ticket then.