Windows Region & Language-Format causes horizontally flipped widgets

[kdschlosser]

Windows 10 & Windows 7 SP1 x64 wxPython 4.0.3 & wxPython 3.0.2, gotten from pypi Python 2.7.15 & Python 3.5.4, Both are Stackless builds

How to replicate the issue Start -> Control Panel -> Region & Language In the Formats tab select Hebrew in the Format drop down. Click Apply

Launch any Python script that uses wxPython >= 3.0.2 and creates a Frame. if you have any kind of text control, the text gets right justified even if the language is set to a language that is not read from right to left. and all of the controls seem to get flipped horizontally.

[RobinD42]

It's not behaving that way for me. Are there any other settings that may be contributing to it on your system?

[kdschlosser]

This is the setting I am changing. I am changing from English (United States) to Hebrew (Israel) I am making no other change. Just the one.

This is what my application looks like in English

And this is what happens when I make that setting change.

Now the text s not so much of an issue. as Hebrew is a language that is read from right to left. So the text being right justified would be expected behavior. The whole window being flipped is not.. the close/max/min buttons are on the left now. the 2 panes also got flipped.. the icons are on the wrong sides.

t effects all of the controls. You can see that here in the drop down menu.

Now I did a Test and opened up Internet Explorer.. to make sure this was not some kind of an odd Windows glitch.

You can see here the right justified text. But the controls are all in the correct place.

I find it strange even tho the language is still set to English that it right justifies the text. But this is the behavior in IE as well, So I am going to assume this is expected behavior. But the whole window getting mirrored I would imagine is not supposed to happen.

This is not an isolated incident. There are users of this software elsewhere in the world that have had the same behavior. So the setup of the PC can be removed from the equation. as well as the operating system version (Windows that is) because this issue has been seen on Windows from 7 to 10

In our software there are no changes of any locale settings for wxPython. we only detect the current language in order to load translation files and pass the unicode strings into the controls. Anything further then that is handled by wxPython internally.

[RobinD42]

I'm not sure I would compare behaviors with IE, since it is not using a standard native Frame and everything (more or less) in the window is custom drawn.

If other applications which are using legacy stype stock native frames keep the window decorations on the normal side then one idea that comes to mind is to explicitly set the frame to LTR with SetLayoutDirection, and then set the top-level child within the frame to the layout direction returned from wx.GetApp().GetLayoutDirection(). However, the more I dug into this the more I realized that I don't know enough about how RTL UIs are really supposed to work, so you'll probably be better off asking about this with a ticket over at https://trac.wxwidgets.org/ or in the wx-users mail list.

[kdschlosser]

I believe I found the problem. wx.Locale.GetSystemLanguage() returns wx.LANGUAGE_HEBREW when I never actually changed the language. I changed the formatting to Hebrew But I did not change the actual displayed text in any way. Nor did I change any of the system language settings. I changed the user settings. In my application I am subclassing wxApp and in the init method I have this code

self.locale = wx.Locale(wx.Locale.GetSystemLanguage())

I did a simple print of the returned value of wx.Locale.GetSystemLanguage() with the format settings set to English (United States) and the returned value is 60. I did it again with the format set to Hebrew and it returned 100. even tho I never changed the display language just the formatting. I did come to find that even if I bypass the setting above and not set it. The main program is now correct but any dialogs I open are not. and when using StyledTextCtrl in one of the dialogs the text displayed in the control is in Hebrew.

There seems to be some kind of issue with the language bits of wxPython. changing the text and numeric formatting should not actually change the displayed language. the returned value from wx.Locale.GetSystemLanguage does not return the system language nor does it return the user language (the first being the one it should return)

Here is a test script you can run to see what I am talking about

import wx
import locale
import ctypes

print(ctypes.windll.kernel32.GetUserDefaultUILanguage())
print(ctypes.windll.kernel32.GetSystemDefaultUILanguage())
print(wx.Locale.GetSystemLanguage())

print(locale.windows_locale[ctypes.windll.kernel32.GetUserDefaultUILanguage()])
print(locale.windows_locale[ctypes.windll.kernel32.GetSystemDefaultUILanguage()])
print(wx.Locale.GetLanguageCanonicalName(wx.Locale.GetSystemLanguage()))

this is going to print the different default languages. If you change the format option in the Region and Language settings panel and then run the script you will see that the value returned by wxLocale.GetSystemLanguage has change and the calls to the windows API has not. Now I would use the windows API. But wx was not coded to use the Windows LCID codes I am not exactly sure as to why it was made this way. but it was. In the docs it states that init is overloaded to accept a short name. but no where in the docs does it tell you what the short name implies. does that mean it accepts the canonical name?? Or do I have to create a create a cross reference from windows LCID to wxPython language codes in order to make this work properly??

Now as far as the whole window getting getting horizontally flipped. this is 100% an issue inside of wxPython. I do not have any other application that exhibits this problem. If I do not set the locale in the application the problem does not exist in the main app frame only. if i create a dialog it does exist.

[kdschlosser]

Please excuse me for getting wxPython and wxWidgets mixed up here.

The problem is with wxWidgets. as far as returning the incorrect language code.

\src\common\intl.cpp line 789 reads

LCID lcid = GetUserDefaultLCID();

and it should read

 LCID lcid = GetUserDefaultUILanguage();

the Microsoft docs have a description for this method as returning a "locale identifier" not a language identifier.

and with the GetUserDefaultUILanguage the description reads that it returns a "language identifier"

With windows you can set the locale, keyboard, displayed language, and system locale to be completely different for each other. So having the text displayed in a fashion that reads from right to left when the language is set to English would not be a correct thing to do. You would use the locale (Format) for displaying time/date or currency. for text formatting other then text direction.

Also. I did want to mention the system locale is typically used for non unicode applications.

The above code would fix the incorrect language code being returned. I do not know what else this would affect or what else would need to be changed.

And the method of GetSystemLanguage in both wxWidgets as well as wxPython's wxLocale is technically speaking not correct as far as Windows is concerned. There is a SystemDefaultUILanguage and a UserDefaultUILanguage in Windows. and these might not be the same. an example that would cause it to not be the same would be. If i change the language for non unicode applications to something different then what I have set the user language to

I do not know how any of the Linux variants or OSX work in regards to this.

[RobinD42]

Please create a ticket for this at https://trac.wxwidgets.org/, if one doesn't already exist, and add a link to the ticket here so we can track it. I don't know what the right answer here is, but they will.

[kdschlosser]

Ok no worries m8 I have spent a great deal of time scowering over the windows API writing python connectors. And I know Microsoft has made a mess of their API adding missing prices by adding additional functions. And their documentation is extremely confusing sometimes.

In the interrum I did write a cross reference for Windows LANGID's to wx.LANGUAGE_* codes which has placed a band-aide on the problem for the time being. Thankfully it was pretty simple to do because the symbol names used in Windows are almost identical to the ones you used in wxPython just wrote a small script to do the dirty work. And then did a little cleaning up afterwards. There was a direct cross reference for about 98% of them. There are some codes in wx that do not map but they were all really obsecure languages.

I will do up that support ticket. And I will also dig through the source for wxWidgets and see if there is anything that relies on that function returning the locale instead of the language. You may know if there is. I am not all that familiar, and it's a sizeable project. Easier for someone that knows their way about it.

Thanks for your assistance. And I do appologize for getting wxPython and wxWidgets mixed up. I did not realize they were 2 different projects.

K

[kdschlosser]

There is already a ticket for this issue. it was posted about 9 years ago.

https://trac.wxwidgets.org/ticket/11594

It has already been a confirmed issue

[DarkFenX]

One of users of my app experiences this (or very similar) issue. It was not there in 4.0.6, it appeared when I switched the app to 4.0.7.post2.

[RobinD42]

There was a change in Python 3.8 on windows where it seems to be initializing the locale to match the system settings, (they didn't do anything with it before 3.8 AFAIK) so I had to add some code to wxPython starting in 4.0.7 to try to make the wxWidgets locale settings match. I didn't do it for just 3.8 as I figured it should be consistent no matter which version is used. Without the change then wxWIdgets complains about the mismatch when using Python 3.8.

You can see the code for this change in wx.App.InitLocale. If I should be doing something differently there then please let me know. I'm mono-lingual so things like this are still not quite natural for me.

For Python < 3.8 then calling app.ResetLocale() just after the app is created may restore the old behavior.

[kdschlosser]

There are other issues related to language and locale running on Windows 10.

The problems are woven between Windows 10 wxPython and Python. There are "virtual" LCID's that exist. i think the id that gets used is 0x1000. More then one locale language combination can use this id. It is seen quite a bit in Windows 10. not so much on earlier versions of Windows.

I wrote a module navigates around this problem. There is a problem with how Pythons locale module works with windows. I do not specifically remember the exact issue(s) off the top of my head. I do know that the module I wrote does address and correct this problem. The module I wrote also provides Windows LCID to wx.LANGUAGE_* mappings there are also some added goodies other then that as well. things like being able to get the country name and language name in native tongue/script. also flag icons for countries.

I did want to mention the incorrect descriptions of several functions in the documentation for wx.Locale

• GetSystemEncoding: Tries to detect the user’s default font encoding.
• GetSystemEncodingName: Tries to detect the name of the user’s default font encoding.
• GetSystemLanguage: Tries to detect the user’s default locale setting.

None of those does anything with the users locale. on Windows

There is a conflict of description in the documentation as well. The Init method has this for a description of the language parameter.

language (int) – wx.Language identifier of the locale. LANGUAGE_DEFAULT has special meaning – wx.Locale will use system’s default language (see GetSystemLanguage ).

that description would be correct as it does get the system default but then it points to GetSystemLanguage for more information.

An application that is capable of handling unicode should NOT be using the system default. The system default is only there for backwards compatibility with non unicode applications

wxWidgets use of the System default locale and language is not correct. it has been reported on more then one occasion and has been around for a long long time. It should have been fixed when the ability to handle unicode was first added to wxWidgets. wxWidgets does not have 100% correct handling of languages and locales on Windows and should be updated. I believe that the code that is in place did work correctly before unicode, but got overlooked when unicode was added. I have done what I am able to with the module I wrote and it does help it is not the best way,

if you would like to check out the module I wrote that helps in sorting out the locale/language conundrum you are more then welcome to.

https://github.com/kdschlosser/pyWinLocale

that module contains all of the hard coded locale's and languages from Windows XP to Windows 10. including all of the language packs that can be installed. If an LCID is not available it will not be presented to an application as being able to be used. There are convenience methods in place to set Python's locale (using the code page for the locale that was retrieved using the Windows API) and also wxPython's locale (mapping the LCID to wxLANGUAGE_*) according to what the user of the application has chosen to use (if changed from the user default set in windows). It is not set up to replace Python's locale module so there is still no support when using a language with a custom LCID. It can be a replacement if needed. I do not know if there is a way to "monkey patch" how wxPython/wxWidgets handles locales. It appears that there is not a way because the objects necessary to do that reside in c code and get created by the passing of an integer, This limits what locals and languages are available because this integers are hard coded. so the use of a "non standard" locale/language combination would not be possible.

Windows has a really complex setup for internationalism. In order to support it properly there really needs to be the ability to set the locale, keyboard language and display language completely independent of each other. There also needs to be some kind of a way to load custom locals. In Windows the user is able to modify locale settings like the format of time and currency. This is going to cause Windows to create a virtual LCID and Python will pitch a fit about it as virtual id's are not supported. Something like what I have written in that module would need to be used and the ability to create custom locale classes that contain the proper format specifiers would then be able to be set and be used by wxPython/wxWidgets. I do not know if this is able to be done or if something like this already exists. I have not come across a way in wxPython to be able to do it,

@RobinD42 might be able to suggest a way to go about it. Tho I do feel that this is something that should be in wxPython or wxWidgets.

Python's locale abilities are badly handicapped when using Windows. It should not be relied on at all. If a user is using a locale/language combination that has a virtual id setting the locale in python is going to throw a traceback when calling setlocale. there are 201 locale/language combinations that use 0x1000 as an LCID that I know of, the majority of them were added in Windows 10.

The module I wrote is not perfect and it has not undergone testing to a level that I would like to see. I am sure there are probably issues with it and if someone wants to use it I am more then happy to provide support if needed.

[kdschlosser]

when I run the module I wrote on a stock Windows 7 x64 SP1 machine with no language packs installed it supports 123 locales and 196 languages wxPython/wxWidgets has support for 127 possible locale/language combinations.

I am not sure how many Python is able to support as I did not test for this specifically.

[DarkFenX]

I think I will just upgrade python from 3.6 to 3.8, rather than implementing python version-specific hacks.

[DarkFenX]

I tried updating python to 3.8.1, it did not help for the issue reporter. I tried reproducing it myself - turns out just changing "formats" to "Hebrew" is enough (win8.1).

3.8.1 on my windows machine did not help for me either. Resetting locale right after app has been created breaks my app. Do not have time now to look into traceback, will look into it later, maybe I can fix it on my side.

[kdschlosser]

The problem is with the locale and that being used to set the RTL. This is an incorrect thing to do. The language is what should be used to set the RTL. But the issue is deeper then that because it makes a mirror of the entire GUI. It should only be displaying the text in RTL. I have not found a way to override the locale handling of wxWidgets to correct the problem.

[DarkFenX]

wxpython 4.0.6 + python 3.7 do not have this issue. I failed to build 4.0.7post2 under 3.8, so have no info on this combination

[kdschlosser]

The problem also exists using wxPython Classic I am not sure what could be different between 4.0.6 and the current version of wxPython that would cause this issue to appear. I know this has been discussed over the past 10 years in the wxWidgets issue tracker. But nothing has been done to solve the problem. It has been agreed on that how wxWidgets currently manages locals and languages for Windows is incorrect and it has also been discussed as how to go about setting it up properly But I think that in order to make the changes in a manner that is correct it would require a pretty significant code change to wxWidgets, and those changes will cause the API to get broken. So I can understand why there has been no attempt to repair this problem.

The only way to get this issue to be escalated is to post in the wxWidgets issue tracker that you are also having the same issue and inquiring about it getting fixed. The issue has been open for 10 years now and nothing has been done because of the small number of people that are seemingly effected by the problem. I do believe that there are more people having the issue then is realized.

[kdschlosser]

it appears as the language/locale problems that exist in wxWidgets running on Windows are not going to get fixed https://trac.wxwidgets.org/ticket/11594#comment:14

No sense in keeping this open anymore.

And now the hunt begins for a different GUI framework. The question of why wxWidgets is losing favor is no longer a question.

[kdschlosser]

I am actually reopening this issue. It kind of rubbed me the wrong way when I was told the locale issues in Windows was not going to get fixed. I have not tested to see if the RTL problem has been solved specifically But I have written a Python wrapper around wx.Locale and also wx.LanguageInfo that does correct a very large underlying issue where wxWidgets uses LCID's for identifying a language/locale. The problem with that approach is that not all language/locals have a unique LCID. The other issue is when you go to set a language wxWidgets automatically sets the locale to whatever locale has been preset in wxWidgets for that language.. That is not ideal for everyone. wxWidgets also uses hard coded numeric constants for identification of a language instead of using the universally accepted ISO 639-1 and ISO 639-2 codes for languages and ISO 3166-1 codes for the country/region. Windows has alternate functions that can be used with ISO locale codes, and Microsoft suggests to not use LCID's for identifying languages and to use the ISO codes instead. They recommend this because of the LCID's being shared and there being no way to identify what locale you are wanting.

So this wrapper supports the old constant values as well as being able to pass the ISO locale codes in order to set the locale.

I really did not want to go through the effort of porting all of my applications code to some other GUI framework. That is a heap of work not to mention also having to update the 400+ extensions that are available. Probably close to the 3/4's of a million lines of code in total...

If I knew enough about the mechanics of wxWidgets and I was also stronger with my c/c++ skills I would have made the changes needed to wxWidgets. I was however able to get the job done, and in only 4 hours. I was not sure if I was going to be able to wrap wx.Locale how I did, I did not think that wxPython/wxWidgets would let me. However It did allow it.

The attached file is the wrapper and also a test program that will enumerate all of the available languages and locals installed into Windows and it will create a wx.Locale instance for each of them. This tests the functionality of the wrapper and also the ability to use that language in wxWidgets. on Windows 7 x64 SP1 with no language packs there are 210 language locale combinations that properly load, that is all of them that are installed onto my system.

I hope this might find some use to correct the problems associated with wx.Locale running on Windows.

wx_locale.zip

[Metallicow]

It seems that locale/translation stuff isn't many folks strong suit. I kinda agree with him on this subject. It probably would be better to get in contact with other software developers to handle the issues you are experiencing with wxWidgets. Possibly that or if you might be able to get some students involved with some of them in a program such as google summer of code might help also... but it seems that someone more experienced might or might not want some sort of pay/compensation for the work. Then there would need to be some sort of review before it gets pushed. Sorry but I can't be of help with this one.

[kdschlosser]

the whole locale thing is a snafu of annoyances in general mainly because there is no single universal mechanism that has been adopted by all platforms. The hardest thing to understand about it is that a locale is made up of 2 components. a language and a country. The 2 are connected only for the use of symbols for things like currency and numeric separators those types of things. A language is not dependent upon a country and a country is not dependent upon a language. they should be able to be set independently of one another. I should not have a GUI change the locale when I tell the GUI what language to use. This is what is happening in wxWidgets. Also wxWidgets is unable to use locals that are installed into a Windows system. This is because of it using LCID's which I explain above. Now for reasons I do not understand Windows has several ways to set a locale. You can either do it by using the endlich name of the language/country followed by a . and then the code name.

English_United States.windows-1252

Unfortunately this is not always going to work, for languages and countries that do not have an english name you then use the isocode and a . and the code page

en_US.windows-1252

If the first option is available then the second one will not work. and here seems to be yet another oddity is that if you use the Windows API to get the english names for the country and language you have to pass the isocode. but the isocode has to be formatted differently it has to contain a dash instead of an underscore.

I just red somewhere yesterday that Microsoft suggests not specifying any other code page except utf-8 I am going to run some tests on that and see what the end result is.

[RobinD42]

wx_locale.zip

I haven't taken a closer look at this yet, but I wonder if this is Windows-only or is it expected to work on other platforms too?

[kdschlosser]

I am not aware of any problems that exist with wxLocale on other platforms other then Windows. So I did not write it to handle other platforms.

so using code like the following would be needed.

import sys
if sys.platform.startswith('win'):
    import wx_locale

import wx

Once the module is imported wx.Locale has been overridden by the one in that module. and the one in that module subclasses the original wx.Lcoale and changes out the use of lcid's for a sting locale.

Now that portion is actually fairly interesting because with different versions of Python comes a mess with the locale module. locale.setlocale's locale parameter has to be formatted differently based on what version of Python you are using.

here are some examples.

en_US.windows-1252 English_United States.windows-1252 English_United States en_US

It also accepts a Windows LCID for a language_country as well.

I do not remember what versions of Python support which way of passing a locale. Some locals require you to have the code page in order to work. while others will fail if you have the code page attached. Some locals you need to have the language and country spelled out while others have to be the ISO form. There is really no method to it. It is actually a complete mess. So I have the program try setting the locale a bunch of different ways until it finds one that works.

I do know that in Python 3.7 you can set all locals with the UTF-8 code page. This is actually recommended by Microsoft, but for some reason this cannot be done on Python 2.7.

If I use the Windows API function SetThreadLocale to set the locale for the application would it populate to wxWidgets/wxPython as well as Python? This is something I have not tested. If I am able to then I can bypass using wxSetLocale and also locale.setlocale and go right to the mechanism I know is going to work 100% of the time.