search

wxWidgets / Phoenix

wxPython's Project Phoenix. A new implementation of wxPython, better, stronger, faster than he was before.
http://wxpython.org/
2.32k stars 519 forks source link

Needing help with a test of wx.Locale #1616

Open kdschlosser opened 4 years ago

kdschlosser commented 4 years ago

This is not a new issue but hopefully a solve for other issues that have sprung up when using wxPython and Windows. These problems have become more apparent since Windows 10 has been released.

I will list the issues that I know about that should be fixed if this monkey patch is used. If this all works properly @RobinD42 might consider adding it to wxPython. The problems listed below actually reside in wxWidgets and not wxPython. These problems are not going to be fixed in wxWidgets any time soon so there are people that are having problems that cannot wait until then to have the issues addressed.

Here are the problems I know about.

The largest underlying problem is wxWidgets using LCID's (locale identifiers) as the way to identify a locale when accessing the Windows API. There is a set of defined languages in wxWidgets, these are the wx.LANGUAGE_* constants. these constants do not map directly to a Windows LCID. The problem with using LCID's is that more then one locale can have the same LCID. This is called a virtual LCID. LCID's exist only as a mechanism to allow older software to run on newer versions of Windows. They have been around a long long time and are not going to be removed neither are the functions that use them. Newer applications should not rely in LCID's at all.

There seems to be more people encountering problems with wx.Locale and there is a reason for this. With Windows 10 the use of the "virtual" LCID has become common place almost all new locales added to Windows 10 have a virtual LCID. so what takes place is when wxPython loads and the Locale gets set it grabs the LCID that is being used currently in Windows and it then uses that LCID to set the locale of the thread. Now if there are 50 locales that use the same LCID how is Windows supposed to know what locale to set the thread to.. The answer is.. it doesn't. I think it reaches into the hat and pulls one out. The problem there is most times it will grab the wrong one. So now there is going to be a mismatch between wxWidgets and Python. Hence the odd traceback for that. This same thing is causing the returned values for locale names and canonical names to not be correct.

So I have written a wrapper/monkey patch that should address these issues. While I was at it I decided to add some additional functionality to it.

Here is a brief rundown on how this is going to work.

import sys
if sys.platform.startswith('win'):
    import wx_locale

import wx

locale = wx.Locale(iso_code='en_US')

You do not need to interact with wx_locale at all. once it is imported it does what it needs to do to patch wxPython.

The iso_code keyword i added to the constructor. This makes it easier to use. When the locale instance is constructed it is going to take care of setting wxWidgets's locale as well as Python's locale for you. you will not have to set python's locale separately. I did this because with each minor release of Python the locale module changes. It changes in a manner that makes it a pain to set the locale on a Windows machine because the value that needs to be passed to it is different each minor version bump. I figured it would be nice to have everything done and handled internally so the user would not have to spend the time to figure out how to make it work.

These issues to my knowledge only exist on Windows.

One of the biggest hangups is wxLanguageInfo. this class should represent a language. but it actually does not. the canonicial name can either be an ISO language code or it can be an iso locale code depending on what has been hard coded into wxWidgets. wxLanguageInfo should only be for language information and the CanonicialName property should only return the language ISO code.

That has been fixed with this patch.

wx.Locale.Description: returns the english name for the locale. This is gotten from Windows wx.Locale.NativeDescription: returns the name of the locale using the script for the language of the locale wx.Locale.LocalizedDescription: returns the locale name using the script from the set locale. To use this property you need to do some crafty coding.

in this example lets say that the system locale is de_DE (German in Germany) If you wanted to get the locale description for Spanish in Spain in German script this is how you would do it.


locale_1 = wx.Locale('es_SP')
locale = wx.Locale(iso_code=wx.Locale.GetSystemLanguage())
print(locale_1.LocalizedDescription)
print(locale.LocalizedDescription)

Now you have accessing wx.LanguageInfo.

language = wx.Local.FindLanguageInfo(locale.GetLanguage())

the Locale.FindLanguageInfo static method can be passed the 2 or 3 letter ISO code for the language. Because I have decoupled the locale from the language that is what makes this possible,

So you have the availability grabbing a LanguageInfo instance for any language. You can use wx.Locale.IsAvailable('en') to check if the currently set country and language are installed in windows. The language and country iso codes are what make up a locale.

You can also do a test using the LanguageInfo instance by calling the TrySetLocale method.

language = wx.Local.FindLanguageInfo('en')
try:
    language.TrySetLocale()
    print('locale ' + language.LocaleName + ' set correctly')
except ValueError:
    print('error setting locale ' + language.LocaleName)

The LanguageInfo class also has the Description, NativeDescription and LocalizedDescription properties.

There is a method that can be used to get the locale name with ANSI code page attached. The returned value is going to be the formatting of the ANSI name that is needed in order to set the locale of Python correctly.

print(language.GetANSIName(wx.GetLocale()))

The CanonicalName property of wx.LanguageInfo is going to return ONLY the language canonical name. 'en_US' is the canonical name of the locale. 'en' is the canonical name of the language.

the LocaleName property of wx.LanguageInfo is going to return the canonical name for the locale, 'en_US'

Give it a shot and let me know if it has any issues or if it works and solves any issues that you have been having.

You can run the attached file and it will print out all of the locales and languages you have installed into Windows. On a vanilla Windows 7 x64 SP1 install there are 210 language/country (locale) combinations. I am able to use all of the available wx.LOCALE_* format types that get passed to Locale.GetInfo without any errors. I am also able to set the wxWidgets locale and also the python locale with all 210 of them without any problems.

wx_locale.zip

@RobinD42 If this is not a good place to have this message/test placed let me know where to move it to wherever you want me to. I figured this would be seen by users having issues with the locale and they might be willing to give this a go to see if it fixes their problems. If it does and you are interested in putting together a solution that would be added to wxPython I would be happy to help. This script contains all of the mechanics needed using pure python.

DarkFenX commented 4 years ago

@kdschlosser seems like you understand how locales work on different platforms decently, could you check if your implementation has any issues with doing this: https://github.com/wxWidgets/Phoenix/issues/1515#issue-564069756 and https://github.com/wxWidgets/Phoenix/issues/1515#issuecomment-585764207 ?

kdschlosser commented 4 years ago

@DarkFenX it should fix both of those problems.

robertluwang commented 4 years ago

Is any planning to merge @kdschlosser patch to wxpython soon?

RobinD42 commented 4 years ago

@kdschlosser, I finally got around to taking a closer look at this and giving it some thought. I wouldn't mind seeing a PR for this, although I'd like to see it organized a little differently.

RobinD42 commented 4 years ago

In the meanwhile, I'm going to go ahead and merge the new InitLocale implementation discussed on discuss.wxpython.org a while back.

kdschlosser commented 4 years ago

The problem is that the problem does not only lie in wxWidgets. It is also in Python (every version). while I have not looked at the python code specifically for dealing with the locale I can just about say with 100% certainty that how Python handles locales on Windows is really close to being the same as wxWidgets.

Python and wxWidgets should be in sync with one another to make sure there is not doing to be any kind of an issue. Here is an example of an issue. In the main thread if I set the locale in wxWidgets and then I set the locale, this may seem like a simple thing to do but it really is not. Python has it's own set of problems because with each new minor version the parameters that locale.setlocale take change. It is pretty easy to get it goof up and end up having it not match what was set in wxWidgets. Then you toss in the fact that wxWidgets uses LCID's and a single LCID can actually be assigned to more then one locale and now you end up with a really high chance of it getting outta sorts.

The 2 of them really should be the same. An ideal solution would be to have wxPython/xWidgets get it's locale set if a user uses the locale module to set the locale, you would want this same thing but in reverse as well. It keeps everything on the same level so to speak, removing the possibility of the above mentioned locale problem.

This is where it gets really tricky. we need to remove the use of LCID's from both Python and also wxWidgets. Now I would love to be able to do this without having to "monkey patch", there is no way to do it without monkey patching. The locale module needs to be updated and the only way to go about doing that is to patch it. wxWidgets and also wxPython are written in a manner that does not allow for easy manipulation of these classes. The reason why monkey patching wxPython has to take place is because of back end code accessing the static methods in wx.Locale. so wx.Locale has to be patched in order to have it work properly. Otherwise it ends up calling the static methods in the original class which will conflict in a lot of cases with the sub classed one that may currently be set. Or it will simply return the wrong information because of using the LCID's to get the information from Windows.

I do not have enough knowledge into the wxPython build process and the code for whatever it is you are using now to write the extension modules. I know you were using swig, I am not sure if that has changed or not. either way I am still not familiar with it enough to be able to make the modifications that would be needed in order to get around the above mentioned issue.

I have written an extremely simple version of this that simply makes everything work with no additional fluff. It changes locale.setlocale and getlocale to take ISO codes and also wx.Locale. It changes the static methods in wx.Locale to use the ISO codes instead if the LCID's in order to get locale information from windows.

You can see how I did it here.

This are windows specific patches but It would not be to hard to make them cross platform.

This link is to the code dealing with the Python locale module. https://github.com/kdschlosser/EventGhost/blob/wx_python_locale_wrapper/eg/WinApi/locale_patches/__init__.py

This link is to the code dealing with the wxPython/wxWidgets locale https://github.com/kdschlosser/EventGhost/blob/wx_python_locale_wrapper/eg/WinApi/wx_patches/WXLocale.py

This link has code in it that is used by both of those other links. some of the code in here is going to be application specific but can be removed easily enough. https://github.com/kdschlosser/EventGhost/blob/wx_python_locale_wrapper/eg/WinApi/Locale.py

I know the above works and it works without problems. I have not had any of the users that are running it report any problems with it. It's been almost 2 months since I wrote that.

It's the least aggressive way I could come up with to provide a solution to the issue. while I do not have that set up to change the locale in wxPython/wxWidgets from the python locale module it is set up to change the python locale when the locale in wxPython\wxWidgets gets changed. I did it this way because the user has access to controls in the GUI that facilitate the changing of the language/locale so It made it easy to just have it all done via wxPython\wxWidgets.

As you can see it is really not a large amount of code to fix the issues. It was a royal pain to get it all mapped out properly so it would function correctly. The links above only make it work when using Python 2.7. there would need to be different patches made depending on the version of python that is running, this is because of the several different renditions of the locale module.

PS. This internally handles using

when using locale.setlocale, I can't remember off hand but I also think it handles the English_United States variations as well.

I might have a better handle on dealing with languages and locales on Windows then you have. Where as you are going to have a better handle on the Posix OS's then I have. you also know the inner workings of the build/scripts for wxPython in the event that changes need to be made there. I think that making a separate development repository for just handling this one thing would be the thing to do. and the both of us spend some time working the problems and writing the best solution. There are simply to many variables and problems that can occur especially with writing something that is cross platform and dealing with Windows. having locale problems are not fun for one can cause some really strange behavior and errors for 3 and 3 they can also stop an application from running all together. Microsoft wrote language support into Windows wayyy back in the 3.1 days. I will almost gurantee you that used copy and paste and planted the code into Windows NT which has evolved into what you see today. They have added onto was existed and make a complete mess out of it. Thre are cases when using the erase button and starting over does have it benifits. I tried to get the author(s) of wxWidgets to correct the problem where it should be fixed. I didn't get anywhere with that. The locale issues with Python and windows have been brought up 100's of times. and all that seems to get done are changes that break API but don't fix the locale problems.

That being said the only choice is to patch the problems. The important thing is to make sure it solves the problems. This is going to need more then just my ideas in there.. You input would really be the input that is needed. I am willing to help as much as I can and I can provide you with a heap of information about anything Windows API. I only know enough about Posix to get my toes wet and then watch them get bitten off by the piranhas LOL.

Let me know what you would like to do. I think a place to be able to have conversation and also only have the changes to handle this one problem taking place would be ideal.

kdschlosser commented 4 years ago

Thi can also be written in C and be compiled as an extension module if you wanted it to be apart of the core. I know if using Cython a pure python file can be compiled into a C extension and the needed code for documentation should e able to be handled that way. I do not know if whatever linking software you are using is able to do that or not. using Cython to do this also offers one hell of a speed boost and you should consider it for use with the agw modules. They would run leaps and bounds faster I will say that. The skipping and studdering of some of the animations/resizes I am betting would go away and everything would run nice and smooth.

kdschlosser commented 4 years ago

OK I am sorry this has taken me a while to get back to. If still needed I am going to reorganize the code to function like @RobinD42 wants me to do. Just let me know if this still needs to be done.

tatarize commented 4 years ago

If you have a snapshot that might have the bugs fixed tag me. I have some folks with native language Windows being translated to English and it's failing to allow the images to show. See #1804 and meerk40t/meerk40t#242 .