Upgrade NVDA codebase to Python 3

[josephsl]

Hi,

This issue serves as an umbrella issue for several related issues, one of them being #7077.

Python 3 introduces changes to syntax and inner workings of Python. Some of the changes include:

• Unicode strings.
• Use range function.
• _winreg -> winreg.
• threading.Barrier.
• collections.ChainMap.
• Reorganized standard library folder structure.
• Timer-based GIL instead of instruction count.
• Possibilities with asynchronous functions (async/await).
• Virtual environments.
• Built-in compatibility with Windows 10 (Python 3.5 or later).
• Losing Windows XP support (Python 3.5 and later).
• No more .pyo files.

Upgrading to Python 3 will require huge coordination, as NVDA developers, add-on writers, translators, users and others will need to adapt new attitudes, assumptions and so on. In terms of dependencies, the only thing left is SCons (wxPython is discussed in #7077).

Thanks.

[OzancanKaratas]

Python's 2to3 script can be used for this request. Its developers included 2to3 into Python 3.

[josephsl]

Hi,

A few years ago, I tried this approach and it produced mixed results. Although 2to3 did work to some extent, the resulting code needed corrections. Also, this is just not as easy as letting the current code run through this script: dependencies must be satisfied, and we need just two more dependencies in order to even start transition planning, namely SCons and wxPython (wxPython is another story, which is covered in #7077).

Thanks.

[josephsl]

Hi,

Although the actual transition plan is a bit cloudy at the moment, @JCSTeh suggests via IRC that next year would be a good timeframe, and @LeonardDer concurring with this thinking. A tentative plan is thus:

Prerequisites: wxPython 4 testing must be completed i.e. merged into master and tested in at least two NVDA stable releases. For the purposes of this plan, N refers to NVDA release with wxPython 4, N+1 referring to next one and so on.

1. Proposal and research: a suitable Python 3 release must be chosen and should be aimed for during the transition. At the moment Python 3.5 is the lowest practical version due to availability of wxPython 4 wheels for it. Ideally, for maximum market availability, Python 3.4 should be targeted as it is the last version to support Windows XP. Based on the proposed time frame below, the most practical version to aim for is 3.6, with 3.7 coming out in 2018.
2. Issue identification: once a suitable Python 3 release is chosen, showstopper issues must be identified. These include incompatible syntax, deprecated code paths and modules, replacement modules and so on. This cannot commence until NVDA N is released.
3. Preparation: developers (Core and add-ons) should learn about differences between Python 2.7 and the chosen 3.x release while NVDA N+1 and N+2 are under development. This is also the ideal time frame to research fixes and workarounds for major issues identified.
4. Transition workflow: once issues were identified and all the dependencies are satisfied, transition work can begin. The ideal timeframe is NVDAN+3 and N+4 development. During this time transition work will be incubated in next.
5. Add-ons: at the start of NVDA N+4 development, add-ons community will be asked to start porting their code to Python 3. A few authors will do some work during N+3 development. Most of this should be done no later than next release after Python 3 work is integrated into master branch. For example, if Python 3 work is done by NVDA 2018.4 master, the work from add-ons community should be done by 2019.1 (or 2019.2 at the latest).
6. Aftermath: any dependencies on Python 2.7 should be kept to a minimum, or if to be removed, should be done when it is ready.

The above plan is subject to change. Thanks.

[nishimotz]

Is it possible to build a NVDA distribution, which contains selectable Python 2.7 runtime and Python 3.x runtime simultaneously? If single version of NVDA can be switchable between Python 2.7 mode and Python 3.x mode during the migration period (for several official releases), possible benefits are:

• the official NVDA build will not be separated during the migration. In other words, I my concern is the separation of two official versions of NVDA binary, which are based on Python 2.7 and 3.x.
• the evaluators or add-on developers of NVDA can compare the behavior of apps or add-ons easily without using two distributions of NVDA.
• the user can easily choose between sticking around 2.7-based NVDA or testing 3.x-based NVDA during the migration period.
• the release management will be flexible, because the real change is just to switch the default mode from 2.7-based mode to 3.x-based mode. During the migration period, the usage statistics can be obtained to judge that 3.x-based mode is mature or not.

[jcsteh]

Thanks for your thoughts. I don't think having both versions in the same distribution is feasible/workable:

• There is probably at least some code that cannot be compatible with both Python 2 and 3; it must be one or the other. This means the Python 3 work has to be kept in a separate branch.
• I think this whole concept is too complicated for many users to grasp. It's reasonable to expect that those who have some understanding of such issues (and are willing to test) will be quite able to separately test another branch.
• We'd effectively be shipping code in a final release that we aren't certain is ready for use in production. That goes against the idea of a final release.
• All the Python byte code and extension dlls would be different between the two versions.
• We wouldn't be able to have both versions in the same directory due to file naming conflicts.
• It would significantly increase the size of the download (probably near double).
• Our build system would require major refactoring to allow for this, if it's even possible.
• A user would still have to select which executable to run anyway, so they're pretty much separate except for the download.

[Brian1Gaff]

Would you maybe consider that by the time Python 3 versions are stable that it be only a 64 bit app? Or can it be made to install in both 64 and 32 bit systems. It would get rid of the issue of people attempting to install it in xp, as will I am sure happen otherwise. Brian

[LeonarddeR]

Ideally, for maximum market availability, Python 3.4 should be targeted as it is the last version to support Windows XP.

I don't want to start the whole Xp discussion over again (#6718), but I disagree with this. When moving to Python 3, we should target the last version available at the time of the switch.

[josephsl]

Hi, hence the “ideal” case. In reality, 3.5 or higher should be targeted (3.6 is optimal), hence we can officially start working on #6718 as well. Thanks.

[LeonarddeR]

Thanks for your clarification.

Just for reference, one major argument for not targetting 3.4 is this overview. Note that the EOL date for Python 3.4 is 2019-03-16. After this date, major security risks won't be patched any more.

[josephsl]

Hi,

Laying the initial foundation in an internal branch (located at a personal fork of NVDA source code):

The trivial case at this time is winreg versus _winreg compatibility. Because API's and side effects are the same, this is perhaps the easiest milestone: import winreg, and if not, use _winreg and rename it to winreg. I'm intentionally using the newer winreg.* methods to incorporate forward thinking principles.

For future reference: if anyone wants to investigate Python 3 compatibility (for personal research purposes), or wish to test and send PR's, please do NOT send them to NV Access yet (there are higher priority works to be done, hence I propose p3 for this year). Instead, use josephsl/NVDA branch py3000 for now.

In doing this, I recommend folks to follow the below advice and guidelines:

1. Become familiar with python 3 syntax, some inner workings, and standard library folder structure.
2. Install Python 3.5 or later (Python 3.4 won't cut it, as noted above).
3. Test a concept in Python 3 interpreter first before applying to NVDA.
4. Use 2to3 script with -f all (or a combination of needed fixers) with a diff output being sent to a text file for examination later. This should be done regularly.
5. After modifying each file, run NVDA to make sure your edits didn't break features or introduce regressions.
6. For now, try using compatibility approach so NVDA can at least run on 2.7 and 3.5/3.6/3.x. This should be done until SCons and/or wxPython and what not are ready.
7. If you are aiming for stricter compliance, use Python 3 syntax and modules from the beginning.
8. Get Python experts involved so they can serve as additional reviewers and help us iron out things here and there before the official transition begins.

Note: the above procedure is for those who're thinking about tackling this issue for personal research purposes. During this process, someone should write a report to be sent to NV Access, collecting our data and findings.

When meeting ambiguous code or code paths that could work well on both 2 and 3: try either strict compliance or compatibility approach. The following winreg and dictionary key iteration gives examples of these approaches:

• Imports:

Strict: import winreg and use winreg. Compatibility: import _winreg as winreg and use winreg., or use import winreg as _winreg and use existing code paths without modifications.

• Dictionary key iteration:

Strict: for key in dict.keys(): pass Compatibility: Have a flag that checks sys.version.startswith("3") and: for key in dict.keys() if py3 else dict.iterkeys(): pass

Once you choose an approach, please do not be tempted to switch methods, otherwise you'll end up with more headaches than before.

Thanks.

[ctoth]

I am concerned about the long tail of addons that will either never be ported or will require community intervention or otherwise will be slow to Python 3 and recommend moving up the timeline for pushing addon developers into at least using the subset of Python that is compatible with both versions to as early as possible. I further recommend including something like Python Future in core to ease the porting process.

[josephsl]

To @CToth: As for add-ons, I'm taking your advice by making sure future versions of my add-ons are source-compatible with both 2 and 3. Sometime this year, I'll advise others to follow suit. Thanks.

[josephsl]

Follow-up to SCons: see #7520 for an update. With that in place, we're ready to start researching. Thanks.

[josephsl]

Waiting for #7546 to come to master branch before continuing. Thanks.

[josephsl]

As noted on #7520, SCons 3 is out, and #7546 is part of master, so let's start researching. Thanks.

[josephsl]

Update: Once #7568 is part of NVDA master, that's when I propose code should be checked in. Thanks.

[LeonarddeR]

I belief that first, all dependencies for Python 3 should be satisfied, so this includes the update to WX Python Phoenix 4.0 being incorporated into Master.

[josephsl]

Hi, yep, that’ll be done as part of a pull request once wxPython 4 officially ships (now up to beta 2). Thanks.

[Brian1Gaff]

If I may make a users comment here, using my idiot head.." There sould, in my opinion be a new snapshot branch that we can test. Also when thinking about add ons, maybe some kind of protection should be put into the loading of add ons so nobody can load the old ones, only the new ones in the new code versions of nvda.

As you will want testing to at least be done by as many basic users as possible when the major issues are corrected, it is then when it can be put out in a master. Also, because of changes in syntax, does this mean app modules already in nvda will possibly need to be rewritten? If yes, could I ask that we still support legacy software aslong as people have it running on at least windows 7. That will stop a lot of problems trying to solve things later on. I and others do use such software, but I'm not sure how many of these use specific app modules or just work m in nvda anyway. Outlook Express/live mail is one, andprobably older versions of Office like 2007, and several third party apps as well.

As I've said before, I'm OK to test at present with portable versions of new builds in my bog standard windows 7. I'm expecting to get some more modern machines soon so will be up to speed then, but do have a history of using legacy software simply cos it works.

Lastly in this issue, there are comments about xp, but as you have already dropped support there, don't let it mean that you also drop support for legacy software as well. I think blind people on average do hang on to older software longer than most simply to avoid the learning curve. I am now 67, and find my learning curve is somewhat harder to climb than when I was 25!

[josephsl]

Hi, I’ll make sure all app modules that ships with NVDA are properly ported, or if not completely, at least made compatible with both 2 and 3. Thanks.

[LeonarddeR]

How about the liblouis python bindings? According to tests by @roelvdwal, these are not yet py3 ready.

[josephsl]

Hi, uh oh, another setback for us. CC @JCSTeh, @DKager

[LeonarddeR]

Turns out Python 3 x64 was used for that test with an x86 dll, so no, that does not work indeed. :P Still a thing that should be tested.

[LeonarddeR]

Some thoughts and questions:

• I've done some quick investigations regarding the braille code. Especially native braille display drivers rely heavily on the byte nature of strings. ctypes functions like create_string_buffer will fail if their parameter isn't a bytes. We could make hwIo convert every single unicode string to latin-1, but I'm slightly concerned about the performance impact. Note that the new bytes type in python3 is quite charming, it allows using bytes(cells) instead of "".join(chr(cell) for cell in cells). Furthermore, when getting the int value from a byte string, you can just index the byte string, so ord(packet[3]) can be changed to just packet[3]. Many constants used in braille display drivers are saved as one byte strings, converted to ordinals when required, I propose changing byte constants to integers as much as possible, as with the new bytes class, it is much easier in the end to deliver them as bytes to the display.
• Is our aim to create a code base that is compatible with both versions of python, or is py2 incompatibility allowed? I assume that this has a major impact on how decisions are made.

@josephsl commented on 25 aug. 2017 04:15 CEST:

... making sure future versions of my add-ons are source-compatible with both 2 and 3.

• How do you check whether add-ons are source compatible, or do you just assume that they are? :)
• May be it is a good idea to lay the foundation for the transition by providing some NVDA specific wiki articles?

[josephsl]

Hi, since most add-ons use Python standard library, it can be checked via 2to3 and runtime checks (the code, that is) by running the affected code in Python 3 interpreter first. As for compatibility, eventually, only Python 3 should be supported, but for sake of ease of transition, compatibility should be maintained for about six months (two NVDA releases). Thanks.

[Brian1Gaff]

Not sure if this has been asked for before, but it would I think be very handy to have an archive section for add ons that work on 2017.3 so that xp users can still get hold of them. I was asked this the other day, and really could say no more than try one and see if it works.

I realise that the suitability of add ons is not strictly a version of the language issue and is also a change of support issue in the main nvda system, but thought it worth noting here as well as in the other thread about add ons. Brian

[josephsl]

Update: wxPython 4 is out (at least it is now up to 4.0.1), and given that #7104 is planned for 2018.2, I'd say let us designate 2018.2 as N release (start of transition). This means:

1. 2018.2: play around with wxPython 4.
2. 2018.3: more testing and research. Most external dependencies should be ready to go.
3. 2018.4: start of transition. More internal dependencies (such as Liblouis Python bindings) sould be ready or porting should be in progress.
4. 2019.1: Python 3 port should begin. Add-on porting becomes "optional". By this time, all dependencies must be ready, including Liblouis Python bindings.
5. 2019.2: Transition in progress, add-on porting will be set to "required".
6. 2019.3: more and more testing.
7. 2019.4: Python 2 code should be ready to leave this world (if decided to go that route).

Comments are appreciated. Thanks.

[LeonarddeR]

From what' I've seen, it looks like that py2exe does not support Python 3.6. Honestly, Pyinstaller never impressed me that much. Cx_Freeze looks promising though.

@Josephsl: Do you have any ideas or comments on this topic?

[josephsl]

I'm not really an expert on these. Sorry.

[LeonarddeR]

I did some quick tets with both pyinstaller and cx_Freeze, and it seems that the latter is prefered. Pyinstaller doesn't integrate into distutils, whereas cx_Freeze does.

The version of use would be version 5.1.1. Version 6 is currently in beta and targets Python 3.5+.

[josephsl]

I see. Can we create a new issue that references this one so we can do more research? I’m battling a problem with inability to add items in speech dictionaries at the moment. Thanks.

[josephsl]

July 2018 updates:

• N = 2018.3: this will serve as the "N" release where wxPython 4 is used. With this done, wxPython 4 dependency is satisfied.
• Research period: I propose that the community do the research this fall and early months of 2019.
• Python version: let us target Python 3.7.

As for my role: due to various reasons, I might not be able to provide active contributions for this one. But I'm sure there are many in the NVDA community who are ready to take this on and make it happen.

[Jitender46559]

How can i start contribution to this project. I have almost one year of experience in python 3X.

[larry801]

Please see contributing guide in wiki. https://github.com/nvaccess/nvda/wiki/Contributing Then you can help nvaccess process issues and pull requests. This group may also be helpful. https://nvda.groups.io/g/devlearning @Jitender46559

[josephsl]

Update as of April 15, 2019: several add-ons are Python 3 ready or are being ported to Python 3. Broad transition won't begin until NV Access officially declares start of Python 3 transition phase.

[feerrenrut]

We will be targeting python 3.7, at this stage we expect to do at least one more releases of NVDA (possibly two) before we do a python 3 release of NVDA. We want to make it as simple as possible to find and fix issues caused bt high risk changes. So the aim is to have the minimal changes, just those related to python 3 and the speech refactor PR.

There has been investigation into the packaging tool, and some success with py2exe. See #8375

Some thought is currently being put into how to keep the threshold branch python 3 compatible. We need to be able to stay on top of changes coming from master (bug fixes, new features). There are also many PR's that will not be merged. Some way to detect problems in these changes sets (and ideally automatically rectify) will save a lot of time and mistakes.

2to3 (From python 3.7) to automatically apply updates has been considered, but has not shown much promise. The changes need to be inspected carefully. Running the tool a second time seems to further modify the code making unnecessary changes. This is problematic, since we need to be able to merge another branch then run the tool again. Perhaps if we could get it to run just on a diff / patch?

We are considering adding the -3 argument to our python 2 build, this should highlight problems in master which can be made compatible with both 2 and 3. This will reduce the burden of merges, however it is likely to push some risk back into prior releases.

[LeonarddeR]

We are considering adding the -3 argument to our python 2 build, this should highlight problems in master which can be made compatible with both 2 and 3. This will reduce the burden of merges, however it is likely to push some risk back into prior releases.

Hmm, this seems like an interesting idea, though I wonder whether py2exe can deal wit this while freezing the code. That needs to be found out. It is probably very trivial to add the -3 flag to test builds only, so release builds don't get affected.

[feerrenrut]

I'm going to close this issue now, as I think it has served it's purpose. It has been superceded by the Python 3 project board