Python 3.9 readiness: check for and mitigate deprecation warnings and code removal

[josephsl]

Hi,

To serve as a portal to related issues. As such, any pull requests stemming from discussion below should address specific, concrete issues.

Background

Python 3.9, scheduled for October 2020 release, plans to drop old remains of Python 2.x, given that Python 2.7 is now part of history. To help developers with this endeavor, older Python 3 releases (including 3.7) emits deprecation warnings when encountering items marked for deprecation in 3.9 such as no more collections.mapping import (seen in configobj).

Is your feature request related to a problem? Please describe.

As Python development advances, NVDA and its dependencies will need to keep deprecations in mind, especially when it concerns backwards incompatible changes. As noted above, Python 3.7 logs deprecation warnings for things to be removed in Python 3.9 and later, which will affect NVDA, dependencies, and add-on code.

Describe the solution you'd like

Catalog code paths where deprecation warnings are raised so developers can be notified:

• NVDA: NV Access via GitHub issues
• Dependencies: module authors
• Add-ons: add-on authors

Describe alternatives you've considered

Keep the code as is.

Additional context

To see deprecation warnings, one must either:

1. Run NVDA from source: py -3.7 -Werror path/to/source/nvda.pyw
2. From Python console: import wanrings; warnings.filterwarnings("error"); warnings.filterwarnings("always")

The first option will prevent NVDA from running if errors are found, and the second option is useful for add-on authors as errors will be printed to the log. As this issue and related ones may bring up compatibility changes, I suggest concrete implementations should target 2021.1 unless early integration can be justified (such as dependency update).

Thanks.

[Brian1Gaff]

Does this new version of Python 3 offer any concrete improvements for nvda over the current python 3 version in use? I seem to recall a question like this being raised during the Python 2.x life span.

[josephsl]

Hi, Python 3.9 includes changes from 3.8, which may benefit NVDA in the long term.

[feerrenrut]

We may consider upping the python version with the 2021.1 release. In the mean time I believe we already get deprecation warning for things that are removed in python 3.8. These should be addressed first. Since the differences between versions are much more minor than the 2 to 3 update we should be able to have NVDA working across both versions way in advance. We probably will target 3.8 rather than 3.9 so that there is time for this and our dependencies to be updated.

[dpy013]

hi @josephsl python3.9 Is there a significant increase in operating speed?

[josephsl]

Hi, as for that, we shall see in the future. Thanks.

[josephsl]

Hi, I think we can do that first – when I run source code copy with -Werror flag, there are invalid escape sequence issues in some files (backslash W, for instance). Eventually we should also address deprecation of Yield function in wxPython. Thanks.

[feerrenrut]

I'd be happy to take PR's for these. Though please implement and test them independently, it's much easier to ensure a set of small changes is tested well than it is for a single large change. It's also easier to revert small change sets if they cause a problem.

[josephsl]

Hi, will do (note that an upcoming issue/PR pair will concern configobj). Thanks.

[LeonarddeR]

I agree with @feerrenrut that we're probably better to target Python 3.8, as it is unlikely that all our dependencies are 3.9 ready when 2021.1 is planned. Also agree that work on this wouldn't hurt us.

[josephsl]

Hi,

A few months later...

Ideally 3.9 would be better due to Python 3.8 support status. The most prominent thing left is wxPython - 4.1.0 includes a Python 3.8 wheel, whereas 4.1.1-dev also includes a 3.9 wheel. For practicality reasons, I'm in agreement that 3.8 would be best for now, with this issue focusing on 3.9 issues (to be linked from a new proposal for upgrading to Python 3.9).

Thanks.

[LeonarddeR]

As py2exe just updated to a version that supports 3.9, I don't think that apart from wx, there's not much holding us back from targetting 3.9 if we have an expected time frame for the release of wxPython 4.1.1

[josephsl]

Hi, don’t forget about brlapi and other dependencies with a C extension. Thanks.

[lukaszgo1]

Please also note that Python 3.9 drops support for Windows 7/Server 2008. While it might be tempting to use Python 3.9 migration as an opportunity for dropping support for these os-es I believe we need to consider how many our users are still using them before the migration starts. @feerrenrut Would you be able to share statistics collected by NV Access regarding Win7/Server 2008R2 usage?

[LeonarddeR]

I think we should be very hesitant to drop support for Windows 7 and Server 2008 R2 for at least another year,

[dpy013]

hi In addition, appveyor now officially supports python 3.9. Here are the support notes

[josephsl]

Hi, one test we can do is build a binary version of Python 3.9 based NVDA and attempt to run it on Windows 7. I bet this will work, otherwise we should stick with 3.8 and wait for 3.10 in fall 2021. Thanks.

[josephsl]

Hi,

It might be my set up, but when trying to launch Python 3.9 based NVDA build on Windows 7, I get hung program window (title: NVDA application has stopped working). With Python 3.8 build, at least the launcher closes running NVDA before Windows shows hung program window. I suspect it may have to do with dependency issue - Py2exe, Numpy, pywin32, wxPython, etc.

AT least for 2021, Python 3.8 will work. Note that binary builds will be discontinued soon, making it harder for developers to obtain latest 3.8 minor release unless we are willing to build Python ourselves or Appveyor guarantees support for security releases. I'm in favor of terminating support for Windows 7/Server 2008 R2 in 2022.1 cycle, meaning we must skip 3.9 in favor of 3.10. This issue will show up again in 2023 when supposed Python 3.12 shows up, which will drop support for Windows 8.x and Server 2012/2012 R2.

Before I move to take this off the radar for a while, we need solid data from update checks proving overwhelming usage of Windows 7/Server 2008 R2. Even if 1 percent of people claim they use NVDA on Windows 7, it is quite significant, as it may indicate that actual usage is higher (think about corporate environments), giving us a possible reason for closing this issue in favor of a new one for Python 3.10 in 2021. If usage is at least 5 percent, then there's no response except to move onto 3.10 (at least for me). Nevertheless, we do need to keep an eye on code maintenance in preparation for future Python releases, including dependencies.

Thanks.

[feerrenrut]

10% of users using 2020.3 are on Windows 7 / Windows Server 2008R2 10% of users with recent NVDA versions (2020.3, 2020.2, 2020.1, 2019.3 and patch releases too) are on Windows 7 / Windows Server 2008R2 17% of all users (including those with old versions of NVDA) are on Windows 7 / Windows Server 2008R2

I don't think we should consider dropping support for Windows 7 unless we really must.

[feerrenrut]

Please also note that Python 3.9 drops support for Windows 7/Server 2008.

This is a bit confusing, the python docs say:

As specified in PEP 11, a Python release only supports a Windows platform while Microsoft considers the platform under extended support. This means that Python 3.9 supports Windows Vista and newer.

But then on the release page it says:

The installer now also actively disallows installation on Windows 7. Python 3.9 is incompatible with this unsupported version of Windows.

PEP 11 says:

Microsoft has established a policy called product support lifecycle [1]. Each product's lifecycle has a mainstream support phase, where the product is generally commercially available, and an extended support phase, where paid support is still available, and certain bug fixes are released (in particular security fixes).

CPython's Windows support now follows this lifecycle.

The Microsoft Product support lifecycle page for Windows 7 says support for Windows 7 and Windows 7 sp1 ended on 14/01/2020.

[LeonarddeR]

The Microsoft Product support lifecycle page for Windows 7 says support for Windows 7 and Windows 7 sp1 ended on 14/01/2020.

Note the table below that that contains this note:

Extended Security Updates (ESU) are available through specific volume licensing programs for Windows 7 Professional, Enterprise, and Professional for Embedded Systems editions of this product, for up to an additional three years past the end of support.

May be it is worthy to file an issue against Python to point them at this inconsistency.

[feerrenrut]

There is the following issue talking about correcting the python 3.9 docs: https://bugs.python.org/issue42523

I think it would be a good idea to create an issue to point this out.

[seanbudd]

gui.dpiScalingHelper.scaleSize returns Union[float, Tuple[float, float]] which causes many python deprecation warnings when passing these values into certain wx functions. It would be good to change this to return Union[int, Tuple[int, int]] or create a new method if any usages of scaleSize expects floats.

This may be fixed by wxPython: https://github.com/wxWidgets/Phoenix/issues/1678

For example:

DEBUGWARNING - Python warning (17:08:44.053) - MainThread (1824):
C:\nvda\source\gui\nvdaControls.py:51: DeprecationWarning: an integer is required (got type float).  Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python.
  wx.ListCtrl.__init__(self, parent, id=id, pos=pos, size=size, style=style)
DEBUGWARNING - Python warning (17:08:44.067) - MainThread (1824):
C:\nvda\.venv\lib\site-packages\wx\lib\scrolledpanel.py:112: DeprecationWarning: an integer is required (got type float).  Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python.
  wx.ScrolledWindow.__init__(self, parent, id,
DEBUGWARNING - Python warning (17:08:44.068) - MainThread (1824):
C:\nvda\.venv\lib\site-packages\wx\lib\scrolledpanel.py:116: DeprecationWarning: an integer is required (got type float).  Implicit conversion to integers using __int__ is deprecated, and may be removed in a future version of Python.
  self.SetInitialSize(size)

[feerrenrut]

A while ago, wxWidgets was trying make this kind of scaling happen automatically. If that is done, we might be able to remove this custom solution.

[lukaszgo1]

Since update to a newer version of Python is covered by #12064 is there any point in keeping this issue open?

[seanbudd]

Closing in favour of #12064