Upgrade default QGIS python from 3.9 to latest stable version

[AmirsamanAhmadi]

Feature description

Dear QGIS Team,

I hope this message finds you well. I am writing to request an upgrade of the default Python version used in QGIS from 3.9 to the latest stable version, which is 3.11.4 at the time of writing.

I understand that QGIS is a complex and multifaceted open-source GIS platform, and I genuinely appreciate all the hard work that the team puts into maintaining and improving it. Python plays a critical role in extending the functionality of QGIS through plugins and scripting, and using the latest Python version can provide numerous benefits, including performance improvements and access to new features and libraries.

I attempted to build QGIS against Python 3.11 myself, but unfortunately, I encountered difficulties and could not successfully complete the process. I believe that many other users may face similar challenges, especially those who rely on QGIS as a fundamental tool in their work. As such, having the latest Python version bundled with QGIS would streamline the installation process and ensure that users have access to the most up-to-date Python environment without the need for manual configuration.

I am aware that some users choose to use QGIS with Anaconda, which allows for greater flexibility in managing Python environments. However, in our organization, we have internal policies and licensing constraints that prevent us from using QGIS with Anaconda. Therefore, having the default QGIS distribution support Python 3.11 would be immensely valuable to us and many other users who share similar restrictions.

I understand that software development and integration decisions are complex and require careful consideration. Still, I kindly request that you evaluate the feasibility of upgrading the default Python version in QGIS to 3.11 in a future release. Your efforts in this regard would be greatly appreciated by the QGIS user community.

Thank you for your time and dedication to the QGIS project. I look forward to any updates or information you can provide regarding this request.

Additional context

No response

[jef-n]

What is the issue with 3.9?

[nyalldawson]

There's big performance optimisations in 3.10+11, for one 👍

[AmirsamanAhmadi]

@jef-n The primary concern with Python 3.9 is its potential limitations when it comes to accessing the latest features and enhancements in libraries and packages. As the Python ecosystem evolves, these libraries and packages are updated to work seamlessly with newer Python versions. Staying on Python 3.9 could restrict our ability to leverage these advancements, which may have an impact on our development and productivity. Therefore, it would be great if we could upgrade to the latest stable version to future-proof our codebase and ensure compatibility with the ever-evolving Python ecosystem.

[nicogodet]

I would be more than happy to try a python update on qgis-dev if this is feasible.

[agiudiceandrea]

@nicogodet, the latest MingW64 Windows 64bit builds use Python 3.11.4. The only issue I've seen till now is that the MetaSearch Plugin crashes.

[jef-n]

@nicogodet, the latest MingW64 Windows 64bit builds use Python 3.11.4. The only issue I've seen till now is that the MetaSearch Plugin crashes.

QGIS is not tied to a specific python version. Most - if not all - linux builds use 3.10 and above - and conda obviously. For OSGeo4W updating to 3.11 would mean that not only the python package has to be updated, but also all of the python c extensions would have to be rebuild, because they use binary dependencies from OSGeo4W. For the lighter dependencies that happens more often (on updates of GDAL, GEOS, PROJ, PDAL, SQLite etc.) anyway, but not for the heaviest one (Qt / PyQt5). But that may be become necessary soon as we're still using OpenSSL 1.1.1 in Python 3.9 and Qt and that's going to be EOL shortly. So that might become one big update.

Hm, and Python 3.12 is expected to be released on October 2nd.

[Gustry]

Thanks for raising this ticket.

Is there a repository where we can find the Python version when packaging QGIS ? (like the one for QGIS 3.28 which is widely used) ? @jef-n Is this file in the Osgeo4W correct for instance ? https://github.com/jef-n/OSGeo4W/blob/master/src/python3/osgeo4w/package.sh#L2

The CMakeList file in QGIS master is mentioning Python 3.7 minimum version : https://github.com/qgis/QGIS/blob/bc6fdf73a8a162c652e0b6e756bd1afc0594170f/CMakeLists.txt#L1036

[Gaudi111]

@jef-n Hi, I would like to second the request. I have been informed by some colleagues which performed a vulnerability analysis on the tool the following report:

The current detected version for QGIS v3.28.11.1 is 3.9.5, and it has some identified vulnerabilities in its code, namely the Improper Input Validation vulnerability (see link here: https://security.snyk.io/vuln/SNYK-ROCKY9-PYTHON39-5878551) Description: An issue in the urllib.parse component of Python before 3.11.4 allows attackers to bypass blocklisting methods by supplying a URL that starts with blank characters.

Suggestion: upgrade to Python version >3.11.

Comments on this issue are appreciated.

Update: As per article here: https://thesecmaster.com/how-to-fix-cve-2023-24329-url-parsing-issue-in-python/#How_to_Fix_CVE-2023-24329_%E2%80%93_URL_Parsing_Issue_in_Python updating to Python 3.9.17 will address the urllib.parse vulnerability.

[readmanr]

Hi,

Microsoft 365 Defender is now flagging QGIS 3.28.14-1 and QGIS 3.34.2-1 as Vulnerable with 15x discovered vulnerabilities due to Python 3.9.5.

CVE-2022-37454 Critical CVE-2015-20107 Critical CVE-2023-40217 High CVE-2023-24329 High CVE-2022-45061 High CVE-2022-26488 High CVE-2020-10735 High CVE-2018-25032 High CVE-2023-27043 Medium CVE-2021-3737 Medium CVE-2021-28861 Medium CVE-2016-3189 Medium CVE-2019-12900 Medium CVE-2013-0340 Medium CVE-2007-4559 Medium

I won't link to each CVE as I know you are more than capable, from my check of each CVE, Python requires an update exceeding version 3.11.5 see https://nvd.nist.gov/vuln/detail/CVE-2023-40217 EDIT: I read the CVE's incorrectly and missed there was a line for each B version. An update to 3.9.18 will address the majority of the CVE's with https://nvd.nist.gov/vuln/detail/CVE-2023-40217 marked the highest on the 3.9.x branch. CVE-2023-27043 does not have the branched versions listed like the other 14x see https://nvd.nist.gov/vuln/detail/CVE-2023-27043 it stages 3.0 to 3.11

MingW64 MSYS2 packages offering 3.11.7 at this time. https://packages.msys2.org/base/mingw-w64-python

[aclondon]

Hi,

Microsoft 365 Defender is now flagging QGIS 3.28.14-1 and QGIS 3.34.2-1 as Vulnerable with 15x discovered vulnerabilities due to Python 3.9.5. I won't link to each CVE as I know you are more than capable, from my check of each CVE, Python requires an update exceeding version 3.11.5 see https://nvd.nist.gov/vuln/detail/CVE-2023-40217

This is now a requirement for CE+ otherwise organisations will fail. As the vulnerabilities are now flagged as Critical meaning any organisations following CE/ISO/NIST/SOC guidelines will fail at this time. QGIS needs to update Python shared libraries to 3.9.16 minimum to resolve this issue. https://python-security.readthedocs.io/vuln/sha3-buffer-overflow.html

[AScott-WWF]

+1 to this In addition to this ticket originally raised over 4 months ago - I raised a ticket with OSGEO yesterday (as we too are having MDE (Microsoft Defender for Endpoint) alert us of the existence of Critically vulnerable Python 3.9.5 installs as a result of QGIS / OSGEO4W installs on a number of our users devices). FYI: Python 3.9.5 (released 3rd May 2021) currently has at least 2 Critical CVEs with CVSS of 9.8 (about as high as the scale will go)

• Ticket here: https://trac.osgeo.org/osgeo4w/ticket/811

Sadly this still affects a fresh install of QGIS LTR 3.28.14

PLEASE can QGIS and OSGEO be updated ASAP to support either a security bug fixed Python 3.9.x branch or be updated to a later Python build (I incorrectly mentioned in the OSGEO ticket 3.13 is due for release any day now - I misread the US date format, it's actually not due until October 2024)

Thanks in advance for you urgent attention

[greaman]

It apparently will also affect new installs auf LTR 3.34 - technically will have to force deinstallation of QGIS as the software poses a severe security risk. As 3.9.5 receives no regular updates since a while ago und the only updates beyond 3.9.10 have been security updates which are actually quite a few, that situation exists since quite a while - since MDE is flagging it prominently it simply becomes more transparent.

[readmanr]

Jürgen upstream on OSGeo4W has increased the Python version to 3.9.18 yesterday - Jan 14, 2024.

See commit c290dc6 https://github.com/jef-n/OSGeo4W/commit/c290dc6c5d7d2ab4cbe0a5bb256d05839189633c

[Gustry]

Jürgen upstream on OSGeo4W has increased the Python version to 3.9.18 yesterday - Jan 14, 2024.

Thanks @jef-n Is-it possible to know which version will be impacted ? QGIS 3.28.15 and 3.34.3 if I'm correct.

[AScott-WWF]

FWIW, I have just re-tested the OSGEO4W installer as follows: Fresh install on a Windows Sandbox (i.e. no existing QGIS LTR install)

• As I have mentioned on the OSGEO issue: https://trac.osgeo.org/osgeo4w/ticket/811#comment:3, The OSGEO4W installer now correctly extracts / installs the latest security patched Python 3.9.18 alongside QGIS 3.28.14 LTR, so for new installs this is great as it resolves the Security Vulnerabilities for new installs.

• Secondly I have just performed an update / upgrade of an existing QGIS LTR (3.28.13) install using the OSGEO4W installer and it successfully detects an old Python 3.9.5 install, so uninstalls this old version of Python before installing 3.9.18 (and QGIS 3.28.14 LTR) - so for upgrade installs this also resolves the Security Vulnerabilities

Thanks to jef from OSGEO for such a rapid turn around on this.

One small cosmetic issue with QGIS either after a fresh install or after the update with the latest 3.9.18 python is the Help | About dialog still shows it has Python version 3.9.5 installed - even though I now know it is 3.9.18 Here is a screen grab:

I assume someone at QGIS can fix this?

[aclondon]

FWIW, I have just re-tested the OSGEO4W installer as follows: Fresh install on a Windows Sandbox (i.e. no existing QGIS LTR install)

• As I have mentioned on the OSGEO issue: https://trac.osgeo.org/osgeo4w/ticket/811#comment:3, The OSGEO4W installer now correctly extracts / installs the latest security patched Python 3.9.18 alongside QGIS 3.28.14 LTR, so for new installs this is great as it resolves the Security Vulnerabilities for new installs.
• Secondly I have just performed an update / upgrade of an existing QGIS LTR (3.28.13) install using the OSGEO4W installer and it successfully detects an old Python 3.9.5 install, so uninstalls this old version of Python before installing 3.9.18 (and QGIS 3.28.14 LTR) - so for upgrade installs this also resolves the Security Vulnerabilities

Not being a user of the software, just the admin that manages security of our infrastructure! Does this mean that we can install QGIS as separate package from within OSGEO4W or do we still need to wait for a separate install package from QGIS (with updated Python) ?

Many thanks.

[agiudiceandrea]

One small cosmetic issue with QGIS either after a fresh install or after the update with the latest 3.9.18 python is the Help | About dialog still shows it has Python version 3.9.5 installed

I guess that it is the version of Python used when the QGIS binaries were built.

[AScott-WWF]

@aclondon, I use the OSGEO4W installer as it is designed for IT administrators, the main benefit for us is it keeps the install in the same folder structure rather than creating a new folder structure for each QGIS release. The OSGEO 'installer' is linked on this page: https://www.qgis.org/en/site/forusers/download.html - follow the OSGeo4W Network Installer link

The Install script I use to install the latest QGIS LTR version is within the OSGEO ticket I raised: https://trac.osgeo.org/osgeo4w/ticket/811 You can re-run this at anytime and it will update any components that have been updated - and as I've discovered today - Python :-)

I repackage the script and distribute via Intune to any clients that have QGIS installed Hope this helps?

[Gaudi111]

Thank all for the follow up and comments. @AScott-WWF, based on your knowledge, is there any way to package/build an offline installer to deploy? At work we are not able to download external packages during the installation process, but rather need to have a full offline installation. Is this possible or should I need to wait until a new build is posted?

Regards

[jef-n]

The standalone installers are made from OSGeo4W packages - next release is on friday. See https://www.qgis.org/en/site/getinvolved/development/roadmap.html#schedule.

[jef-n]

Thank all for the follow up and comments. @AScott-WWF, based on your knowledge, is there any way to package/build an offline installer to deploy? At work we are not able to download external packages during the installation process, but rather need to have a full offline installation. Is this possible or should I need to wait until a new build is posted?

You can also use the OSGeo4W-Installer to download the packages only and install them offline.

[Gaudi111]

Thank you for the useful information. Will investigate and provide instructions to our IT Team.

Best regards

[readmanr]

A big thank you for acting on this so promptly. The version increase to 3.9.18 immediately addresses 14 of the listed CVE's. The one exception I am unsure of is CVE-2023-27043 This CVE on nist does not define the 3.8.x, 3.9.x, 3.10.x branch versions individually like the other CVE's. It simply states From 3.0 Up To (Including) 3.11. See https://nvd.nist.gov/vuln/detail/CVE-2023-27043

[AScott-WWF]

I've been having a bit of a dig into that CVE, looks like there has been a lot of back and forth with the Python developers and the person that raised the issue. The last comment in here (https://github.com/python/cpython/issues/102988) 'appears' to infer that CVE is going to be back ported to older releases of Python once Python v3.13 has been released for a while (slightly non-committal). According to the Downloads page (https://www.python.org/downloads/) Python v3.13 is not due to be released 'til 1st October 2024... Personally - I kind of feel that the Python developers are kicking this particular CVE into the long grass (shrug)

[aclondon]

@aclondon, I use the OSGEO4W installer as it is designed for IT administrators, the main benefit for us is it keeps the install in the same folder structure rather than creating a new folder structure for each QGIS release. The OSGEO 'installer' is linked on this page: https://www.qgis.org/en/site/forusers/download.html - follow the OSGeo4W Network Installer link

The Install script I use to install the latest QGIS LTR version is within the OSGEO ticket I raised: https://trac.osgeo.org/osgeo4w/ticket/811 You can re-run this at anytime and it will update any components that have been updated - and as I've discovered today - Python :-)

I repackage the script and distribute via Intune to any clients that have QGIS installed Hope this helps?

Thanks for the script. I will have a play and see if can package a IntuneWIN file. Should be able to download the packages first and then just use the package to install them locally.

[AScott-WWF]

A big thank you for acting on this so promptly. The version increase to 3.9.18 immediately addresses 14 of the listed CVE's. The one exception I am unsure of is CVE-2023-27043 This CVE on nist does not define the 3.8.x, 3.9.x, 3.10.x branch versions individually like the other CVE's. It simply states From 3.0 Up To (Including) 3.11. See https://nvd.nist.gov/vuln/detail/CVE-2023-27043

I guess updating QGIS & OSGEO to Python 3.12.x would probably hide this vulnerability if the detection is based purely on the NIST info. However - We now know this vulnerability won't actually get fixed until v3.13.x is released in October:

The last comment in here (https://github.com/python/cpython/issues/102988) 'appears' to infer that CVE is going to be back ported to older releases of Python once Python v3.13 has been released for a while (slightly non-committal).

[AScott-WWF]

One small cosmetic issue with QGIS either after a fresh install or after the update with the latest 3.9.18 python is the Help | About dialog still shows it has Python version 3.9.5 installed

I guess that it is the version of Python used when the QGIS binaries were built.

I have just updated to QGIS LTR 3.28.15 this morning, the Python version is now showing as 3.9.18 - Issue resolved:

[readmanr]

This original 54491 was to request an upgrade to Python for feature/performance improvements. It appears that Microsoft 365 Defender at least, does list the vulnerability purely based on the NIST info and the version of files. The 1 CVE is Medium, and reading the linked cpython issue AScott-WWF mentioned is being managed as low/medium severity + in fact probably not even affected in most use cases.

Depending on the underlying efforts for new releases are what will determine if doing new releases for updating dependencies is a no-brainer or a heap of work for little reward.

Python does offer security release versions for a period of 5 years from the initial branch release. However Active Support releases with bugfixes are only issued in the first 1.5 years (to be increased to 2 years starting with Python 3.13).

The current use of Python 3.9 - released 3 years ago (5th October 2020) had active support end over a year and a half ago on the 17th of May 2022.

As 54491 requests to increase to 3.11 and with this version ending active support / bug fixes in just over 2 months on the 1st of April. I leave it to the powers that be to decide if 3.11 would be more stable vs the more recently released 3.12 which will have Active Support through to April 2025. With most of these scenarios, the longer they are left, the more work is usually required to perform the update.

[AScott-WWF]

Entirely agree @readmanr IMHO a move to Python 3.12.1 (or later) would be better for the long term

[Guts]

Upgrading Python version to latest patched version 3.9.x sounds a good practice.
Following Python major versions (at least for Windows package) sounds a good move. Even if, as being maybe too cautious, I would prefer to follow major version minus 1 (i.e. 3.11.x when 3.12.x is released).
But I think that changing Python major version in the middle of a QGIS minor version life-cycle is a bad idea regarding the plugins ecosystem.

Make the Python version something predictable (again, mainly for Windows package) enforces QGIS ecosystem. We could go for a relationship between QGIS LTR and Python stable versions:

QGIS	Python
3.28	3.9
3.34	3.10
3.40	3.11

And so being...

[lbartoletti]

Thanks for raising this ticket.

Is there a repository where we can find the Python version when packaging QGIS ? (like the one for QGIS 3.28 which is widely used) ? @jef-n Is this file in the Osgeo4W correct for instance ? https://github.com/jef-n/OSGeo4W/blob/master/src/python3/osgeo4w/package.sh#L2

Yes, and I think we can separate the OSgeo4W things and QGIS requirement. We should be able to open issue on OSGeo4W project ; afaik @jef-n it's “your” project.

The CMakeList file in QGIS master is mentioning Python 3.7 minimum version :

https://github.com/qgis/QGIS/blob/bc6fdf73a8a162c652e0b6e756bd1afc0594170f/CMakeLists.txt#L1036

Right! I vote to update this minimum version once OSgeo4W will update the python version.

But, as a packager, I'm concerned to note update directly to 3.12. We have to be compatible with latest release, but we can't enforce it, since some systems/distributions may not have 3.12 as Python default version.

So, OK for a min-version progressively. Maybe 3.10 or 3.11 for the next LTR.

[Gustry]

Thanks @lbartoletti for your PR. I agree with @Guts with his message above, as a Python plugin developer, it seems important to have some "guidelines" somewhere about which Python version we can safely use when we deliver a plugin to a customer. I tend to be very careful when developing and to rely only on this hidden variable in the "CMakeLists.txt".

I would be in favor for instance to expose the minimum Python version required on the PyQGIS API documentation https://qgis.org/pyqgis/3.34/ There isn't any mention for now, I can have a look on that (maybe in Grenoble at the next QGIS code sprint)

[Guts]

I would be in favor for instance to expose the minimum Python version required on the PyQGIS API documentation https://qgis.org/pyqgis/3.34/ There isn't mention for now, I can have a look on that (maybe in Grenoble at the next QGIS code sprint)

+1!

[WillB1985]

At the time of writing, Defender shows up a python 3.9.18 as vulnerable. CVE-2023-27043. Is there a plan to update it?

[jef-n]

At the time of writing, Defender shows up a python 3.9.18 as vulnerable. CVE-2023-27043. Is there a plan to update it?

To what? That vulnerability should still apply to all released python versions. Only unreleased Python 3.13 has a fix.

[AScott-WWF]

At the time of writing, Defender shows up a python 3.9.18 as vulnerable. CVE-2023-27043. Is there a plan to update it?

To what? That vulnerability should still apply to all released python versions. Only unreleased Python 3.13 has a fix.

According to the ongoing Python discussion about fixing this issue (https://github.com/python/cpython/issues/102988#issuecomment-1957298613), they 'might' back port the fix from 3.13 beta 2 once it has been made available (Beta 1 is due out in May 2024, but I'm unsure of the Beta 2 schedule). I personally would not hold them to this - it is a low priority / low severity security issue, and I'm certain they have bigger fish to fry

[danceb]

I came across this issue as the update from 3.34.5 to 3.34.6 resulted in plugin errors because of missing libs in python 3.12. I understand, that you want to consider security issues. But I would appreciate it very much, if upgrades to newer stable versions of underlaying packages are linked to new QGIS ltr versions in the future, as @Guts suggested. It is a bit unfortunate to run in such unexpected issues with updates of a minor release...

[prusswan]

I came across this issue as the update from 3.34.5 to 3.34.6 resulted in plugin errors because of missing libs in python 3.12. I understand, that you want to consider security issues. But I would appreciate it very much, if upgrades to newer stable versions of underlaying packages are linked to new QGIS ltr versions in the future, as @Guts suggested. It is a bit unfortunate to run in such unexpected issues with updates of a minor release...

This was surprising to me as well, including the sudden jump from 3.9.18 to 3.12.x without anything in between for the OSgeo4W build. Bold move cotton (j/k)

[nyalldawson]

@prusswan there's NEVER been any promise that library versions won't change throughout the lifespan of an LTR. You need to adapt your processes accordingly.

[prusswan]

@prusswan there's NEVER been any promise that library versions won't change throughout the lifespan of an LTR. You need to adapt your processes accordingly.

Still a pretty hasty transition from 3.9.5 to 3.9.18 (briefly) then to 3.12.4. It is not only the LTR affected btw. Since I can't fix everything for 3.12, I will have to request users to use the last QGIS version with Python 3.9 until further notice.

As I am using OSGeo4w build, I rely on the roadmap and OSGeo instructions and changes to osgeo4w/package.sh to determine what I need.

Last versions using 3.9 are (3.36.1 | 3.34.5). I went with this snapshot url (https://download.osgeo.org/osgeo4w/v2/snapshots/20240322-185340/) to install 3.36.1

[ptitjano]

@prusswan there's NEVER been any promise that library versions won't change throughout the lifespan of an LTR. You need to adapt your processes accordingly.

@nyalldawson On could also argue that it is never written anywhere that a patch upgrade of an LTR might upgrade the minor or major version of a dependency. This goes against what most people think of an LTR release.

That being said, I am well aware that this is difficult problem to handle. We should at least write somewhere that the major version of a dependency might change throughout the lifespan of an LTR. Where would be the best place to document it?

In the longer term, I think we should only upgrade to patched version during the lifespan of an LTR as @Guts suggested in https://github.com/qgis/QGIS/issues/54491#issuecomment-1907829509

[prusswan]

Does QGIS even have a Python upgrade policy at this time? Afaik it was on 3.9.5 for a fairly long time, sure an upgrade to 3.11 or 3.12 was due but this is just not properly communicated/managed. It should have been featured in the changelog or on the website. Many of our users are not developers and should not have to find out the "hard" way from Github.

Something like this at the very least: https://developers.arcgis.com/python/latest/guide/system-requirements/

[agiudiceandrea]

Probably it is not clear that QGIS itself doesn't ship Python or GDAL or PROJ or GEOS or other needed libraries. It needs a minimum version of such libraries in order to be correctly built and to work on a system. The actual library version that QGIS is built against and that it uses depends on the available library version on such system. On some systems (like Windows) all the needed libraries are provided by a dedicated installer created by OSGeo4W but the needed libraries can also be provided by a environment management system like Conda and they can be different from those provided by OSGeo4W.

Just some examples of the different Python versions used by the same QGIS 3.34.11 on various systems currently:

Windows: OSGeo4W installer 3.12.6 / Conda-forge 3.12.7 (or previous versions) Ubuntu: jammy 3.10, mantic 3.11, noble 3.12 Debian: bullseye 3.9, bookworm 3.11, sid 3.12 ArchLinux: 3.12 Flatpak: 3.11 macOS: dmg installer 3.9.5 / Conda-forge 3.12.7 (or previous versions) / MacPorts 3.12 (or previous versions) ...

[prusswan]

QGIS itself doesn't ship Python or GDAL or PROJ or GEOS or other needed libraries.

Thanks for the explanation. Now I get that the same QGIS version can be built with/on different versions of Pythons across different OSes, making the situation rather complicated. Still, the normal user will believe QGIS comes with a working Python environment included (built-in console, or through the OSGeo4w shell), they just don't have much of a choice over the Python version unless they are capable of building QGIS including dependencies.

[nyalldawson]

@prusswan

Still, the normal user will believe QGIS comes with a working Python environment included (built-in console, or through the OSGeo4w shell)

Correct, and they always do.

they just don't have much of a choice over the Python version unless they are capable of building QGIS including dependencies.

Correct. It's not something the end user can/should be in control over if they are using the official installers. You'll just need to ensure your plugins and scripts and processes are flexible enough to handle this.