Add python 3.12 support

ni-jfitzger commented 4 months ago

[X] This contribution adheres to CONTRIBUTING.md.
[X] I've updated CHANGELOG.md if applicable.
[ ] ~I've added tests applicable for this pull request~

What does this Pull Request accomplish?

There are 3 major changes made to add Python 3.12 support.

Adds official Python 3.12 support to nimi-python through testing. Based on:
- 1903
- 1944

I considered updating the version of Python used in the .readthedocs.yaml, but I'd rather be intentional about changes to documentation.

Replace usage of pkg_resources in __init__.py with importlib.metadata for Python 3.10 and later.

Tests failed on Python 3.12 because we tried to import a non-built-in module (pkg_resources) that we don't declare a dependency on. Full explanation below.

easy_install, pkg_resources, setuptools and distutils are no longer provided by default in environments created with venv or bootstrapped with ensurepip, since they are part of the setuptools package. For projects relying on these at runtime, the setuptools project should be declared as a dependency and installed separately (typically, using pip).

https://docs.python.org/3/whatsnew/3.12.html

Why the if/else implementation?

I don't want to add a setuptools dependency or a importlib_metadata dependency (backport of importlib.metadata)
Python 3.10 is the earliest version where importlib.metadata offers functionality equivalent to what we were doing with pkg_resources
Unlike pkg_resources, importlib.metadata is built-in.

I created #2047 to track the work to drop pkg_resources usage with Python 3.9.

Update grpcio, grpcio-tools, protobuf deps for Python 3.12 compatibility and regenerate .pb2 files

grpcio-tools compatibility for version 1.59.0 (earliest version with Python 3.12 support):

grpcio >= grpcio-tools version
protobuf >=4.21.6, < 5.0

List issues fixed by this Pull Request below, if any.

Fix #2044

What testing has been done?

PR Checks

Did a before and after prettyprint of nise.get_diagnostic_information() on a local test machine.

Using Python 3.12 before:

>>> pprint.pprint(nise.get_diagnostic_information())
{'driver': {'name': 'NI Switch Executive', 'version': '23.8.0.49182'},
 'module': {'name': 'nise', 'version': '1.4.7'},
 'os': {'bits': '64', 'name': 'Windows', 'version': '10.0.19045'},
 'python': {'bits': '64',
            'is_venv': False,
            'packages': [{'name': 'cachetools', 'version': '5.3.3'},
                         {'name': 'chardet', 'version': '5.2.0'},
                         {'name': 'colorama', 'version': '0.4.6'},
                         {'name': 'distlib', 'version': '0.3.8'},
                         {'name': 'filelock', 'version': '3.13.4'},
                         {'name': 'hightime', 'version': '0.2.2'},
                         {'name': 'nise', 'version': '1.4.7'},
                         {'name': 'packaging', 'version': '24.0'},
                         {'name': 'pip', 'version': '24.0'},
                         {'name': 'platformdirs', 'version': '4.2.0'},
                         {'name': 'pluggy', 'version': '1.4.0'},
                         {'name': 'pyproject-api', 'version': '1.6.1'},
                         {'name': 'setuptools', 'version': '69.2.0'},
                         {'name': 'tox', 'version': '4.14.2'},
                         {'name': 'virtualenv', 'version': '20.25.3'}],
            'version': '3.12.0 (tags/v3.12.0:0fb18b0, Oct  2 2023, 13:03:39) '
                       '[MSC v.1935 64 bit (AMD64)]'}}

Using Python 3.12 after:

>>> pprint.pprint(nise.get_diagnostic_information())
{'driver': {'name': 'NI Switch Executive', 'version': '23.8.0.49182'},
 'module': {'name': 'nise', 'version': '1.4.8.dev0'},
 'os': {'bits': '64', 'name': 'Windows', 'version': '10.0.19045'},
 'python': {'bits': '64',
            'is_venv': False,
            'packages': [{'name': 'cachetools', 'version': '5.3.3'},
                         {'name': 'chardet', 'version': '5.2.0'},
                         {'name': 'colorama', 'version': '0.4.6'},
                         {'name': 'distlib', 'version': '0.3.8'},
                         {'name': 'filelock', 'version': '3.13.4'},
                         {'name': 'hightime', 'version': '0.2.2'},
                         {'name': 'nise', 'version': '1.4.7'},
                         {'name': 'packaging', 'version': '24.0'},
                         {'name': 'pip', 'version': '24.0'},
                         {'name': 'platformdirs', 'version': '4.2.0'},
                         {'name': 'pluggy', 'version': '1.4.0'},
                         {'name': 'pyproject-api', 'version': '1.6.1'},
                         {'name': 'setuptools', 'version': '69.2.0'},
                         {'name': 'tox', 'version': '4.14.2'},
                         {'name': 'virtualenv', 'version': '20.25.3'}],
            'version': '3.12.0 (tags/v3.12.0:0fb18b0, Oct  2 2023, 13:03:39) '
                       '[MSC v.1935 64 bit (AMD64)]'}}

I also checked the output with Python 3.10.

ni-jfitzger commented 4 months ago

Python 3.12 tests currently fail due to use of pkg_resources in __init__.py.

easy_install, pkg_resources, setuptools and distutils are no longer provided by default in environments created with venv or bootstrapped with ensurepip, since they are part of the setuptools package. For projects relying on these at runtime, the setuptools project should be declared as a dependency and installed separately (typically, using pip).

https://docs.python.org/3/whatsnew/3.12.html

importlib.metadata seems like a viable alternative, starting in Python 3.10. There's a backport, called importlib_metadata, but I don't want to add a dependency, so I think we'll wind up with an if/else and conditional import, for a while.

bkeryan commented 4 months ago

Python 3.12 tests currently fail due to use of pkg_resources in __init__.py.

easy_install, pkg_resources, setuptools and distutils are no longer provided by default in environments created with venv or bootstrapped with ensurepip, since they are part of the setuptools package. For projects relying on these at runtime, the setuptools project should be declared as a dependency and installed separately (typically, using pip).

https://docs.python.org/3/whatsnew/3.12.html

importlib.metadata seems like a viable alternative, starting in Python 3.10. There's a backport, called importlib_metadata, but I don't want to add a dependency, so I think we'll wind up with an if/else and conditional import, for a while.

Are you sure you need importlib_metadata with Python 3.8 and 3.9? We were using it in nidaqmx-python, which caused a lot of problems with packages that require specific versions of importlib_metadata. We removed the importlib_metadata dependency when we dropped Python 3.7 support.

The new nidaqmx installdriver feature specifically uses importlib.resources as a replacement for pkg_resources: https://github.com/ni/nidaqmx-python/blob/master/generated/nidaqmx/_install_daqmx.py

Cc: @mshafer-ni

ni-jfitzger commented 4 months ago

Are you sure you need importlib_metadata with Python 3.8 and 3.9?

@bkeryan I'm not going to use it. As I said, I don't want to a new install dependency. I'm going to use pkg_resources for Python 3.8 and 3.9 and importlib.metadata (built-in) for Python 3.10 and later.

ni-jfitzger commented 4 months ago

Hmm ... looks like we might still be blocked.

travis-ci failed.

Building wheels for collected packages: grpcio-tools Building wheel for grpcio-tools (pyproject.toml): started Building wheel for grpcio-tools (pyproject.toml): still running... Building wheel for grpcio-tools (pyproject.toml): still running... Building wheel for grpcio-tools (pyproject.toml): still running... Building wheel for grpcio-tools (pyproject.toml): still running... Building wheel for grpcio-tools (pyproject.toml): finished with status 'error' error: subprocess-exited-with-error

I'll have to look into this.

ni-jfitzger commented 4 months ago

Hmm ... looks like we might still be blocked.

travis-ci failed.

Building wheels for collected packages: grpcio-tools Building wheel for grpcio-tools (pyproject.toml): started Building wheel for grpcio-tools (pyproject.toml): still running... Building wheel for grpcio-tools (pyproject.toml): still running... Building wheel for grpcio-tools (pyproject.toml): still running... Building wheel for grpcio-tools (pyproject.toml): still running... Building wheel for grpcio-tools (pyproject.toml): finished with status 'error' error: subprocess-exited-with-error

I'll have to look into this.

Looks like grpcio-tools 1.49.1 wheels are restricted to Python 3.10 and Python 3.11. grpcio-tools 1.59.0 is the earliest version supporting Python 3.12.

ni-jfitzger commented 4 months ago

nidaqmx already added Python 3.12 support and used grpcio-tools 1.59.0 for Python 3.12. See https://github.com/ni/nidaqmx-python/pull/453

They used grpcio-tools 1.49.1 for Python 3.8 - Python 3.11 and grpcio-tools 1.59.0 for Python 3.12. However, nimi-python gets codegenerated with only a single version, so we'll have to do things differently.

Here are the compatibility requirements for grpcio-tools:

grpcio: version greater than or equal to grpcio-tools version
protobuf: varies; >=4.21 for gprcio-tools 1.49.0, >=4.21.6 for grpcio-tools 1.59.0

ni / nimi-python