Add modern alternatives to `inspect` module, deprecate old incorrect APIs

[sobolevn]

Feature or enhancement

Proposal:

I propose to provide modern alternatives to and deprecate these inspect members:

• getargs() undocumented helper used in getargvalues. It works with __code__ objects. Can it be replaced with modern tooling? Not yet: we have an internal helper to get signatures from __code__ objects: https://github.com/python/cpython/blob/39ef93edb9802dccdb6555d4209ac2e60875a011/Lib/inspect.py#L2423-L2488 But, some refactoring is required

• getargvalues() only works with frame objects, but it does not work correctly with pos-only and kw-only parameters. Bug: inspect.getargvalues does not work correctly for pos-only and kw-only arguments · Issue #107833 · python/cpython · GitHub Can it be replaced with modern tooling? No, I propose adding inspect.Signature.from_frame method

Notice: formatargvalues should also be deprecated, because the only way to use is together with getargvalues.

There was a reverted attempt to depracate them in 3.5

• getcallargs() was documented as deprecated for a long time, it also has known bugs with pos-only params (inspect.getcallargs does not raise TypeError for pos-only passed as keywords · Issue #107831 · python/cpython · GitHub). Can it be replaced with modern tooling? Yes: inspect.Signature.bind

• getfullargspec() is documented as

  Note that signature() and Signature Object provide the recommended API for callable introspection, and support additional behaviours (like positional-only arguments) that are sometimes encountered in extension module APIs. This function is retained primarily for use in code that needs to maintain compatibility with the Python 2 inspect module API.

It has a rich history of deprecation / undeprecation:

• Deprecated: https://github.com/python/cpython/issues/64637
• Undeprecated: https://github.com/python/cpython/issues/71359
• Suggested to be deprecated again: https://github.com/python/cpython/issues/76371

More history: https://github.com/python/cpython/issues/76371#issuecomment-1270007408

It is broken in a sense that it does not differentiate pos-only from pos-or-keyword parameters.

Can it be replaced with modern tooling? Partially: inspect.signature has some differences. But, getfullargspec() uses signature() internally

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/consider-deprecating-a-bunch-of-inspect-functions/31369

CC @gpshead

Linked PRs

• gh-108902
• gh-112236
• gh-112279
• gh-112314
• gh-112639
• gh-116537
• gh-116559

[sobolevn]

The important thing is having modern fully capable alternatives in place that can satisfy the needs of users of the existing APIs. Without that, deprecating these only causes pain as no functional alternatives exist. So overall, agreed: Succeed in doing #1 (modern alternatives) and #2 (open ended deprecation) can actually be started.

I think that we should do it on a per-function basis. So, we can work incrementally on it.

It is also worth considering if the replacement can be done such that it is usable Python 3.8-3.12 as a PyPI package so that existing users of the APIs could go ahead and move off everywhere.

Good idea!

[vstinner]

I propose to provide modern alternatives to and deprecate these inspect members

Why not just using inspect.singature()? Would you mind to elaborate what are signature() issues? Its design allowed to support smoothly positional only arguments.

[sobolevn]

Why not just using inspect.singature()?

I am using inspect.Signature.from_code :) But, right now it was impossible to use Signature created from CodeType. My PR solves that, so there's now no need to use getargs() at all.

[hugovk]

For each of these (https://github.com/python/cpython/pull/112236, https://github.com/python/cpython/pull/112279, https://github.com/python/cpython/pull/112314), please can you add documentation either to Porting to Python 3.13 (and/or their relevant docs pages) that shows examples of code using the old API, and the recommended replacement?

This will help people upgrade.

[sobolevn]

Sure! I will update all PRs soon.

[hugovk]

Some usage stats from the top 8,000 (not 5k) PyPI projects, downloaded 2023-09-01. Some false positives in here, but it gives a good overview of use.

• "\binspect\b.*\bgetcallargs\b" - Found 56 matching lines in 40 projects

Details

```console ❯ python3 ~/github/misc/cpython/search_pypi_top.py -q . "\binspect\b.*\bgetcallargs\b" ./pytest-sftpserver-1.3.0.tar.gz: pytest-sftpserver-1.3.0/pytest_sftpserver/compat.py: from inspect import getcallargs ./os-brick-6.4.0.tar.gz: os-brick-6.4.0/os_brick/utils.py: all_args = inspect.getcallargs(f, *args, **kwargs) ./os-brick-6.4.0.tar.gz: os-brick-6.4.0/os_brick/utils.py: call_args = inspect.getcallargs(func, *args, **kwargs) ./dimod-0.12.11.tar.gz: dimod-0.12.11/dimod/decorators.py: bound_args = inspect.getcallargs(f, *args, **kwargs) ./dimod-0.12.11.tar.gz: dimod-0.12.11/dimod/decorators.py: bound_args = inspect.getcallargs(f, *args, **kwargs) ./ipywidgets-8.1.0.tar.gz: ipywidgets-8.1.0/ipywidgets/widgets/interaction.py: from inspect import getcallargs ./keras-2.13.1.tar.gz: keras-2.13.1/keras/src/utils/tf_inspect.py: """TFDecorator-aware replacement for inspect.getcallargs. ./webexteamssdk-1.6.1.tar.gz: webexteamssdk-1.6.1/webexteamssdk/generator_containers.py: self.arguments = inspect.getcallargs( ./apache-beam-2.50.0.zip: apache-beam-2.50.0/apache_beam/typehints/decorators.py: with inspect.getcallargs. ./apache-beam-2.50.0.zip: apache-beam-2.50.0/apache_beam/typehints/decorators.py: Works like inspect.getcallargs, with some modifications to support type hint ./apache-beam-2.50.0.zip: apache-beam-2.50.0/apache_beam/typehints/typecheck.py: actual_inputs = inspect.getcallargs(self._process_fn, *args, **kwargs) # pylint: disable=deprecated-method ./os_sys-2.1.4.tar.gz: os_sys-2.1.4/server/contrib/auth/__init__.py: inspect.getcallargs(backend.authenticate, request, **credentials) ./os_sys-2.1.4.tar.gz: os_sys-2.1.4/server/template/base.py: from inspect import getcallargs, getfullargspec, unwrap ./python-language-server-0.36.2.tar.gz: python-language-server-0.36.2/pyls/_utils.py: call_args = inspect.getcallargs(func, *args, **kwargs) ./Nuitka-1.8.tar.gz: Nuitka-1.8/nuitka/specs/ParameterSpecs.py: Based loosely on "inspect.getcallargs" with corrections. ./dm-haiku-0.0.10.tar.gz: dm-haiku-0.0.10/haiku/_src/utils.py: arg_values = inspect.getcallargs(cls.__init__, None, *args, **kwargs) # pylint: disable=deprecated-method ./bravado-core-6.1.0.tar.gz: bravado-core-6.1.0/bravado_core/util.py: param_name_to_value_mapping = sorted(iteritems(inspect.getcallargs(func, *args, **kwargs))) ./newrelic-9.0.0.tar.gz: newrelic-9.0.0/newrelic/common/signature.py: from inspect import getcallargs ./newrelic-9.0.0.tar.gz: newrelic-9.0.0/newrelic/packages/wrapt/__init__.py: # Import of inspect.getcallargs() included for backward compatibility. An ./newrelic-9.0.0.tar.gz: newrelic-9.0.0/newrelic/packages/wrapt/__init__.py: from inspect import getcallargs ./pyroute2-0.7.9.tar.gz: pyroute2-0.7.9/pyroute2/wiset.py: from inspect import getcallargs ./wagtail-5.1.1.tar.gz: wagtail-5.1.1/wagtail/images/image_operations.py: inspect.getcallargs(self.construct, *args) ./oslo.versionedobjects-3.2.0.tar.gz: oslo.versionedobjects-3.2.0/ChangeLog: * Replace safe\_utils.getcallargs with inspect.getcallargs ./oslo.versionedobjects-3.2.0.tar.gz: oslo.versionedobjects-3.2.0/oslo_versionedobjects/exception.py: call_dict = inspect.getcallargs(f, self, context, ./thriftrw-1.9.0.tar.gz: thriftrw-1.9.0/thriftrw/spec/struct.pyx: # inspect.getcallargs() with adjustments for our requirements of ./keras-nightly-2.15.0.dev2023090107.tar.gz: keras-nightly-2.15.0.dev2023090107/keras/src/utils/tf_inspect.py: """TFDecorator-aware replacement for inspect.getcallargs. ./darts-0.25.0.tar.gz: darts-0.25.0/darts/utils/utils.py: from inspect import Parameter, getcallargs, signature ./django-user-tasks-3.1.0.tar.gz: django-user-tasks-3.1.0/user_tasks/tasks.py: return inspect.getcallargs(cls.run, *all_args, **kwargs) # pylint: disable=deprecated-method ./CherryPy-18.8.0.tar.gz: CherryPy-18.8.0/cherrypy/lib/__init__.py: inspect.getcallargs(obj.close) ./Cython-3.0.2.tar.gz: Cython-3.0.2/Cython/Build/Inline.py: all = inspect.getcallargs(self._f, *args, **kwds) ./os-win-5.9.0.tar.gz: os-win-5.9.0/os_win/_utils.py: all_args = inspect.getcallargs(func, *args, **kwargs) ./os-win-5.9.0.tar.gz: os-win-5.9.0/os_win/utils/storage/initiator/iscsi_utils.py: call_args = inspect.getcallargs(f, *args, **kwargs) ./autobahn-23.6.2.tar.gz: autobahn-23.6.2/autobahn/wamp/protocol.py: arguments = inspect.getcallargs(func, *args, **kwargs) ./plivo-4.40.0.tar.gz: plivo-4.40.0/plivo/utils/validators.py: params = inspect.getcallargs(wrapped, *args, **kwargs) ./ddtrace-1.18.3.tar.gz: ddtrace-1.18.3/ddtrace/vendor/wrapt/__init__.py: # Import of inspect.getcallargs() included for backward compatibility. An ./ddtrace-1.18.3.tar.gz: ddtrace-1.18.3/ddtrace/vendor/wrapt/__init__.py: from inspect import getcallargs ./nplusone-1.0.0.tar.gz: nplusone-1.0.0/nplusone/ext/django/patch.py: context = inspect.getcallargs(create_forward_many_to_many_manager, *args, **kwargs) ./nplusone-1.0.0.tar.gz: nplusone-1.0.0/nplusone/ext/django/patch.py: context = inspect.getcallargs(create_reverse_many_to_one_manager, *args, **kwargs) ./nplusone-1.0.0.tar.gz: nplusone-1.0.0/nplusone/ext/sqlalchemy.py: context = inspect.getcallargs(original_populate_full, *args, **kwargs) ./azureml-0.2.7.zip: azureml-0.2.7/azureml/services.py: args = inspect.getcallargs(self.func, *args, **kwargs) ./mdc-1.2.1.tar.gz: mdc-1.2.1/mdc/decorators.py: bound = inspect.getcallargs(func, *args, **kwargs) ./uplink-0.9.7.tar.gz: uplink-0.9.7/uplink/utils.py: from inspect import getcallargs as get_call_args, getargspec as _getargspec ./catboost-1.2.1.tar.gz: catboost-1.2.1/catboost_all_src/contrib/python/ipywidgets/py2/ipywidgets/widgets/interaction.py: from inspect import getcallargs ./catboost-1.2.1.tar.gz: catboost-1.2.1/catboost_all_src/contrib/python/ipywidgets/py3/ipywidgets/widgets/interaction.py: from inspect import getcallargs ./pylint-2.17.5.tar.gz: pylint-2.17.5/pylint/checkers/stdlib.py: "inspect.getcallargs", ./dora_search-0.1.12.tar.gz: dora_search-0.1.12/dora/lightning.py: kwargs = inspect.getcallargs(init, [None] + list(args), **kwargs) ./pony-0.7.16.tar.gz: pony-0.7.16/pony/orm/sqltranslation.py: name_mapping = inspect.getcallargs(monad.func, *(monad.params + args), **kwargs) ./wrapt-1.15.0.tar.gz: wrapt-1.15.0/src/wrapt/__init__.py: # Import of inspect.getcallargs() included for backward compatibility. An ./wrapt-1.15.0.tar.gz: wrapt-1.15.0/src/wrapt/__init__.py: from inspect import getcallargs ./mongo_tooling_metrics-1.0.8.tar.gz: mongo_tooling_metrics-1.0.8/src/mongo_tooling_metrics/lib/hooks.py: from inspect import getcallargs ./sktime-0.22.0.tar.gz: sktime-0.22.0/sktime/utils/estimators/_base.py: from inspect import getcallargs, getfullargspec ./eliot-1.14.0.tar.gz: eliot-1.14.0/eliot/_action.py: from inspect import getcallargs ./gpiozero-1.6.2.tar.gz: gpiozero-1.6.2/gpiozero/mixins.py: inspect.getcallargs(wrapped_fn, *args) ./gpiozero-1.6.2.tar.gz: gpiozero-1.6.2/gpiozero/mixins.py: inspect.getcallargs(wrapped_fn, *(args + (self,))) ./catboost-1.2.1.tar.gz: catboost-1.2.1/catboost_all_src/contrib/tools/cython/Cython/Build/Inline.py: from inspect import getcallargs ./magicinvoke-2.4.6.tar.gz: magicinvoke-2.4.6/magicinvoke/magicinvoke.py: # inspect already has _too_many, but only for getcallargs. Time: 0:00:32.549634 Found 56 matching lines in 40 projects ```

• "\binspect\b.*\bgetargs\b" - Found 20 matching lines in 12 projects

Details

```console ❯ python3 ~/github/misc/cpython/search_pypi_top.py -q . "\binspect\b.*\bgetargs\b" ./ipython-8.14.0.tar.gz: ipython-8.14.0/IPython/testing/tests/test_decorators.py: args, varargs, varkw = inspect.getargs(func_obj.__code__) ./Cheetah3-3.2.6.post1.tar.gz: Cheetah3-3.2.6.post1/Cheetah/Template.py: inspect.getargs(templateAPIClass.compile.__func__.__code__)[0] ./Cheetah3-3.2.6.post1.tar.gz: Cheetah3-3.2.6.post1/Cheetah/Parser.py: availableKwArgs = inspect.getargs(co)[0] ./Nuitka-1.8.tar.gz: Nuitka-1.8/tests/basics/FunctionsTest.py: inspect.getargs(main_value.__code__), ./pdbpp-0.10.3.tar.gz: pdbpp-0.10.3/testing/test_pdb.py: if "readrc" not in inspect.getargs(pdb.pdb.Pdb.__init__.__code__).args: ./pyodps-0.11.4.1.tar.gz: pyodps-0.11.4.1/odps/lib/cloudpickle.py: func_args = set(inspect.getargs(fun.__code__).args) ./checkov-2.4.22.tar.gz: checkov-2.4.22/checkov/common/multi_signature.py: arguments = inspect.getargs(wrapped.__code__) ./checkov-2.4.22.tar.gz: checkov-2.4.22/checkov/common/multi_signature.py: if arguments == inspect.getargs(value.__code__): ./checkov-2.4.22.tar.gz: checkov-2.4.22/checkov/common/multi_signature.py: is specified with `varargs`. For keyword args it is `varkw`. See :func:`inspect.getargs` to get the ./azureml-0.2.7.zip: azureml-0.2.7/azureml/services.py: args = inspect.getargs(func.__code__) ./azureml-0.2.7.zip: azureml-0.2.7/azureml/services.py: args = inspect.getargs(function.__code__) ./catboost-1.2.1.tar.gz: catboost-1.2.1/catboost_all_src/contrib/python/ipython/py2/IPython/core/ultratb.py: # This is a patched version of inspect.getargs that applies the (unmerged) ./catboost-1.2.1.tar.gz: catboost-1.2.1/catboost_all_src/contrib/python/ipython/py2/IPython/core/ultratb.py: save_getargs = inspect.getargs ./catboost-1.2.1.tar.gz: catboost-1.2.1/catboost_all_src/contrib/python/ipython/py2/IPython/core/ultratb.py: inspect.getargs = getargs ./catboost-1.2.1.tar.gz: catboost-1.2.1/catboost_all_src/contrib/python/ipython/py2/IPython/core/ultratb.py: inspect.getargs = save_getargs ./pony-0.7.16.tar.gz: pony-0.7.16/pony/orm/decompiling.py: argnames, vararg, kwarg = inspect.getargs(codeobject) ./executing-1.2.0.tar.gz: executing-1.2.0/tests/samples/bird.py: arg_info = inspect.getargs(new_func.__code__) ./catboost-1.2.1.tar.gz: catboost-1.2.1/catboost_all_src/contrib/python/traitlets/py2/traitlets/utils/getargspec.py: args, varargs, varkw = inspect.getargs(func.__code__) ./guppy3-3.1.3.tar.gz: guppy3-3.1.3/guppy/heapy/Spec.py: (args, varargs, varkw) = inspect.getargs(code) ./spyder-kernels-2.4.4.tar.gz: spyder-kernels-2.4.4/spyder_kernels/utils/dochelpers.py: args, _, _ = inspect.getargs(func_obj.__code__) Time: 0:00:33.657835 Found 20 matching lines in 12 projects ```

• "\binspect\b.*\bgetfullargspec\b" - Found 5967 matching lines in 365 projects

> There was a problem saving your comment. Your comment is too long (maximum is 65536 characters). Please try again. ! Take 2: [inspect.getfullargspec.txt](https://github.com/python/cpython/files/13440131/inspect.getfullargspec.txt)

That last one looks pretty high.

[vstinner]

What's the status of this issue?