[BUG] setup tools related python functions are taking more time after upgrading to 65.3.0 from 57.0.0 as number of calls getting increased

[mbhata]

setuptools version

65.3.0

Python version

3.9.16

OS

custom linux distro

Additional environment information

issue observed env: openssl version -> 3.0.8 cryptography -> 40.0.2 setuptool -> 65.3.0 this issue is observed in all usecases that is imporitng setuptool package.

Description

As part of upgrading openssl to 3.0.8 version, there requires indirect dependency with setuptools package, hence upgraded to version 65.3.0 to match with minimum requirement to openssl.

After update ,there is some degrade in performance during the runtime which is observed to be appeared from setuptools approx 0.6 seconds more than usual as the total number of calls are getting increased for same code when executed with two different setuptool version

Is this expected behaviour ?

Expected behavior

number of calls and total time taken should be approx same when setuptools upgraded to (65.3.0)

How to Reproduce

sample Script uscase where the issue can be observed easily

# cat tesversion.py 
#!/usr/bin/env python
import pkg_resources
version = pkg_resources.require("redis")[0].version
print(version)

# cat tesversion.py 
#!/usr/bin/env python
import pkg_resources
version = pkg_resources.require("redis")[0].version
print(version)

#  python -m cProfile -o profile.out tesversion.py
# python showprofile.py

Output

profiling of openssl(3.0.8) + cryptography( 40.0.2) + setuptool upgrade(65.3.0)

Thu Jun 22 11:29:02 2023 profile.out
428104 function calls (412759 primitive calls) in **0.937 seconds**
Ordered by: internal time
List reduced from 2437 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
3185 0.046 0.000 0.105 0.000 /usr/lib/python3.9/sitepackages/
pkg_resources/_vendor/packaging/version.py:261(__init__)
215 0.039 0.000 0.039 0.000 {built-in method marshal.loads}
504/101 0.027 0.000 0.083 0.001 /usr/lib/python3.9/sre_parse.py:494(_parse)
827/775 0.025 0.000 0.162 0.000 {built-in method builtins.__build_class__}
2498 0.025 0.000 0.025 0.000 {built-in method posix.stat}
38302 0.019 0.000 0.040 0.000 {built-in method builtins.isinstance}
13831 0.018 0.000 0.018 0.000 /usr/lib/python3.9/sre_parse.py:234(__next)
651 0.018 0.000 0.062 0.000 <frozen
importlib._bootstrap_external>:1514(find_spec)
2025 0.017 0.000 0.019 0.000 /usr/lib/python3.9/sitepackages/
pkg_resources/_vendor/packaging/version.py:444(_cmpkey)

/usr/lib/python3.9/sitepackages/pkg_resources/_vendor/packaging/version.py is coming from setuptools and this is always topping the list of functions which is causing more time. This file/methods from this file were not at all listed when profiling on the old versions.

profiling of openssl(1.1.1o) + cryptography( 3.2.1) + setuptool upgrade(57.0.0)

Thu Jun 22 11:25:08 2023 profile.out
55287 function calls (54295 primitive calls) in **0.130 seconds**
Ordered by: internal time
List reduced from 752 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
18 0.008 0.000 0.010 0.001 {built-in method _imp.create_dynamic}
42 0.007 0.000 0.007 0.000 {built-in method marshal.loads}
142 0.005 0.000 0.016 0.000 {built-in method builtins.__build_class__}
19 0.005 0.000 0.012 0.001 /usr/lib/python3.9/enum.py:179(__new__)
173 0.004 0.000 0.014 0.000 <frozen
importlib._bootstrap_external>:1514(find_spec)
309 0.004 0.000 0.004 0.000 {built-in method posix.stat}
94/24 0.004 0.000 0.011 0.000 /usr/lib/python3.9/sre_parse.py:494(_parse)
8 0.003 0.000 0.003 0.000 {built-in method builtins.eval}
4 0.003 0.001 0.003 0.001 {method 'recv' of '_socket.socket' objects}
38 0.003 0.000 0.006 0.000 /usr/lib/python3.9/sitepackages/
redis/connection.py:759(pack_command)

=========================================================================== Sample scripts output:

profiling of openssl(3.0.8) + cryptography( 40.0.2) + setuptool upgrade(65.3.0)

Mon Jun 26 18:25:56 2023    profile.out

         318328 function calls (309872 primitive calls) in 0.667 seconds

   Ordered by: internal time
   List reduced from 1342 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     3187    0.046    0.000    0.102    0.000 /usr/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py:261(__init__)
   413/65    0.023    0.000    0.071    0.001 /usr/lib/python3.9/sre_parse.py:494(_parse)
     1896    0.019    0.000    0.019    0.000 {built-in method posix.stat}
       82    0.018    0.000    0.018    0.000 {built-in method marshal.loads}
    11952    0.017    0.000    0.017    0.000 /usr/lib/python3.9/sre_parse.py:234(__next)
     2027    0.017    0.000    0.018    0.000 /usr/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py:444(_cmpkey)

================================================================

profiling of openssl(1.1.1o) + cryptography( 3.2.1) + setuptool upgrade(57.0.0)

================================================================

Mon Jun 26 18:18:03 2023    profile.out

         311194 function calls (307101 primitive calls) in 0.630 seconds

   Ordered by: internal time
   List reduced from 1046 to 20 due to restriction <20>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2150    0.027    0.000    0.027    0.000 {built-in method posix.stat}
   452/76    0.025    0.000    0.076    0.001 /usr/lib/python3.9/sre_parse.py:494(_parse)
     2043    0.023    0.000    0.055    0.000 /usr/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py:271(__init__)
     1166    0.019    0.000    0.056    0.000 /usr/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py:205(_legacy_cmpkey)
     8152    0.019    0.000    0.029    0.000 /usr/lib/python3.9/site-packages/pkg_resources/_vendor/packaging/version.py:187(_parse_version_parts)
    12756    0.017    0.000    0.017    0.000 /usr/lib/python3.9/sre_parse.py:234(__next)
      507    0.015    0.000    0.055    0.000 <frozen importlib._bootstrap_external>:1514(find_spec)
     2367    0.013    0.000    0.019    0.000 /usr/lib/python3.9/posixpath.py:71(join)
       59    0.012    0.000    0.012    0.000 {built-in method marshal.loads}

[abravalheri]

Hi @mbhata, please note that the module pkg_resources is deprecated. We are providing maintenance to it, but I don't think performance parity is at the top of the priorities (specially because we also depend on 3rd party dependencies).

Have you considered using importlib.metadata.version as a replacement for the example you provided? You can also use importlib.metadata.distribution to get a distribution object...