[BUG] SETUPTOOLS_USE_DISTUTILS=local makes any `distutils` import very slow

[asottile]

setuptools version

setuptools==65.3.0

Python version

python 3.8.10

OS

ubuntu 20.04

Additional environment information

No response

Description

when using the bundled distutils, importing any part of distutils takes almost 10x longer than previously

---

I originally started with trying to figure out why virtualenv got way slower recently and ended up tracing it to setuptools>=60

Expected behavior

I expect the bundled distutils to not incur the import time problems of setuptools

How to Reproduce

compare the following:

• time env SETUPTOOLS_USE_DISTUTILS=local venv3/bin/python3 -c 'import distutils.version'
• time env SETUPTOOLS_USE_DISTUTILS=stdlib venv3/bin/python3 -c 'import distutils.version'

Output

$ best-of -n 10 -- env SETUPTOOLS_USE_DISTUTILS=local venv3/bin/python3 -c 'import distutils.version'
..........
best of 10: 0.254s
$ best-of -n 10 -- env SETUPTOOLS_USE_DISTUTILS=stdlib venv3/bin/python3 -c 'import distutils.version'
..........
best of 10: 0.021s

best-of here is just a script which runs the command in a loop and reports the fastest time -- time also works to reproduce this

[CAM-Gerlach]

FWIW, on Windows x64 with Python 3.10.2 (conda) and Setuptools 65.5.0, I'm seeing a 25x slowdown (40 ms to 1 s) with -X importtime, and a ≈80x slowdown (10-11 ms to 780-860 ms) with -m timeit. (Just importing distutils (rather than distutils.version) only shaved around 2 ms off on average, though while it was well within the margin of error for local, it made a non-trivial relative difference for stdlib, to about 8 ms or around a 100x speedup under timeit.)

With timeit:

$ SETUPTOOLS_USE_DISTUTILS=local python -m timeit -n 1 -r 1 'import distutils.version'
1 loop, best of 1: 806 msec per loop

$ SETUPTOOLS_USE_DISTUTILS=stdlib python -m timeit -n 1 -r 1 'import distutils.version'
1 loop, best of 1: 10 msec per loop

With importtime:

$ SETUPTOOLS_USE_DISTUTILS=local python -X importtime -c 'import distutils.version'
import time: self [us] | cumulative | imported package
[...]
import time:      9827 |    1014210 |   distutils
import time:      2366 |    1016576 | distutils.version

$ SETUPTOOLS_USE_DISTUTILS=stdlib python -X importtime -c 'import distutils.version'
import time: self [us] | cumulative | imported package
[...]
import time:      2598 |      39466 |   distutils
import time:      2423 |      41889 | distutils.version

For reference, I've also provided the full -X importtime tree output for both scenarios, profiling where most of the import-time cost comes from. If lazy imports makes it into 3.12, perhaps that could defray a large amount of the cost?

Full `-X importtime` output with `local`

```console $ SETUPTOOLS_USE_DISTUTILS=local python -X importtime -c 'import distutils.version' import time: self [us] | cumulative | imported package import time: 350 | 350 | _io import time: 121 | 121 | marshal import time: 410 | 410 | nt import time: 206 | 206 | winreg import time: 2280 | 3365 | _frozen_importlib_external import time: 739 | 739 | time import time: 626 | 1364 | zipimport import time: 286 | 286 | _codecs import time: 1808 | 2093 | codecs import time: 1617 | 1617 | encodings.aliases import time: 3370 | 7079 | encodings import time: 1067 | 1067 | encodings.utf_8 import time: 1082 | 1082 | encodings.cp1252 import time: 135 | 135 | _signal import time: 126 | 126 | _abc import time: 1432 | 1558 | abc import time: 1596 | 3153 | io import time: 266 | 266 | _stat import time: 1546 | 1812 | stat import time: 3741 | 3741 | _collections_abc import time: 1236 | 1236 | genericpath import time: 2137 | 3372 | ntpath import time: 3008 | 11931 | os import time: 1359 | 1359 | _sitebuiltins import time: 4810 | 4810 | _distutils_hack import time: 3728 | 3728 | sitecustomize import time: 1546 | 1546 | usercustomize import time: 7218 | 30590 | site import time: 1604 | 1604 | warnings import time: 2255 | 3859 | importlib import time: 1952 | 1952 | importlib.machinery import time: 1032 | 1032 | importlib._abc import time: 1163 | 1163 | itertools import time: 1189 | 1189 | keyword import time: 170 | 170 | _operator import time: 1629 | 1798 | operator import time: 1153 | 1153 | reprlib import time: 184 | 184 | _collections import time: 3597 | 9082 | collections import time: 1153 | 1153 | collections.abc import time: 1460 | 1460 | types import time: 221 | 221 | _functools import time: 2910 | 4590 | functools import time: 2686 | 7276 | contextlib import time: 2484 | 2484 | enum import time: 485 | 485 | _sre import time: 1571 | 1571 | sre_constants import time: 2293 | 3864 | sre_parse import time: 2036 | 6385 | sre_compile import time: 274 | 274 | _locale import time: 1472 | 1472 | copyreg import time: 3201 | 13813 | re import time: 6226 | 37548 | typing import time: 3279 | 43809 | importlib.abc import time: 1973 | 1973 | importlib.util import time: 1378 | 1378 | token import time: 3549 | 4926 | tokenize import time: 1139 | 1139 | distutils.debug import time: 1395 | 1395 | distutils.errors import time: 2096 | 2096 | posixpath import time: 1748 | 3844 | fnmatch import time: 243 | 243 | errno import time: 1566 | 1566 | urllib import time: 3291 | 4856 | urllib.parse import time: 3635 | 12576 | pathlib import time: 1572 | 1572 | email import time: 129 | 129 | _string import time: 3230 | 3359 | string import time: 4598 | 4598 | gettext import time: 2156 | 6754 | getopt import time: 2555 | 12667 | distutils.fancy_getopt import time: 2296 | 2296 | signal import time: 1262 | 1262 | _weakrefset import time: 2671 | 3933 | threading import time: 1567 | 1567 | fcntl import time: 170 | 170 | msvcrt import time: 215 | 215 | _winapi import time: 3162 | 11341 | subprocess import time: 2048 | 2048 | sysconfig import time: 840 | 840 | distutils.dep_util import time: 1088 | 1088 | distutils.log import time: 1133 | 2220 | distutils.spawn import time: 2172 | 18620 | distutils.util import time: 2925 | 48358 | distutils.dist import time: 1098 | 1098 | distutils.dir_util import time: 1051 | 1051 | distutils.file_util import time: 2190 | 2190 | binascii import time: 206 | 206 | zlib import time: 1309 | 1309 | _compression import time: 5625 | 5625 | _bz2 import time: 2070 | 9003 | bz2 import time: 4912 | 4912 | _lzma import time: 2001 | 6913 | lzma import time: 3119 | 19241 | shutil import time: 206 | 206 | _struct import time: 1635 | 1840 | struct import time: 3120 | 26388 | zipfile import time: 1638 | 1638 | pwd import time: 1576 | 1576 | grp import time: 1735 | 31335 | distutils.archive_util import time: 2118 | 35600 | distutils.cmd import time: 6275 | 6275 | configparser import time: 1498 | 7773 | distutils.config import time: 1291 | 1291 | distutils.extension import time: 8353 | 108832 | _distutils_hack.override import time: 1005 | 1005 | setuptools._deprecation_warning import time: 2200 | 2200 | weakref import time: 2047 | 4246 | pkgutil import time: 6983 | 6983 | platform import time: 171 | 171 | math import time: 304 | 304 | _datetime import time: 3036 | 3510 | datetime import time: 1501 | 1501 | xml import time: 1409 | 2910 | xml.parsers import time: 3791 | 3791 | pyexpat import time: 1799 | 8499 | xml.parsers.expat import time: 3996 | 16004 | plistlib import time: 1661 | 1661 | email.errors import time: 1143 | 1143 | email.quoprimime import time: 1790 | 1790 | base64 import time: 1327 | 3116 | email.base64mime import time: 1182 | 1182 | quopri import time: 1113 | 2295 | email.encoders import time: 1660 | 3954 | email.charset import time: 3053 | 11266 | email.header import time: 132 | 132 | _bisect import time: 1733 | 1865 | bisect import time: 137 | 137 | _random import time: 116 | 116 | _sha512 import time: 2670 | 4786 | random import time: 6794 | 6794 | _socket import time: 2282 | 2282 | select import time: 2620 | 4902 | selectors import time: 5870 | 17566 | socket import time: 3063 | 3063 | locale import time: 2489 | 5552 | calendar import time: 1766 | 7317 | email._parseaddr import time: 2591 | 32258 | email.utils import time: 2226 | 45749 | email._policybase import time: 2318 | 49727 | email.feedparser import time: 1854 | 51580 | email.parser import time: 1693 | 1693 | tempfile import time: 3502 | 3502 | textwrap import time: 2310 | 2310 | _ast import time: 5474 | 7783 | ast import time: 142 | 142 | _opcode import time: 1847 | 1988 | opcode import time: 2286 | 4273 | dis import time: 1252 | 1252 | linecache import time: 5497 | 18804 | inspect import time: 2236 | 2236 | pkg_resources.extern import time: 1112 | 1112 | pkg_resources._vendor import time: 1488 | 2599 | pkg_resources._vendor.jaraco import time: 1404 | 6238 | pkg_resources.extern.jaraco import time: 978 | 978 | importlib._adapters import time: 1251 | 2229 | importlib._common import time: 1416 | 3645 | importlib.resources import time: 175 | 175 | _heapq import time: 2705 | 2880 | heapq import time: 2561 | 2561 | _queue import time: 1821 | 4382 | queue import time: 1271 | 1271 | pkg_resources._vendor.more_itertools.recipes import time: 3173 | 11704 | pkg_resources._vendor.more_itertools.more import time: 1559 | 13262 | pkg_resources._vendor.more_itertools import time: 868 | 14129 | pkg_resources.extern.more_itertools import time: 1659 | 15788 | pkg_resources.extern.jaraco.functools import time: 1753 | 1753 | pkg_resources.extern.jaraco.context import time: 3221 | 30642 | pkg_resources.extern.jaraco.text import time: 14852 | 14852 | win32api import time: 3026 | 3026 | _win32sysloader import time: 2721 | 5747 | pywintypes import time: 23570 | 29316 | pythoncom import time: 920 | 920 | win32com.gen_py import time: 3920 | 49007 | win32com import time: 1491 | 50498 | win32com.shell import time: 1409 | 51906 | pkg_resources._vendor.appdirs import time: 722 | 52627 | pkg_resources.extern.appdirs import time: 1164 | 1164 | pkg_resources._vendor.packaging.__about__ import time: 1743 | 2906 | pkg_resources._vendor.packaging import time: 495 | 3401 | pkg_resources.extern.packaging import time: 872 | 872 | pkg_resources.extern.packaging._structures import time: 7822 | 8694 | pkg_resources.extern.packaging.version import time: 1787 | 1787 | traceback import time: 172 | 172 | atexit import time: 5963 | 7921 | logging import time: 1897 | 1897 | pkg_resources._vendor.packaging._manylinux import time: 1471 | 1471 | pkg_resources._vendor.packaging._musllinux import time: 2627 | 13915 | pkg_resources.extern.packaging.tags import time: 2013 | 15927 | pkg_resources.extern.packaging.utils import time: 17582 | 33509 | pkg_resources.extern.packaging.specifiers import time: 2130 | 2130 | pkg_resources._vendor.pyparsing.util import time: 1900 | 1900 | pkg_resources._vendor.pyparsing.unicode import time: 6088 | 7988 | pkg_resources._vendor.pyparsing.exceptions import time: 1224 | 1224 | pkg_resources._vendor.pyparsing.actions import time: 1551 | 1551 | org import time: 408 | 1958 | org.python import time: 352 | 2309 | org.python.core import time: 1430 | 3738 | copy import time: 2083 | 2083 | dataclasses import time: 1857 | 3939 | pprint import time: 1802 | 5741 | pkg_resources._vendor.pyparsing.results import time: 20025 | 29503 | pkg_resources._vendor.pyparsing.core import time: 3108 | 3108 | html.entities import time: 2928 | 6036 | html import time: 573 | 6608 | html.entities import time: 7789 | 14396 | pkg_resources._vendor.pyparsing.helpers import time: 1521 | 1521 | pkg_resources._vendor.pyparsing.testing import time: 13619 | 13619 | pkg_resources._vendor.pyparsing.common import time: 4689 | 75067 | pkg_resources._vendor.pyparsing import time: 583 | 75649 | pkg_resources.extern.pyparsing import time: 5159 | 5159 | pkg_resources.extern.packaging.markers import time: 22587 | 103395 | pkg_resources.extern.packaging.requirements import time: 184535 | 519610 | pkg_resources import time: 1908 | 521518 | setuptools.version import time: 1071 | 1071 | distutils.filelist import time: 1282 | 2352 | setuptools.monkey import time: 1407 | 3759 | setuptools.extension import time: 1453 | 1453 | numbers import time: 903 | 903 | distutils.command import time: 1558 | 1558 | glob import time: 1155 | 1155 | setuptools.extern import time: 928 | 928 | setuptools._vendor import time: 1161 | 1161 | setuptools._vendor.packaging.__about__ import time: 1835 | 3924 | setuptools._vendor.packaging import time: 739 | 4663 | setuptools.extern.packaging import time: 1558 | 1558 | setuptools._vendor.ordered_set import time: 552 | 2110 | setuptools.extern.ordered_set import time: 1122 | 1122 | setuptools._vendor.more_itertools.recipes import time: 2943 | 4064 | setuptools._vendor.more_itertools.more import time: 1437 | 5501 | setuptools._vendor.more_itertools import time: 574 | 6074 | setuptools.extern.more_itertools import time: 188 | 188 | _csv import time: 1758 | 1945 | csv import time: 1380 | 1380 | uu import time: 1335 | 1335 | email._encoded_words import time: 1681 | 1681 | email.iterators import time: 3440 | 7833 | email.message import time: 890 | 890 | importlib.metadata._functools import time: 1271 | 2160 | importlib.metadata._text import time: 2111 | 12103 | importlib.metadata._adapters import time: 1294 | 1294 | importlib.metadata._meta import time: 1215 | 1215 | importlib.metadata._collections import time: 833 | 833 | importlib.metadata._itertools import time: 4737 | 22125 | importlib.metadata import time: 1367 | 23491 | setuptools._importlib import time: 2059 | 2059 | distutils.command.bdist import time: 1558 | 3617 | setuptools.command import time: 27218 | 27218 | setuptools.windows_support import time: 1843 | 1843 | setuptools._vendor.pyparsing.util import time: 1624 | 1624 | setuptools._vendor.pyparsing.unicode import time: 4246 | 5869 | setuptools._vendor.pyparsing.exceptions import time: 1289 | 1289 | setuptools._vendor.pyparsing.actions import time: 1531 | 1531 | setuptools._vendor.pyparsing.results import time: 19838 | 21368 | setuptools._vendor.pyparsing.core import time: 7374 | 7374 | setuptools._vendor.pyparsing.helpers import time: 1568 | 1568 | setuptools._vendor.pyparsing.testing import time: 11755 | 11755 | setuptools._vendor.pyparsing.common import time: 4005 | 55068 | setuptools._vendor.pyparsing import time: 491 | 55559 | setuptools.extern.pyparsing import time: 1506 | 1506 | setuptools._vendor.packaging._manylinux import time: 1416 | 1416 | setuptools._vendor.packaging._musllinux import time: 2039 | 4960 | setuptools.extern.packaging.tags import time: 973 | 973 | setuptools.extern.packaging._structures import time: 2809 | 3781 | setuptools.extern.packaging.version import time: 1780 | 10519 | setuptools.extern.packaging.utils import time: 2973 | 13491 | setuptools.extern.packaging.specifiers import time: 4931 | 18422 | setuptools.extern.packaging.markers import time: 9815 | 83794 | setuptools.extern.packaging.requirements import time: 1357 | 1357 | setuptools._path import time: 3735 | 5092 | setuptools.config.expand import time: 3330 | 92216 | setuptools.config.setupcfg import time: 1815 | 94030 | setuptools.config import time: 1016 | 1016 | setuptools.errors import time: 5210 | 5210 | email._header_value_parser import time: 2164 | 7374 | email.headerregistry import time: 2746 | 10120 | setuptools.config._apply_pyprojecttoml import time: 2576 | 13710 | setuptools.config.pyprojecttoml import time: 2138 | 2138 | setuptools.discovery import time: 972 | 972 | setuptools._vendor.jaraco import time: 616 | 1587 | setuptools.extern.jaraco import time: 1192 | 1192 | setuptools.extern.jaraco.functools import time: 1112 | 1112 | setuptools.extern.jaraco.context import time: 4001 | 7892 | setuptools.extern.jaraco.text import time: 1401 | 9292 | setuptools._reqs import time: 1414 | 1414 | setuptools._itertools import time: 2423 | 3836 | setuptools._entry_points import time: 5387 | 200628 | setuptools.dist import time: 820 | 820 | setuptools.py34compat import time: 1345 | 2164 | setuptools._imp import time: 1573 | 3737 | setuptools.depends import time: 982 | 982 | setuptools.logging import time: 153 | 153 | _json import time: 2034 | 2187 | json.scanner import time: 1965 | 4151 | json.decoder import time: 1785 | 1785 | json.encoder import time: 1911 | 7846 | json import time: 1128 | 1128 | unittest.util import time: 1481 | 2608 | unittest.result import time: 1909 | 1909 | difflib import time: 2119 | 4028 | unittest.case import time: 1228 | 1228 | unittest.suite import time: 1865 | 1865 | unittest.loader import time: 3592 | 3592 | argparse import time: 958 | 958 | unittest.signals import time: 1549 | 2507 | unittest.runner import time: 1605 | 7703 | unittest.main import time: 2818 | 20246 | unittest import time: 1184 | 1184 | concurrent import time: 3440 | 3440 | concurrent.futures._base import time: 1790 | 6412 | concurrent.futures import time: 7964 | 7964 | _ssl import time: 9274 | 17238 | ssl import time: 1391 | 1391 | asyncio.constants import time: 868 | 868 | asyncio.format_helpers import time: 1117 | 1984 | asyncio.base_futures import time: 820 | 820 | asyncio.log import time: 1571 | 4374 | asyncio.coroutines import time: 145 | 145 | _contextvars import time: 2057 | 2202 | contextvars import time: 1602 | 1602 | asyncio.exceptions import time: 1230 | 1230 | asyncio.base_tasks import time: 3242 | 6072 | _asyncio import time: 2104 | 10377 | asyncio.events import time: 1124 | 1124 | asyncio.futures import time: 1119 | 1119 | asyncio.protocols import time: 1142 | 1142 | asyncio.transports import time: 1674 | 2816 | asyncio.sslproto import time: 860 | 860 | asyncio.mixins import time: 1351 | 2210 | asyncio.locks import time: 2600 | 2600 | asyncio.tasks import time: 2229 | 7038 | asyncio.staggered import time: 1900 | 1900 | asyncio.trsock import time: 5153 | 58936 | asyncio.base_events import time: 1002 | 1002 | asyncio.runners import time: 1074 | 1074 | asyncio.queues import time: 1249 | 1249 | asyncio.streams import time: 1088 | 1088 | asyncio.subprocess import time: 1053 | 1053 | asyncio.threads import time: 3728 | 3728 | _overlapped import time: 1143 | 1143 | asyncio.base_subprocess import time: 2159 | 2159 | asyncio.proactor_events import time: 1679 | 1679 | asyncio.selector_events import time: 1086 | 1086 | asyncio.windows_utils import time: 3436 | 13228 | asyncio.windows_events import time: 3165 | 80792 | asyncio import time: 3879 | 104916 | unittest.mock import time: 1526 | 1526 | distutils.ccompiler import time: 9827 | 1014210 | distutils import time: 2366 | 1016576 | distutils.version ```

Full `-X importtime` output with `stdlib`

```console $ SETUPTOOLS_USE_DISTUTILS=stdlib python -X importtime -c 'import distutils.version' import time: self [us] | cumulative | imported package import time: 354 | 354 | _io import time: 99 | 99 | marshal import time: 375 | 375 | nt import time: 170 | 170 | winreg import time: 1999 | 2995 | _frozen_importlib_external import time: 740 | 740 | time import time: 590 | 1330 | zipimport import time: 271 | 271 | _codecs import time: 1776 | 2047 | codecs import time: 1661 | 1661 | encodings.aliases import time: 3131 | 6838 | encodings import time: 944 | 944 | encodings.utf_8 import time: 966 | 966 | encodings.cp1252 import time: 113 | 113 | _signal import time: 122 | 122 | _abc import time: 1401 | 1522 | abc import time: 1357 | 2879 | io import time: 181 | 181 | _stat import time: 1794 | 1975 | stat import time: 2946 | 2946 | _collections_abc import time: 1108 | 1108 | genericpath import time: 2004 | 3111 | ntpath import time: 2994 | 11025 | os import time: 1277 | 1277 | _sitebuiltins import time: 5237 | 5237 | sitecustomize import time: 1356 | 1356 | usercustomize import time: 6361 | 25254 | site import time: 1623 | 1623 | warnings import time: 278 | 278 | itertools import time: 1877 | 1877 | keyword import time: 205 | 205 | _operator import time: 1821 | 2025 | operator import time: 1326 | 1326 | reprlib import time: 1163 | 1163 | _collections import time: 4015 | 10680 | collections import time: 1702 | 1702 | types import time: 217 | 217 | _functools import time: 2845 | 15442 | functools import time: 2614 | 2614 | enum import time: 200 | 200 | _sre import time: 1563 | 1563 | sre_constants import time: 1938 | 3500 | sre_parse import time: 2976 | 6675 | sre_compile import time: 165 | 165 | _locale import time: 1200 | 1200 | copyreg import time: 2944 | 13596 | re import time: 1141 | 1141 | token import time: 3498 | 18235 | tokenize import time: 1570 | 35246 | linecache :1: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives import time: 2598 | 39466 | distutils import time: 2423 | 41889 | distutils.version ```

[thesamesam]

In Gentoo:

# SETUPTOOLS_USE_DISTUTILS=stdlib python -m timeit -n 1 -r 1 'import distutils.version'
<timeit-src>:6: DeprecationWarning: The distutils package is deprecated and slated for removal in Python 3.12. Use setuptools or check PEP 632 for potential alternatives
1 loop, best of 1: 2.32 msec per loop

# SETUPTOOLS_USE_DISTUTILS=local python -m timeit -n 1 -r 1 'import distutils.version'
1 loop, best of 1: 414 msec per loop

Note that the number of setuptools plugins installed probably matters (which is why using setuptools itself is slow).