Open jaraco opened 6 months ago
It looks like the bulk of the import time is spent in re
, ast
, and dis
:
import time: self [us] | cumulative | imported package
import time: 223 | 223 | types
import time: 991 | 1214 | enum
import time: 47 | 47 | _sre
import time: 187 | 187 | re._constants
import time: 270 | 456 | re._parser
import time: 81 | 81 | re._casefix
import time: 257 | 839 | re._compiler
import time: 81 | 81 | itertools
import time: 100 | 100 | keyword
import time: 40 | 40 | _operator
import time: 202 | 241 | operator
import time: 131 | 131 | reprlib
import time: 41 | 41 | _collections
import time: 554 | 1146 | collections
import time: 32 | 32 | _functools
import time: 590 | 1766 | functools
import time: 125 | 125 | copyreg
import time: 384 | 4326 | re
import time: 1325 | 1325 | _ast
import time: 458 | 458 | contextlib
import time: 861 | 6968 | ast
import time: 281 | 281 | _opcode
import time: 160 | 160 | _opcode_metadata
import time: 220 | 660 | opcode
import time: 594 | 1253 | dis
import time: 129 | 129 | collections.abc
import time: 116 | 116 | importlib
import time: 43 | 158 | importlib.machinery
import time: 141 | 141 | token
import time: 20 | 20 | _tokenize
import time: 569 | 729 | tokenize
import time: 1548 | 10783 | inspect
If those imports could be deferred, that would likely address the concern.
Visualised with https://github.com/nschloe/tuna:
./python.exe -X importtime -c 'import inspect' 2> /dev/null ; ./python.exe -X importtime -c 'import inspect' 2> import.log && tuna import.log
Some notes:
re
is imported both directly by inspect
and also by ast
. Removing one of the two imports wouldn't actually help import time.ast
is used heavily in inspect.py, including as a base class. Deferring it would be difficult.re
is used only once in inspect.py in a relatively obscure case (formatannotation
); that import could be deferred.re
is similarly only used once in ast.py and for a single function (https://docs.python.org/3.10/library/ast.html#ast.get_source_segment).inspect
imports re
.enum
is imported by inspect
and re
. In both cases, it's for defining enums in the global scope, so it cannot easily be deferred.ast
spends about 1 ms of import time on things outside importing other modules. What is that time being spent on, and can we improve it?yeah, I looked at this in the past and decided that it wasn't realistically possible to improve this situation without either heavily refactoring inspect.py
or making the code quite a lot uglier in order to defer imports. But I'd be very happy to be proved wrong if somebody can see a good way of making progress here :-)
I think we can get rid of the re
import, which would shave off about 1 ms.
- I haven't checked if there are other paths by which
inspect
importsre
.
re
is also used in tokenize
:
re.compile
s which could be moved into a function with a local import re
re.compile
in a dedicated _compile
function which looks unused and could be removed or given a local import re
Special = group(*map(re.escape, sorted(EXACT_TOKEN_TYPES, reverse=True)))
which looks harder to refactorA bunch of the re
-module usage in tokenize
is backwards-compatibility stuff that isn't actually used by tokenize
's core functionality anymore. We could easily deprecate that now, I think. (Previously discussed in https://github.com/python/cpython/issues/104719. Looks like I promised to open a followup issue there about deprecating the obsolete constants but never did so...)
The half of the re
import cost is in importing enum
and collections
. It is difficult to find a program that does not import them. Look not only at the cumulative time, but also at the pure time. The slowest are modules that create a large number of classes, especially with metaclasses.
Using a tool I have for lazy imports/exports I did some quick testing on deferring the imports for ast
, dis
, re
and tokenize
in inspect
to see what impact that has on both inspect
and dataclasses
.
This involved making inspect
a package and splitting out the definitions of _ClassFinder
and the compiler flags into separate files to be lazily imported if needed. It didn't break ./python -m test -v test_inspect
at least (after making __main__.py
).
My machine isn't stable enough for really consistent testing but this gives a rough idea of the result.
Benchmarks 4 and 5 include actually creating a dataclass. The dataclass_no_docstring
difference is because without a docstring dataclasses calls inspect.signature
to generate a basic docstring[^1] and this pulls in the code that uses dis
[^2].
[^1]: Is this documented anywhere?
[^2]: This also means -OO will always pull in dis
.
From post by @AlexWaygood in https://github.com/python/cpython/issues/117372#issuecomment-2047024740_:
Perhaps inspect could be refactored to limit the import-time cost.
Currently on my m3 mac pro, on the second attempt, the import times are 1.2ms and 7.9ms (cumulative):
I've searched the issue tracker and didn't see anything tracking this concern, so right now it's a naïve feature request.
Related: https://github.com/python/cpython/issues/108901
Linked PRs