Speed up TypeQuery - Githubissues

python / mypy

Optional static typing for Python

https://www.mypy-lang.org/

Other

18.35k stars 2.81k forks source link

Speed up TypeQuery #7128

Open JukkaL opened 5 years ago

JukkaL commented 5 years ago

mypy.type_visitor.TypeQuery is constructed often with the any or all builtin function as an argument. The function is used as a callable inside the query. Calling callable variables is kind of slow when compiled with mypyc. Since type queries are used a lot, switching to an integer constant or a compiled object instead of a callable could speed up runtimes measurably, perhaps by as much as 0.5% or so (random guess). Since the optimization is pretty simple, this may be worth doing.

Discussion:

Maybe have separate type query visitor which produces boolean values. The strategy could be specified as an integer constant.
For non-bool type queries, experiment with using an object instead of a callable for the strategy, since callables are currently pretty slow when compiled.
Maybe also special case self.strategy([]) since it's used a lot.

mridullpandey commented 4 years ago

hi @JukkaL i want to work on can you assign it to me thanks :-)

mridullpandey commented 4 years ago

can you please let me know how to get started

JukkaL commented 4 years ago

@mridullpandey I've assigned this to you. Here are some hints:

Figure out how to run mypy in compiled mode so that you can see whether you've improved performance. A good benchmark is type checking mypy. You need to figure out a way to measure the time consistently with accuracy of about 0.2% to validate whether you've made progress (or that at least you haven't regressed anything). Victor Stinner has written about how to do stable benchmarking, in case you want to learn about this.
Create a copy of TypeQuery that is specialized for bool instead of accepting an arbitrary type.
Replace the strategy constructor argument with an integer constant (corresponding to either any or all).
Implement a strategy method that uses the integer constant to implement whatever is needed.

vbarbaresi commented 4 years ago

I just opened a PR with my attempt at TypeQuery specialized for bool

In your first description of the issue, you wrote:

For non-bool type queries, experiment with using an object instead of a callable for the strategy, since callables are currently pretty slow when compiled.

I wanted to try that out too. You mean replacing strategy type here https://github.com/python/mypy/blob/ec76ccafa757c6d8fbb2c83195170bf75e0ff324/mypy/type_visitor.py#L257 by a new type of object that can be compiled by mypyc? There would still have to be a callable method someplace in that object, right?

Hyperion101010 commented 3 years ago

Hi there I am Shivam, I know Python to an intermediate level, I am interested to solve this issue, will you assign me?

JukkaL commented 3 years ago

@Hyperion101010 There already is an open PR here: https://github.com/python/mypy/pull/9604. However, it's not clear if it speeds things up significantly. If you want to help, you can try measuring the impact of the PR on performance. Note that if the difference is small, you may need a specialized environment, such as a machine where you've turned off turbo boost for a constant CPU frequency, and where you've disabled most background processes.