search

pytoolz / toolz

A functional standard library for Python.
http://toolz.readthedocs.org/
Other
4.65k stars 258 forks source link

`_check_sigspec` is slow #495

Open uriva opened 4 years ago

uriva commented 4 years ago

Hi, while optimizing some code I noticed an unreasonable amount of the time is spent in _check_sigspec. I suspect this is the heavy usage of inspect methods in curry code.

I think this is not justified as one can run inspect stuff once, and store for later.

Did anyone encounter this?

uriva commented 4 years ago
2224/1 0.01107 0.01107 3.424 3.424 html_parser.py:200(_parse)
3592/1 0.004297 0.004297 3.423 3.423 functional_generic.py:162(pipe)
3981/2 0.003494 0.001747 3.423 1.711 functional_generic.py:125(ternary_inner)
110576 0.1263 1.142e-06 2.802 2.534e-05 functoolz.py:301(call)
31393 0.07909 2.519e-06 2.317 7.382e-05 functoolz.py:309(_should_curry)
116488/65010 0.1243 1.911e-06 1.546 2.378e-05 functoolz.py:829(_check_sigspec)
12498/10521 0.00642 6.102e-07 1.422 0.0001351 functional.py:99(star_and_run)
9315/7713 0.02187 2.836e-06 1.266 0.0001642 functional_generic.py:166(_curry_helper)
2934/1332 0.007844 5.889e-06 1.209 0.0009078 html_parser.py:183(_shift_reduce)
297127/177935 0.05967 3.353e-07 1.188 6.675e-06 ~:0()
71689/42691 0.02959 6.931e-07 1.186 2.778e-05 itertoolz.py:370(first)
1368 0.007718 5.642e-06 1.175 0.000859 html_parser.py:166(_reduce)
79441 0.04964 6.249e-07 1.173 1.477e-05 inspect.py:3091(signature)
3826/3219 0.00591 1.836e-06 1.149 0.0003569 functional_generic.py:209(indirection)
79441 0.05162 6.497e-07 1.124 1.415e-05 inspect.py:2839(from_callable)
79443/79442 0.4248 5.348e-06 1.072 1.35e-05 inspect.py:2206(_signature_from_callable)
61964/30857 0.03227 1.046e-06 1.043 3.38e-05 functional.py:306(apply_inner)
57132/31393 0.06314 2.011e-06 0.9531 3.036e-05 functoolz.py:917(is_partial_args)
7919 0.005044 6.369e-07 0.8553 0.000108 html_parser.py:157(_apply)
51478/25739 0.09437 3.666e-06 0.6487 2.52e-05 functoolz.py:891(has_varargs)
65483/65077 0.03838 5.898e-07 0.5533 8.502e-06 ~:0()
25739 0.03523 1.369e-06 0.4799 1.864e-05 _signatures.py:643(signature_or_spec)
25739 0.04606 1.79e-06 0.4523 1.757e-05 _signatures.py:756(_is_partial_args)
51478 0.01648 3.201e-07 0.3955 7.683e-06 _signatures.py:761()
25739 0.03391 1.318e-06 0.379 1.473e-05 _signatures.py:743(check_partial)
37047 0.2379 6.423e-06 0.3661 9.882e-06 inspect.py:2889(_bind)
31393 0.03032 9.659e-07 0.3496 1.114e-05 functoolz.py:335(bind)
31393 0.02092 6.664e-07 0.3435 1.094e-05 inspect.py:3027(bind_partial)
31393 0.1279 4.075e-06 0.3193 1.017e-05 functoolz.py:194(init)
4568/4190 0.00218 5.203e-07 0.2955 7.051e-05 functoolz.py:779(call)
25739 0.03551 1.379e-06 0.2534 9.847e-06 _signatures.py:788(_has_varargs)
25739 0.01406 5.463e-07 0.2111 8.2e-06 _signatures.py:792()
25739 0.01684 6.543e-07 0.197 7.654e-06 _signatures.py:783(check_varargs)
79441 0.09083 1.143e-06 0.1863 2.345e-06 inspect.py:493(unwrap)
uriva commented 3 years ago

to whom it may concern, gamla is a drop in replacement for most of what toolz does, and does it considerably faster by avoiding issues such as this.

groutr commented 3 years ago

@uriva can you share the code that you were profiling? It would make it easier to understand what is happening.

uriva commented 3 years ago

It's been awhile, but iirc it was just using a curried function in a loop, or building a composition using compose in a loop.