dgilland
commented
9 years ago No worries about the criticism of those performance numbers :smile:. Admittedly, I have not done much in terms of profile/optimizing for speed :disappointed:.
I went ahead with some basic cProfile analysis and determined that the majority of the time was spent in is_builtin and has which were called from here and here respectively.
I did some quick rework and now the timeits are:
$ python -m timeit "
from pydash import filter_
x = [{'a': 1, 'b': 2, 'c': 3}, {'a': 2, 'b': 3, 'c': 4}]
filter_(x, {'a': 1})"
10000 loops, best of 3: 29.6 usec per loop
$ python -m timeit "
x = [{'a': 1, 'b': 2, 'c': 3}, {'a': 2, 'b': 3, 'c': 4}]
filter(lambda y: y['a'] is 1, x)"
1000000 loops, best of 3: 0.882 usec per loopwhich brings the difference down to ~34x slower than the native filter.
You can view the changes on the feature-performance branch. If you know of any other poor performance areas, I (or you if you are so inclined) can take a look and submit a patch.
UPDATE: Looks like the is_builtin change in call_callback caused some tests to fail on Python 3.4. I've since reverted that change which makes the timeit as follows:
$ python -m timeit "
from pydash import filter_
x = [{'a': 1, 'b': 2, 'c': 3}, {'a': 2, 'b': 3, 'c': 4}]
filter_(x, {'a': 1})"
10000 loops, best of 3: 49.4 usec per loopwhich is ~56x slower than the native filter.
jacobbridges
As a precursor to my question: I'm not here to cast doubt upon the usefulness of this project or the adequacy of those who have worked on it. In truth I am very impressed by this project and when I first discovered it I began using it in most of my projects! (I do lots of data mining, so a library such as this is priceless.)
But I noticed recently that some of the functions in Pydash are very slow compared to their Python counterparts. For instance consider the following code:
Now in truth I would never expect the Pydash equivalent functions to be faster than the Python counterpart, but ~150x slower?
Any idea how the speed of these functions could be improved? Since I made this performance discovery I've stopped using the pydash library so much and went back to filter + lambda...but goodness that can get ugly! I love the idea of this library, I just wished performance was better.