Closed bkhant1 closed 1 year ago
Hi, thanks for your effort and this very detailed pull request. The errors in the pipeline are my fault (connected to #384). I'll try to sort it out this week and have your PR merged as well. Looks like a great improvement already
Sounds good, thanks for taking the time to review!
@bkhant1 I've just merged the fixes in master. Can you please pull master and see if the pipeline still fails?
Just done it - the workflow needs approval to run.
LGTM! I'll merge it. Sorry again for the broken pipeline in the first place. Looking forward to see more contributions
I used the hash-encoder the other day and was surprised by the time it takes for a theorically quite lightweight algorithm.
I looked into it in a bit more detail, generating a random dataframe with 100,000 rows to measure performance. It turns out that in that case >50% of the time is spent in the
pandas.Series
constructor. Here's my notebook if you are curious!Proposed Changes
hash_fn
instead ofpandas.Series
, and useresult_type="expand"
inapply
the expand the lists into a dataframe.Result
Here's the performance table comparision
pandas.Series
list
For larger dataframes, it slashes the time it takes to
transform
in two. Here's the notebook those results are from.Follow-up
I thought I would start with this change as it's a 2-liner!
But there are still some things I would want to look into:
apply
which is know to be slow