search

MaartenGr / PolyFuzz

Fuzzy string matching, grouping, and evaluation.
https://maartengr.github.io/PolyFuzz/
MIT License
725 stars 68 forks source link

Update sparse-dot-topn to v1 #77

Open RUrlus opened 3 months ago

RUrlus commented 3 months ago

Hi @MaartenGr,

We recently refactored sparse-dot-topn significantly and released v1.

The most significant improvements are:

The changes are significant enough that we released a new major version which deprecates awsome_cossim_topn. I also noticed that you encountered a bug when top-n is 1, I added a test-case for this and the issue no longer exists.

The new implementation does not sort the scores but rather returns the matrix in the order as if you didn't select the top-n, i.e. sp_matmul(A, B) == sp_matmul_topn(A, B, B.shape[1]). It wasn't directly clear to me if you (implicitly) depend on the result being sorted so I left sorting on (it has no performance penalty).

MaartenGr commented 3 months ago

Thanks for sharing this, I completely missed this new release!

It wasn't directly clear to me if you (implicitly) depend on the result being sorted so I left sorting on (it has no performance penalty).

Thanks for already sorting this, I might have missed it otherwise. Indeed, the code expects it to be left sorted.

I see that the tests fail but they also use quite old Python versions which have minimal/no support anymore. Could that be the issue?

RUrlus commented 3 months ago

Ah yes, I hadn't realized. We don't support 3.7 through the binding library which is 3.8+. So unfortunately that's a hard limit for us.

We could condition the minimum version based on the python version and add a thin wrapper around the old API to make it compatible with the new one.

RUrlus commented 3 months ago

Yes, my bad I changed the filename last minute. I'll push a fix in a bit.

MaartenGr commented 3 months ago

Thanks for the changes, it seems that the pipeline has problems importing the function you created.

RUrlus commented 3 months ago

Sorry, my laptop doesn't support 3.7 so I'd perhaps relied too much on the CICD for this work smoothly. I figured out the issue, apparently sys.version_info > (3, 7) is true on CPython 3.7... Fixed now, the models/test_tfidf.py passed locally on 3.7

MaartenGr commented 2 months ago

It seems that there are some tests failing. Not sure why that is happening though.

RUrlus commented 4 weeks ago

Hi @MaartenGr, sorry for the stall on this, I'm hoping to pick this back up soon. I did run into a (known) issue with multiple OpenMP binaries being loaded but I'll turn of the multi-threading on our end.