Open sondalex opened 5 months ago
Thanks for the report! @KennethEnevoldsen, this is something you and Dan worked on - can you take a look?
Hi @sondalex, thanks for the report. The reason why we use the boolean array is to avoid counting duplicates twice. E.g. imagine the sentence: "a sentence. a sentence."
It is clearly all duplicates so n_duplicate_characters = n_characters and the fraction 1/1. However, you can get in a situation where you count some characters twice duplicate 2-gram ("a sentence", "sentence.") to get a n/1 fraction where n>1. A solution to this would simply be to create a boolean array of characters (instead of the current array of tokens) and set the respective characters to True (I believe this would also be faster).
However with your specific example: range=(3, 7), span=test. This is
, does not seem to be a duplicate, which seems to a bug, though I can't see a clear point where that error should happen.
Is this bug something you'd have time to take a closer look at, @KennethEnevoldsen? Otherwise, a PR is more than welcome, @sondalex!
Kenneth is busy with MTEB sprint and I'm wrapping up my dissertation, so neither of us have the time to look into this right now. You're more than welcome to submit a PR if you find a solution for this, @sondalex, otherwise it might have to wait a while before we get the chance to take a deeper look.
Hi @KennethEnevoldsen, thank you for your explanation and thank you @HLasse for following up. I am planning to look more in depth into this when I have some time. I will definitely PR once I find a solution.
I have added unit test for duplicate_ngram_fraction on my fork's branch add_quality_unittest . The test fails which makes wonder whether my test expected values are wrong or whether the current behaviour is not intended.
The report of my test
pytest tests/test_quality.py -k test_duplicate_ngram_fraction
:With a bit more debug information:
(result for n=2)
Should the spans be equivalent across the two for loop blocks ?
Respectively, 1) https://github.com/HLasse/TextDescriptives/blob/93b1d59278dc86cd4e8df80247e8a1a5afda2f59/src/textdescriptives/components/quality.py#L262-L270 2) https://github.com/HLasse/TextDescriptives/blob/93b1d59278dc86cd4e8df80247e8a1a5afda2f59/src/textdescriptives/components/quality.py#L274-L276
In this case, the function could be modified as such: