After playing around with the real reporting data, I felt like our current description match logic (just any single token needs to match) was a bit too generous -- you can get a lot of hits sometimes, and they aren't all meaningful!
So, to help with that, I made two changes...
I migrated our stop word filtering to the library [stopword](https://www.npmjs.com/package/stopword) -- it's a bit more comprehensive than our current list, and is now one less thing for us to maintain! 🎉
We now have a stricter threshold to clear. The ratio of (the total number of hits in the description) / (the number of meaningful tokens in the search term) must be >= 0.6 (60%).
Why did that I take that approach? Several notes...
It's common practice to use a ratio threshold in these kinds of searches. This allows the amount to scale with the length of the search.
I specifically wanted the case where if you used two search terms, you needed two search hits, but I didn't want to overshoot too far, which pushed me to 0.6.
I wanted to use total hits in the numerator in case a word comes up a lot in the description
I wanted to use meaningful words in the denominator so that stop words don't artificially inflate the requirements and make things too strict
Testing Instructions
Unit tests should still pass!
Best way to test is to use a semi-realistic reporting config locally REACT_APP_REPORTING_CONFIG_NAME='realistic-config' yarn start, and just play around and trust your gut! How does it feel? Do we need to pull back or push further?
What Does This PR Add/Change?
After playing around with the real reporting data, I felt like our current description match logic (just any single token needs to match) was a bit too generous -- you can get a lot of hits sometimes, and they aren't all meaningful!
So, to help with that, I made two changes...
[stopword](https://www.npmjs.com/package/stopword)-- it's a bit more comprehensive than our current list, and is now one less thing for us to maintain! 🎉Why did that I take that approach? Several notes...
Testing Instructions
Unit tests should still pass!
Best way to test is to use a semi-realistic reporting config locally
REACT_APP_REPORTING_CONFIG_NAME='realistic-config' yarn start, and just play around and trust your gut! How does it feel? Do we need to pull back or push further?Issues
Related to #
Closes #