Open PaulMcInnis opened 3 years ago
Thinking about this... we can simply identify matches in the backwards direction after we perform the matching in the forwards direction? (Currently its incoming job vs master dict, but we want master job vs master dict)
Accidentally tapped close...
Description
Currently we remove duplicates everywhere but we only remove duplicates by description (TFIDF) between the masterlist and all scrape data.
We should allow masterlist to perform a content match to itself.
Steps to Reproduce
Expected behavior
We should be running TFIDF inter-scrape data and inter-master csv
Actual behavior
Only duplicates in the ncoming dict are identified based on master CSV
Environment