De-duplicating of nodes via similarity & community detection

[cassinius]

Use something along the line of the ingredient de-duplicating pipeline demonstrated in a neo4j tutorial using the BBC goodfood ingredients

Pipeline

• [ ] download the goodfood dataset (scraping required !) - or something equivalend
• [ ] normal NLP preprocessing steps
  • [ ] character encodings
  • [ ] tokenization
  • [ ] stemming (plurals)
  • [ ] stopwords / length etc.
• [ ] connect tokens to an ingredient
  • ingredient: cherry tomato => parts: cherry and tomato
• [ ] Use string distance to create similarity edges
  • [ ] sorensenDiceSimilarity ??
• [ ] Use phonetic similarity to create similarity edges
  • [ ] doubleMetaphone ??
• [ ] Run a community detection algorithm (like Louvain) to cluster similar ingredients together