Automate inter-article linking

[arvindpdmn]

Words or phrases in an article can link to other articles within Devopedia. An author can code these links manually. However, we want the Devopedia platform to automatically add the links.

For example, consider the sentence "Database replication, partitioning and clustering are concepts related to sharding." which appears in an article on Database Sharding. Database Replication is another article on Devopedia. The NLP model should identify that the phrase "Database replication" should be linked to that article.

We don't want too many links in a page. Therefore, some ranking must be done to display only some of the links. This may be a separate model or handled as a post-processing task.

Here are some useful observations:

• Exact match: "Database replication" links to article titled Database Replication
• Contextual match: "partitioning" appearing in a sentence about databases links to article titled Database Partitioning
• Concept match: "abstract function" and "pure virtual function" mean the same thing and should link to the same article
• Phrasal variations: "cloud platforms for IoT" links to article titled IoT Cloud Platforms
• Sub-match: "hypothesis testing" links to article titled Hypothesis Testing and Types of Errors
• Acronyms match: "CSS" links to article titled Cascading Style Sheets
• Maximal match: "CSS-in-JS" links to article titled CSS-in-JS and avoids linking to articles CSS or JS
• Ignore punctuation: "CSS in JS" links to article titled CSS-in-JS
• Ignore variations: case insensitive, singular/plural forms, US/UK spelling, etc.: "ACID transaction" links to article titled ACID Transactions
• Combinations of the above: "basics of OOP" links to article titled Object-Oriented Programming Concepts

[arvindpdmn]

Resources:

• Wikify: http://web.eecs.umich.edu/~mihalcea/papers/mihalcea.cikm07.pdf
• BERT: https://www.aclweb.org/anthology/K19-1063/
• PEL-BERT: https://arxiv.org/abs/2002.00744
• Recent trends: https://github.com/izuna385/Entity-Linking-Recent-Trends
• Papers with Code: https://paperswithcode.com/task/entity-linking
• spaCy: https://github.com/explosion/projects/blob/master/nel-emerson/scripts/notebook_video.ipynb
• https://cloud.google.com/architecture/building-real-time-embeddings-similarity-matching-system
• https://github.com/UKPLab/sentence-transformers

[arvindpdmn]

Related to #423 and #96