feat(doc2vec) : Semantic Text Similarity Algorithm with dataset & training code

[hastagAB]

Description

New Open Source License Scanning Algorithm: Semantic Text Similarity find similarity between documents according to its semantics. The Gensim implementation of Doc2Vec converts the whole document (unlike word2vec) into vector with their labels. The Doc2Vec model is trained using the filename as its label and license text as the document. The current training dataset is the txt format of license-list-data provided by SPDX.

Files

• semanticTextSim.py (Implementation of Algorithm)
• spdxDoc2Vec.model (the Trained model in binary)
• train.py (Code to train the model)
• text (Folder containing all the spdx license dataset)

Test

• Test the agent for scanning any file for license statements

  atarashi -a semanticTextSim <pathToFile> Currently, it returns the license name with the highest Cosine Sim Score.

  Note: The agent is able to return top ten most similar license names with its sim score

• Train the model (Optional)

  cd to semanticTextSim folder. Run Command: python train.py

[amanjain97]

Please fix the travis CI. Code looks good. Nice work. I think the accuracy of the model is not good but we can improve later. @hastagAB

[hastagAB]

Have rebasing issue in branch. Unable to fix. Closing this