feat(semanticTextSim): Semantic Text sim algorithm using doc2vec

[hastagAB]

Description

New Open Source License Scanning Algorithm: Semantic Text Similarity find similarity between documents according to its semantics. The Gensim implementation of Doc2Vec converts the whole document (unlike word2vec) into vector with their labels. The Doc2Vec model is trained using the filename as its label and license text as the document. The current training dataset is the txt format of license-list-data provided by SPDX.

Files

• semanticTextSim.py (Implementation of Algorithm)
• spdxDoc2Vec.model (the Trained model in binary)
• train.py (Code to train the model)
• text (Folder containing all the spdx license dataset)

Test

• Test the agent for scanning any file for license statements

  atarashi -a semanticTextSim <pathToFile> Currently, it returns the license name with the highest Cosine Sim Score.

  Note: The agent is able to return top ten most similar license names with its sim score

• Train the model (Optional)

  cd to semanticTextSim folder. Run Command: python train.py

[amanjain97]

@hastagAB Can you please share the results about time taken to find the license and precision of the algorithm.

[amanjain97]

@hastagAB When these license files are used ? It seems that a lot of files may slow down the build process. If they are only used once, (I think only for creating doctovec model) so we can download during build process and create the model and delete them later. It will make the process more clean (I am not sure). @GMishx Please review the process.

[hastagAB]

@hastagAB Can you please share the results about time taken to find the license and precision of the algorithm.

I have tested all the algorithms using our evaluator and created a report to check the accuracy and time.

• Atarashi-Report.pdf

As of now, semanticTextSim has very less accuracy because the dataset used and the code is at its primitive stage. This PR is created to add the primary establishment of the doc2vec ML model and the model will be improved gradually by manually creating a better training dataset.

[hastagAB]

@hastagAB When these license files are used ? It seems that a lot of files may slow down the build process. If they are only used once, (I think only for creating doctovec model) so we can download during build process and create the model and delete them later. It will make the process more clean (I am not sure). @GMishx Please review the process.

Yes, I totally agree with your suggested method. An alternative method might be to re-use the model's binary file only. once the model is created we don't need to train it again and again, so we can skip the training part as well as the files used for training. What do you suggest? @amanjain97 @GMishx