[8] Automated Tagging and Validation of tags for all packages in NPM ecosystem

[rootAvish]

Epic: #1248

User Story

The PGM will be able to get automated tags for NPM packages so that it can be trained to show alternate and companion recommendations for the NPM ecosystem.

Description

In order to show recommendations for the NPM ecosystem, we need to train our current recommendation engine(the PGM) using a package tag map and a set of reference manifest files(reference package.json) of that ecosystem. In order to create the package tag map we need to have tags for the packages of the ecosystem for which we are training the PGM.

This tasks involves furthering the work done in the previous sprint to tag NPM packages automatically to generate the package tag map. We will containerize the tagging script and push it to prod to collect tags for all the NPM packages for which we had previously collected the Github readme/project description page.

We will continue this process iteratively by evaluating the accuracy of the tags and re-running the tagger with improvements till we attain optimal accuracy.

Acceptance Criteria

• There will be tags present in a json in the corresponding S3 bucket
• We will also have the ROUGE-1 accuracy measure of these tags

Task list

• [x] Set up an automated dev cluster workflow
• [x] Write the endpoint in the kronos service to push the job to EMR
• [x] Modify the pytextrank code to not require a set of files for every stage but operate on strings
• [x] Write the enhancements in the tagger to break down n-grams based on parts of speech
• [x] Figure out a way to not have n-gram based tags in the final output
• [x] Write the evaluation script as a job to calculate the ROUGE-1 recall
• [x] Perform the first tagging iteration and calculate the ROUGE-1 recall score
• [ ] (Experimental) Check the running time of the tagger and look for ways to parallelize parts of the code using apache spark
• [ ] (Experimental) Implement any parallelization logic.

[rootAvish]

https://docs.google.com/a/redhat.com/spreadsheets/d/1bGq1z4NpzGdPcDFl0MZ-i3qNxECej-aWe2ZtLYFwK8k/edit?usp=sharing <-- numbers from first tagging run

[rootAvish]

Future scope: Add some parallelization logic and run subsequent runs of the tagger.