Keyword Extraction : Dependency Parsing Links

[codingAku]

Issue Description

With Issue #11, the keywords of requirements for tracing is extracted. However, the initial trace results are noisy to operate. The noise needs to be reduced with more relevant keywords. One way to provide this is with dependency parsing links.

Step Details

Steps that will be performed:

• [x] Create a program that shows dependency parsing links.
• [x] Use the program on keywords extracted.
• [x] Increase the importance of keywords with parsing links.

Final Actions

The findings need to be documented in Wiki page.

Deadline of the Issue

10.04.2023 - 23.59

[codingAku]

The dependency parser program has been implemented, and the Results for the requirements of BUcademy have been added. The comparison and analysis of the usage will be discussed in Meeting 7.

Additionally, word frequency program was implemented, to extend the stopword list.

[codingAku]

Update: The results of word frequency for 5 keywords and 10 keywords on BUcademy requirements are as follows:

[codingAku]

Using the dependency links and Part-of-Speech Tagging, enhanced keyword extractor pipeline implemented.

[ersoykadir]

First results with new keywords added to repo. First impressions:

• prefix and suffixes seems to effect the results, we might want to preprocess(stemming?) issue documents as well.
  • verification codes vs verification code
• noun phrases duplicates
  • for email address phrase, email is kept and address is removed from keywords. We should keep both or remove both.
• noun phrase triples,
  • email verification process, produces two phrase like email verification and verification process
  • this can be improved, but do we want to improve?
• verb-obj pairs is constructed with single noun
  • edit profile page, produces edit page
  • enter verification codes produces enter codes
  • we can improve with verb-nounphrase, however enter codes might have higher recall!
• For now, only direct objects for verb-obj pairs are captured, it can be improved with pobj(object of preposition)
  • agree to privacy policy link is missing, can be recovered with pobj link from 'to' preposition

[codingAku]

UPDATE: Keyword lemmatizer implemented in PR #17

[codingAku]

Keyword extraction completed.