Evaluation of Prototype Results

codingAku commented 1 year ago

Issue Description

The initial prototype with captured traces exists. The next step is to compare the results with a set of ground truth and document the evaluation.

Initializing Ground Truth Set: For this purpose, a subset of requirements from the Learnify App Repository will be selected and their traces will be documented. This will be a manual analysis made by @codingAku and @ersoykadir. The selected requirement subset is as follows:

1.1.1.2.1. Users shall provide their usernames and passwords to log in. - @codingAku
1.1.1.4.2. Users shall be authenticated after verification and be logged in. - @ersoykadir
1.1.2.14.1. Users shall be able to annotate post images and texts in learning spaces. @codingAku
1.1.2.14.2. Users shall be able to view annotations made by other users. - @ersoykadir
1.1.2.15.1. Users shall be able to see all learning spaces they created or enrolled in. - @codingAku
1.1.3.2.5.1. Participants shall be able to create community events for that learning space. - @codingAku
1.1.2.2 Users shall be able to edit their profile page. - @ersoykadir
1.1.3.2.7.1. Participants of a learning space shall be able to create discussion posts. - @codingAku

The results of semantically related traces in the software artifacts will be used as ground truth set for evaluation.

The results of initial prototype:

The tool will be run with two different trace selection methods; keyword extraction and word vectors. The results will be compared with the ground truth set to document recall and precision values.