[summary] Info-Theory-Traceability

[danaderp]

Time: 13:00 -14:00 Purpose: To discuss results and organization of the paper in information theory for traceability Attendees: Denys Poshyvanyk, Kevin moran, and David Nader

Summary: In the meeting, it was introduced the importance of having a robust statistical method based on information theory to analyze traceability data. Usually, Software Researchers have been focusing only on evaluating models or algorithms to assess the effectiveness of traceability techniques, paying little attention to the data performance. However, using information transmission analysis on software artifacts allow us to identify inconsistencies or imbalances in the traceability data. We hypothesized that using standard Neural Unsupervised approaches, which captures concepts or semantic entities from artifacts, might not generate reliable trace-links. This research proposes a robust data science methodology to be employed in future empirical research for software traceability evaluations.

Main Discussion Points:

• David made a quick recap of Information Theory for the analysis of traceability datasets. He states that information metrics can be measured for all text-based documents and they give us an idea to what extent a corpus is good enough for traceability. If the mutual information levels are low, then recovering a trace link would be more difficult than recovering with a high level of mutual information. Neural unsupervised techniques are based on data compression. If the pattern does not exist, then the link cannot be established.
• Kevin mentioned two key aspects to enhance the research evaluation: 1) introducing a statistical analysis where all possible correlations among information measures and traceability effectiveness are exhibit and contextualized. The idea of this part of the evaluation is to show how the information theory measures complement the accuracy analysis of traceability algorithms. And 2) after observing how information measures behave under certain conditions, we are able to introduce some type of prescriptive info-traceability guidelines to inspect the traceability process. For example, if we observe that source and target artifacts are extremely imbalanced in terms of information content, then we can affirm that there is no traceability algorithm that can create reliable links until the information is balanced. We want to introduce some qualitative analysis on specific SE case studies.
• Denys highlighted the fact that we need to position this research as an approach rather than an empirical study since we are not actively making interventions on the datasets. However, we must find a way of defending the statistical methodology by showing how useful could be with concrete cases.

Open Questions and Action Items:

• [ ] What would be exactly the research questions that will link correlation analysis of entropy and the effectiveness of traceability algorithms?
• [ ] Organize a first draft of the paper with Kevin's recommendations for the evaluation section
• [ ] Check out recent traceability research that we should consider for this research
• [ ] Check out robust statistical methods and how they were introduced in software empirical analysis (i.e., Kitchenham )

[kpmoran]

Hi David, thanks for this summary. I think the most important thing to emphasize from the meeting is measuring, quantitatively, how well these information theory measures predict traceability performance. Then for certain patterns that we observe, we can prescribe some recoomendations. Just wanted to say this in my own words so I didn't forget my line of thinking ;-)

[danielrcardenas]

Paper writing task

• [ ] Related work review and update
• [ ] Explain the methodology - empirical design [David]
• [ ] Show why to use descriptive, predictive, and causal inference for info-transmission [David]
• [ ] Organize results as a summary excel
• [ ] Organize benchmarking results and identify special cases
• [ ] Benchmarking examples identification and explanation
• [ ] Define what is a benchmark
• [ ] Motivation example [David]
• [ ] Related work: 1) How traceability has been made 2) Information Theory Applied in SE

[danaderp]

Tasks Week 7.13.2021 @danielrcardenas

• [ ] Where are the boxplots?
• [ ] Answer Research Question with Results supported on real data (boxplots)
• [ ] Organize Case Studies (e.g., orphan links, min-max loss, min-max noise, etc)

Please Edit this in overleaf