Divide and Conquer the EmpiRE: A Community-Maintainable Knowledge Graph of Empirical Research in Requirements Engineering
A Sustainable Literature Review for Analyzing the State and Evolution of Empirical Research in Requirements Engineering

[](https://github.com/okarras/EmpiRE-Analysis) [](https://github.com/okarras/EmpiRE-Analysis/issues) [](https://github.com/okarras/EmpiRE-Analysis/issues) [](LICENSE) [](https://mybinder.org/v2/gh/okarras/EmpiRE-Analysis/HEAD?labpath=%2Fempire-analysis.ipynb) [](https://doi.org/10.5281/zenodo.8083528) [](https://orkg.org/observatory/Empirical_Software_Engineering?sort=combined&classesFilter=Paper,Comparison,Visualization) [](https://orkg.org/api/rdf/dump)

[!IMPORTANT]
The content of this repository was reviewed on the Artifact Evaluation track of the 32nd IEEE International Requirements Engineering Conference 2024 and received the , the , and the .

Table of Contents

About the Project

This project contains the constantly updated data, analysis, and results of a sustainable literature review on the state and evolution of empirical research in requirements engineering (RE) using the developed KG-EmpiRE.

KG-EmpiRE is a community-maintainable knowledge graph (KG) of empirical research in requirements engineering based on scientific data extracted from currently 680 papers published in the research track of the IEEE International Conference on Requirement Engineering from 1994 to 2022. We are currently organizing scientific data in KG-EmpiRE using a defined template for the six themes research paradigm, research design, research method, data collection, data analysis and bibliographic metadata with the long-term plan to expand the themes.

KG-EmpiRE itself is maintained in the Open Research Knowledge Graph (ORKG). The ORKG is a cross-domain and cross-topic research knowledge graph (RKG) with corresponding technical infrastructure and services for the organization of Findable, Accessible, Interoperable, and Reusable (FAIR) scientific data from papers in accordance with the FAIR data principles. The TIB - Leibniz Information Centre for Science and Technology develops and maintains the ORKG permanently and has committed itself to the long-term archiving of all data. As a central access point to all curated papers in KG-EmpiRE, we established a more general ORKG observatory on empirical research in software engineering. In addition, the ORKG provides a RDF dump of all its data, including the most recent data from KG-EmpiRE. We also store the data used for analysis as CSV files.

[!NOTE]
For each related publication, we provide a folder containing the respective CSV files to enable the replication of the results. The details on the replication of the results can be found in the usage instructions.

In this project, we perform the data analysis of KG-EmpiRE, which has two purposes:

1. We evaluate the coverage of the curated topic of empirical research in RE by KG-EmpiRE.

2. We gain insights into the state and evolution of empirical research in RE.

The data analysis is based on competency questions regarding empirical research in SE, including RE, derived from the vision of Sjøberg et al. (2007). They describe their vision of the role of empirical methods in SE, including RE, for the period of 2020 – 2025 by comparing the "current" state of practice (2007) with their target state (2020 - 2025). We analyzed these descriptions and derived a total of 77 competency questions. The number of competency questions answered reflects the coverage of the curated topic in KG-EmpiRE (1), and the answers to competency questions provide insights into the state and evolution of empirical research in RE (2). For each competency question that can be answered with KG-EmpiRE (currently 16 of 77), we specified SPARQL queries to retrieve and analyze the data of KG-EmpiRE from the ORKG. We provide all details of the analysis with its SPARQL queries, data, visualizations, and explanations in the Jupyter Notebook hosted on binder for interactive replication and (re-)use, always using the most recent data from KG-EmpiRE.

The analysis of the individual competency questions always follows the same structure:

1. Data Selection: Explaining the competency question and the required data for the analysis.
2. Data Collection: Executing the specified SPARQL query to retrieve the data.
3. Data Exploration: Exploring the data, including its cleaning and validation, to prepare the data for data analysis.
4. Data Analysis: Analyzing the data and creating visualizations.
5. Data Interpretation: Interpreting the data and derive insights.

Overall, this project serves to make the data, analysis, and results openly available in the long term according to the FAIR data principles to enable a replicable, (re-)usable and thus sustainable literature review.

In this way, this project can be used for:

1. Replication of the results from the related publications.

2. (Re-)use of KG-EmpiRE with its most recent data.

3. Repetition of our research approach for sustainable literature reviews on other topics.

(back to top)

Folder Structure and Files

In the following, we first show a graphical overview of the folder structure and files of the project before we describe them in more detail.

Graphical Overview

EmpiRE-Analysis/
┣━ .vscode/
┃   ┗━ settings.json
┣━ Figures/
┃   ┣━ CQ1/...contains 2 visualizations
┃   ┣━ CQ2/...contains 8 visualizations
┃   ┣━ CQ3/...contains 2 visualizations
┃   ┣━ CQ4/...contains 4 visualizations
┃   ┣━ CQ5/...contains 2 visualizations
┃   ┣━ CQ6/...contains 3 visualizations
┃   ┣━ CQ7/...contains 6 visualizations
┃   ┣━ CQ8/...contains 2 visualizations
┃   ┣━ CQ9/...contains 2 visualizations
┃   ┣━ CQ10/...contains 2 visualizations
┃   ┣━ CQ11/...contains 2 visualizations
┃   ┣━ CQ12/...contains 4 visualizations
┃   ┣━ CQ13/...contains 2 visualizations
┃   ┣━ CQ14/...contains 2 visualizations
┃   ┣━ CQ15/...contains 2 visualizations
┃   ┗━ CQ16/...contains 2 visualizations
┗━ SPARQL-Data/
    ┣━ ESEM 2023/...contains 22 CSV files from the SPARQL queries to replicate the results from the ESEM 2023 publication
    ┣━ Most recent data/...contains 22 CSV files from the SPARQL queries to use the most recent data of KG-EmpiRE
    ┗━ RE 2024/...contains 22 CSV files from the SPARQL queries to replicate the results from the RE 2024 publication
┣━ Supplementary materials/
┃   ┣━ approach.png
┃   ┣━ Detailed list of all 77 competency questions.xlsx
┃   ┣━ Detailed ORKG template structure.pdf
┃   ┣━ logo.jpg
┃   ┗━ Overview of all content for data extraction.pdf
┣━ CITATION.cff
┣━ empire-analysis.ipynb
┣━ LICENSE
┣━ README.md
┣━ requirements.txt
┗━ runtime.txt

(back to top)

Description of the Folders and Files

(back to top)

System Requirements

The following system requirements must be met to run this project:

[!NOTE]
We have only tested the analysis on systems with the Windows operating system, which is why we specify Windows as a necessary system requirement. However, the analysis should also run on other operating systems.

All required packages and libraries are installed automatically when running:

pip install -r requirements.txt

We use Visual Studio Code for developing and can recommend the following YouTube Tutorial for setting up Jupyter Notebooks in Visual Studio Code.

(back to top)

Installation Instructions

In the following, we explain how to install the project for execution on a local machine using a terminal, assuming that the system requirements are met.

1. Clone the repository.

git clone https://github.com/okarras/EmpiRE-Analysis

2. Navigate to the main directory of the project.

cd EmpiRE-Analysis

3. Create a virtual environment with Anaconda.

conda create --name empire_venv

4. Activate the virtual environment.

conda activate empire_venv

5. Install the required packages and libraries.

pip install -r requirements.txt

6. Start the Jupyter Notebook.

jupyter notebook empire-analysis.ipynb

(back to top)

Executable Machines and Environments

Binder (Recommended)

We use binder to provide an executable machine and environment where the project is hosted for interactive replication and (re-)use.

[!IMPORTANT]
If you follow the link below, the project will start on binder to run it in a JupyterLab. If a built image already exists, the start should take about 1 - 2 minutes. If a built image needs to be created, the start can take about 5 minutes.

Once the project has been started in JupyterLab, carry out the following steps:

1. Restart the kernel and clear all outputs.
2. Run all cells.

(back to top)

GitHub Codespaces

If you have a GitHub account, you can use GitHub Codespaces to execute the project.

[!IMPORTANT]
You can use GitHub Codespaces to start the project either in Visual Studio Code or JupyterLab, depending on your preferences.

Once the project has been started, carry out the following steps:

1. Open a terminal and install the required packages and libraries.

   pip install -r requirements.txt

2. Open the analysis script: empire-analysis.ipynb.
3. Restart the kernel and clear all outputs.
4. Run all cells.

(back to top)

Usage Instructions

In the following, we explain the steps for using the project for the two cases of replication of results from the related publications and (re-)use of KG-EmpiRE with its most recent data, assuming that the project is executed either locally or on one of the executable machines and environments.

Replication of the results from the related publications

1. In the section "2. Reusable Functions for Data Analysis" of the Jupyter Notebook, change the DATE and PATH variables from "now.strftime('%Y-%m-%d')" and "'SPARQL-Data/Most recent data/query_'" to the date and path specified for the respective related publication to use the corresponding CSV files. Below, we show the code example for the replication of the results from the RE 2024 publication. The results of ESEM 2023 publication can be replicated accordingly.

   #For using the most recent data of KG-EmpiRE, use the following date and path.
   #DATE = now.strftime('%Y-%m-%d')
   #PATH = 'SPARQL-Data/Most recent data/query_'
   
   #For replication of the related publication from ESEM 2023, use the following date and path.
   #DATE = '2023-06-26'
   #PATH = 'SPARQL-Data/ESEM 2023/query_'
   
   #For replication of the related publication from RE 2024, use the following date and path.
   DATE = '2024-04-30'
   PATH = 'SPARQL-Data/RE 2024/query_'

2. Restart the kernel and clear all outputs.
3. Run all cells.

(back to top)

(Re-)use of KG-EmpiRE with its most recent data

1. In section "2. Reusable Functions for Data Analysis" of the Jupyter Notebook, ensure that the DATE and PATH variables are set to "now.strftime('%Y-%m-%d')" and "'SPARQL-Data/Most recent data/query_'".

   #For using the most recent data of KG-EmpiRE, use the following date and path.
   DATE = now.strftime('%Y-%m-%d')
   PATH = 'SPARQL-Data/Most recent data/query_'
   
   #For replication of the related publication from ESEM 2023, use the following date and path.
   #DATE = '2023-06-26'
   #PATH = 'SPARQL-Data/ESEM 2023/query_'
   
   #For replication of the related publication from RE 2024, use the following date and path.
   #DATE = '2024-04-30'
   #PATH = 'SPARQL-Data/RE 2024/query_'

2. Restart the kernel and clear all outputs.
3. Run all cells.

(back to top)

Related Publications

The first version of KG-EmpiRE based on 570 papers from the IEEE International Conference on Requirement Engineering from 2000 to 2022 and the first analysis of the sustainable literature review on the state and evolution of empirical research in RE have been published in:

Oliver Karras, Felix Wernlein, Jil Klünder, and Sören Auer:
Divide and Conquer the EmpiRE: A Community-Maintainable Knowledge Graph of Empirical Research in Requirements Engineering, In: 2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), New Orleans, LA, USA, 2023, pp. 1-12.

The publication received the of the 17th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement 2023.

The second version KG-EmpiRE based on 680 papers from the IEEE International Conference on Requirement Engineering from 1994 to 2022 and the analysis of the sustainable literature review on the state and evolution of empirical research in RE have been published in:

Oliver Karras:
KG-EmpiRE: A Community-Maintainable Knowledge Graph for a Sustainable Literature Review on the State and Evolution of Empirical Research in Requirements Engineering, In: 2024 IEEE International Requirements Engineering Conference (RE), Reykjavík, Iceland, 2024.

The artifact received the , the , and from the Artifact Evaluation track of the 32nd IEEE International Requirements Engineering Conference 2024.

(back to top)

Executive Summary

(back to top)

Corresponding Author

Dr. rer. nat. Oliver Karras

Researcher and Data Scientist - Open Research Knowledge Graph

TIB - Leibniz Information Centre for Science and Technology

Welfengarten 1B

30167 Hannover

E-Mail: oliver.karras@tib.eu

(back to top)

How to Cite

If you want to cite this project, we suggest to use the following reference:

Oliver Karras:
Divide and Conquer the EmpiRE: A Community-Maintainable Knowledge Graph of Empirical Research in Requirements Engineering - A Sustainable Literature Review for Analyzing the State and Evolution of Empirical Research in Requirements Engineering, Computer Software, Version v1.1, https://github.com/okarras/EmpiRE-Analysis, 2024.

You can also use the "Cite this repository" function in the top right menu resulting from the included citation file format file for human- and machine-readable citation information for software and datasets. Further information can be found on the Citation File Format (CFF) website.

If you want to cite the related publications, use the references in the section related publications.

(back to top)

Released under MIT by Oliver Karras.