There are some important aspects missing, especially criteria for the input space (generation), the description of the search space and why it is truly and an exponential problem, and clear discussion of threats to validity. Please refer to our paper: https://t.co/sNQK4DaaTQ?amp=1
In our paper together with SvenApel and Stefan Sobernig, we analyzed papers optimizing software configurations with variability models. Here, features (or options) of a SW system are modeled together with attributes, such as performance. Goal: Try to find the optimal config.
There are three validity issues common in most papers:
Non realisitc inputs: Attribute values have not been measured, but mostly generated by an arbitrary distribution that has no relation to value distributions in the wild. Optimizations should work with realistic data.
Exponential search space: Finding an optimal configuration in an exp number of combinations of SW features is an NP-hard problem. However, if there are only a linear number of effects, we can simply compute every configuration's value with a function having linear # of terms. Hence, the optimization task becomes trivial, because ignoring combination (or interaction) effects renders the problem to a linear one. We saw this problem simplification in all analyzed optimization papers by omiting interaction effects.
Threats to validity: We saw that early papers that define experimental setups have been reused by others. But, the threats to validity of these early setups have not been addressed and even not mentioned by those papers reusing the setup. Hence, make your limitations explicit.
From Norbert Siegmund:
There are some important aspects missing, especially criteria for the input space (generation), the description of the search space and why it is truly and an exponential problem, and clear discussion of threats to validity. Please refer to our paper: https://t.co/sNQK4DaaTQ?amp=1
In our paper together with SvenApel and Stefan Sobernig, we analyzed papers optimizing software configurations with variability models. Here, features (or options) of a SW system are modeled together with attributes, such as performance. Goal: Try to find the optimal config.
There are three validity issues common in most papers:
Non realisitc inputs: Attribute values have not been measured, but mostly generated by an arbitrary distribution that has no relation to value distributions in the wild. Optimizations should work with realistic data. Exponential search space: Finding an optimal configuration in an exp number of combinations of SW features is an NP-hard problem. However, if there are only a linear number of effects, we can simply compute every configuration's value with a function having linear # of terms. Hence, the optimization task becomes trivial, because ignoring combination (or interaction) effects renders the problem to a linear one. We saw this problem simplification in all analyzed optimization papers by omiting interaction effects. Threats to validity: We saw that early papers that define experimental setups have been reused by others. But, the threats to validity of these early setups have not been addressed and even not mentioned by those papers reusing the setup. Hence, make your limitations explicit.