This is a great initiative, and I would like to thank the authors and everyone contributing.
I would suggest adding a point under the "essential" category related to datasets used for evaluation. It is important that there is an appropriate justification of the dataset used for evaluation, and a description of the main features of the dataset that characterise the different problem instances in terms of "hardness". For example, if the size of the problem instances is an important feature that affects the performance of the optimisation approach, the dataset should be described in terms of this feature.
Under the "desirable category" I would suggest to include something around "efforts should be made to avoid any bias in the selection of the dataset"
Under the "invalid criticism" I would include "the paper uses only one dataset". Reviewers should provide valid criticism as to why that single dataset is not sufficient, or ask for more clarification from the authors. Papers should not be rejected because the authors use a single dataset.
This is a great initiative, and I would like to thank the authors and everyone contributing.
I would suggest adding a point under the "essential" category related to datasets used for evaluation. It is important that there is an appropriate justification of the dataset used for evaluation, and a description of the main features of the dataset that characterise the different problem instances in terms of "hardness". For example, if the size of the problem instances is an important feature that affects the performance of the optimisation approach, the dataset should be described in terms of this feature.
Under the "desirable category" I would suggest to include something around "efforts should be made to avoid any bias in the selection of the dataset"
Under the "invalid criticism" I would include "the paper uses only one dataset". Reviewers should provide valid criticism as to why that single dataset is not sufficient, or ask for more clarification from the authors. Papers should not be rejected because the authors use a single dataset.