[REVIEW]: fseval: A Benchmarking Framework for Feature Selection and Feature Ranking Algorithms

[editorialbot]

Submitting author: !--author-handle-->@dunnkers<!--end-author-handle-- (Jeroen Gerard Sebastiaan Overschie) Repository: https://github.com/dunnkers/fseval Branch with paper.md (empty if default branch): paper Version: 3.1.0 Editor: !--editor-->@diehlpk<!--end-editor-- Reviewers: @mcasl, @estefaniatalavera Archive: 10.5281/zenodo.7343417

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/d850aeb67247318aeef735d5eca95c1c"><img src="https://joss.theoj.org/papers/d850aeb67247318aeef735d5eca95c1c/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/d850aeb67247318aeef735d5eca95c1c/status.svg)](https://joss.theoj.org/papers/d850aeb67247318aeef735d5eca95c1c)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@mcasl & @estefaniatalavera, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @diehlpk know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @mcasl

📝 Checklist for @estefaniatalavera

[mcasl]

I still fail to see any progress in some of the topics that I previously mentioned:

• A statement of need: There's no statement of need in the documentation, only in the paper.
• Installation instructions: There is no list of dependencies
• Functionality documentation: No guidelines have been added yet.

The other issues have been appropriately addressed and the checklist reflects the changes.

[dunnkers]

I still fail to see any progress in some of the topics that I previously mentioned:

• A statement of need: There's no statement of need in the documentation, only in the paper.
• Installation instructions: There is no list of dependencies
• Functionality documentation: No guidelines have been added yet.

The other issues have been appropriately addressed and the checklist reflects the changes.

Hi @mcasl! Thanks for the feedback.

See below a response to each of your points 🙂:

Statement of need

• A statement of need: There's no statement of need in the documentation, only in the paper.

✓ Added in:

• https://github.com/dunnkers/fseval/pull/92 (see website)

Dependency list

• Installation instructions: There is no list of dependencies

The checklist says the following:

Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.

Saying; the dependencies should ideally be handled by an automated package management solution. This is the case for fseval. We are using a setup.py file which lists all the required dependencies, which are then automatically installed when the user installs the package.

Furthermore, there exists a requirements.txt file, listing all dependencies in such a way that they can automatically consumed and installed by Python's package manager, pip. This is referred to in the "Installation from source" installation guide.

Additionally, all dependencies required for developing are listed in another requirements.txt file. This is referred to in the Developer Guide. If the user prefers to use our containerized solution, we also packaged up a Docker image for them which can be used by spinning up the project's Devcontainer.

Community guidelines

• Functionality documentation: No guidelines have been added yet.

Looking at your checklist I assume the guidelines mentioned refer to the community guidelines:

• ❌ Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

The community guidelines were added in this Pull Request:

• [x] Added Developer guidelines for contributing (https://github.com/dunnkers/fseval/pull/89)

That is, guidelines for the community were added both to the documentation website and in the repo. I could see, however, that the website was not yet re-deployed after the changes 😬. I made a re-deployment just now so the Developer Guide is now online.

[mcasl]

Thank you @dunnkers for the updates. I'm satisfied with the changes. Now the proposal has met all the requirements specified in the checklist. Good job!

[mcasl]

@diehlpk All items in my checklist are ok

[diehlpk]

@editorialbot check references

[diehlpk]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.eswa.2015.07.007 is OK
- 10.1109/ICCV.2015.478 is OK
- 10.1109/TPAMI.2005.159 is OK
- 10.1007/10968987_3 is OK
- 10.1145/2641190.2641198 is OK

MISSING DOIs

- 10.3390/info10030109 may be a valid DOI for title: An experimental comparison of feature-selection and classification methods for microarray datasets
- 10.1109/access.2019.2928975 may be a valid DOI for title: Comparison of feature selection methods and machine learning classifiers for radiomics analysis in glioma grading
- 10.3390/info10030109 may be a valid DOI for title: An experimental comparison of feature-selection and classification methods for microarray datasets
- 10.1109/access.2019.2928975 may be a valid DOI for title: Comparison of feature selection methods and machine learning classifiers for radiomics analysis in glioma grading
- 10.1007/s12021-015-9292-3 may be a valid DOI for title: Comparison of feature selection techniques in machine learning for anatomical brain MRI in dementia
- 10.1016/j.eswa.2017.03.042 may be a valid DOI for title: Multi-class sentiment classification: The experimental comparisons of feature selection and machine learning algorithms
- 10.1016/j.asoc.2019.105836 may be a valid DOI for title: Evaluation of feature selection methods for text classification with small datasets using multiple criteria decision-making methods

INVALID DOIs

- https://doi.org/10.1016/j.softx.2017.07.005 is INVALID because of 'https://doi.org/' prefix

[diehlpk]

@dunnkers please fix the missing and invalid DOIs.

[dunnkers]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[dunnkers]

@editorialbot check references

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.eswa.2015.07.007 is OK
- 10.1109/ICCV.2015.478 is OK
- 10.1109/TPAMI.2005.159 is OK
- 10.1007/10968987_3 is OK
- 10.1145/2641190.2641198 is OK
- 10.3390/info10030109 is OK
- 10.1109/ACCESS.2019.2928975 is OK
- 10.1007/s12021-015-9292-3 is OK

MISSING DOIs

- None

INVALID DOIs

- https://doi.org/10.1016/j.softx.2017.07.005 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1016/j.eswa.2017.03.042 is INVALID because of 'https://doi.org/' prefix
- https://doi.org/10.1016/j.asoc.2019.105836 is INVALID because of 'https://doi.org/' prefix
- doi:10.2478/cait-2019-0001 is INVALID (failed connection)

[dunnkers]

@editorialbot check references

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.eswa.2015.07.007 is OK
- 10.1109/ICCV.2015.478 is OK
- 10.1109/TPAMI.2005.159 is OK
- 10.1007/10968987_3 is OK
- 10.1145/2641190.2641198 is OK
- 10.1016/j.softx.2017.07.005 is OK
- 10.3390/info10030109 is OK
- 10.1109/ACCESS.2019.2928975 is OK
- 10.1007/s12021-015-9292-3 is OK
- 10.1016/j.eswa.2017.03.042 is OK
- 10.1016/j.asoc.2019.105836 is OK
- 10.2478/cait-2019-0001 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[dunnkers]

References now ok 👍🏻 @diehlpk

[diehlpk]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[diehlpk]

@dunnkers Please do the following to proceed with the paper acceptance:

• [x] Generate a new release with all the changes you made during the review. Please post the version number here.
• [x] Upload this release do Zenodo or FigShare and provide me with the DPI. Note that the authors and title of the Zenodo archive have to match with the paper.

[dunnkers]

Generate a new release with all the changes you made during the review. Please post the version number here.

Released version 3.1.0 ✓ https://github.com/dunnkers/fseval/releases/tag/3.1.0

Upload this release do Zenodo or FigShare and provide me with the DPI. Note that the authors and title of the Zenodo archive have to match with the paper.

DOI: 10.5281/zenodo.7343417

Is that okay? Not sure if the title matches; still looking for how to change or whether to issue a new DOI for that

[dunnkers]

I searched around in the Zenodo interface and managed to change the title ✅. Should be good now. If not let me know ✓.

[diehlpk]

@editorialbot set 3.1.0 as version

[editorialbot]

Done! version is now 3.1.0

[diehlpk]

@editorialbot set 10.5281/zenodo.7343417 as archive

[editorialbot]

Done! Archive is now 10.5281/zenodo.7343417

[diehlpk]

@dunnkers Looks good to me. I can recommend the paper for acceptance.

[diehlpk]

@editorialbot recommend-accept

[editorialbot]

Attempting dry run of processing paper acceptance...

[editorialbot]

:wave: @openjournals/csism-eics, this paper is ready to be accepted and published.

Check final proof :point_right::page_facing_up: Download article

If the paper PDF and the deposit XML files look good in https://github.com/openjournals/joss-papers/pull/3745, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.eswa.2015.07.007 is OK
- 10.1109/ICCV.2015.478 is OK
- 10.1109/TPAMI.2005.159 is OK
- 10.1007/10968987_3 is OK
- 10.1145/2641190.2641198 is OK
- 10.1016/j.softx.2017.07.005 is OK
- 10.3390/info10030109 is OK
- 10.1109/ACCESS.2019.2928975 is OK
- 10.1007/s12021-015-9292-3 is OK
- 10.1016/j.eswa.2017.03.042 is OK
- 10.1016/j.asoc.2019.105836 is OK
- 10.2478/cait-2019-0001 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[danielskatz]

@dunnkers - I'm the track editor for this submission, and will handle the rest of the process. I have some minor changes to make in your bib file, in https://github.com/dunnkers/fseval/pull/94 - undoing some case protection that doesn't match the journal style, and adding a missing ) This otherwise looks good, and if you can merge this PR, we can complete the acceptance and publication process.

[dunnkers]

@dunnkers - I'm the track editor for this submission, and will handle the rest of the process. I have some minor changes to make in your bib file, in https://github.com/dunnkers/fseval/pull/94 - undoing some case protection that doesn't match the journal style, and adding a missing ) This otherwise looks good, and if you can merge this PR, we can complete the acceptance and publication process.

Hi @danielskatz ! Great we are almost there 😃.

Thanks for your PR, it's merged ✅.

[dunnkers]

@dunnkers Looks good to me. I can recommend the paper for acceptance.

That's awesome, thanks @diehlpk !

[danielskatz]

@editorialbot recommend-accept

[editorialbot]

Attempting dry run of processing paper acceptance...

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.eswa.2015.07.007 is OK
- 10.1109/ICCV.2015.478 is OK
- 10.1109/TPAMI.2005.159 is OK
- 10.1007/10968987_3 is OK
- 10.1145/2641190.2641198 is OK
- 10.1016/j.softx.2017.07.005 is OK
- 10.3390/info10030109 is OK
- 10.1109/ACCESS.2019.2928975 is OK
- 10.1007/s12021-015-9292-3 is OK
- 10.1016/j.eswa.2017.03.042 is OK
- 10.1016/j.asoc.2019.105836 is OK
- 10.2478/cait-2019-0001 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

:wave: @openjournals/csism-eics, this paper is ready to be accepted and published.

Check final proof :point_right::page_facing_up: Download article

If the paper PDF and the deposit XML files look good in https://github.com/openjournals/joss-papers/pull/3748, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

[danielskatz]

@editorialbot accept

[editorialbot]

Doing it live! Attempting automated processing of paper acceptance...

[editorialbot]

🐦🐦🐦 👉 Tweet for this paper 👈 🐦🐦🐦

[editorialbot]

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

0. Check final PDF and Crossref metadata that was deposited :point_right: https://github.com/openjournals/joss-papers/pull/3749
1. Wait a couple of minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.04611
2. If everything looks good, then close this review issue.
3. Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

[danielskatz]

This DOI isn't resolving for me yet - I will leave this issue open for now, and come back and check again in a little while. But other than that, the process is complete

[dunnkers]

This DOI isn't resolving for me yet - I will leave this issue open for now, and come back and check again in a little while. But other than that, the process is complete

👍. I also cannot get hold of the DOI link.

By the way: is it possible to have my name displayed as Jeroen G. S. Overschie instead of Jeroen G. s. Overschie (capitalised second names) on the JOSS website?

That would be awesome. Thanks

[danielskatz]

@arfon - can you check on the DOI issue, as well as the author's name as discussed above?

[arfon]

@xuanxu @tarleb – the name formatting looks good in the PDF here: https://github.com/openjournals/joss-papers/blob/f0d0a5e8b17e200bc09a0ecac00c59149ccba189/joss.04611/10.21105.joss.04611.pdf – is there a preferred way we can override the name in the front matter to accomplish this?

[arfon]

@editorialbot re-accept

[editorialbot]

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@editorialbot commands

[arfon]

@editorialbot reaccept

[editorialbot]

Rebuilding paper!

[editorialbot]

🌈 Paper updated!

New PDF and metadata files :point_right: https://github.com/openjournals/joss-papers/pull/3752