Response to comments from @hbaniecki

[enricgrau]

Apart from minor issues with documentation and examples, I have the following major concerns about this contribution:

1. Substantial scholarly effort is (currently) not evident. It is a relatively young software (4k downloads), there is no user base (no issues, no GitHub stars, no blogs or promotion of the software?), and no clear applications (e.g. no citations). With similar software existing already, {pudu} is not likely to be cited unless the software/paper clearly highlights its original purpose, e.g. an original subset of methods implemented, accessible visualisation, better API, unique audience.

2. Statement of need is unclear. I have a hard time understanding the motivation for developing this software. In its current state, the paper's title and description seem too generic. Should "sensitivity analysis" be highlighted in the title?

   • A clear use-case for this software would help, e.g. the paper mentions the RELIEF method but does not elaborate on applications/usefulness of the algorithm to show that the {pudu} package is indeed needed in situation X or for user Y.
   • The terminology of "Importance/speed/synergy/reactivation" is unclear and could be better described in the paper.
   • Adding a figure to the paper could help illustrate the statement of need.
3. State of the field is missing. There is no description of how this software compares to other similar packages. Consider the attached non-exhaustive list of software for explainability and feature selection. Relating {pudu} to other software could partially alleviate concerns mentioned in 1. & 2., i.e. show the added value of {pudu} over the already available solutions, motivate the need for such new package, highlight the target audience etc.

I am open to discussion and hope the software paper can be improved to clearly state the motivation and effort.

References (non-exhaustive list)

• Molnar et al. iml: An R package for Interpretable Machine Learning. JOSS 2018
• Alber et al. iNNvestigate Neural Networks! JMLR 2019
• Kokhlikyan et al. Captum: A unified and generic model interpretability library for PyTorch. arXiv:2009.07896 2020
• Arya et al. AI Explainability 360: An Extensible Toolkit for Understanding Data and Machine Learning Models. JMLR 2020
• (ours) dalex: Responsible Machine Learning with Interactive Explainability and Fairness in Python. JMLR 2021
• Klaise et al. Alibi Explain: Algorithms for Explaining Machine Learning Models. JMLR 2021
• Li et al. InterpretDL: Explaining Deep Models in PaddlePaddle. JMLR 2022
• Zhu et al. abess: A Fast Best-Subset Selection Library in Python and R. JMLR 2022

Quality of writing

• There is some vague language, e.g. "help make sense of machine learning results", "Easy plotting of the results"
• Acronyms "ML/XAI" are defined but later unused
• Typos in references: "Bhatt et al. (2020)](Belle & Papantonis, 2021)" "Bau2018"

[enricgrau]

Regarding the review of @hbaniecki in https://github.com/openjournals/joss-reviews/issues/5873#issuecomment-1761368406

Thank you for your comprehensive and detailed comments. We appreciate the time you took to review our work and constructively provide your concerns. We have thoroughly revised the manuscript following your suggestions and concerns, and we believe that now shows to be much improved. The article does appear to be quite different from the original one. We intentionally removed and changed sections and several phrases and paragraphs to regain focus on the original inspiration for the library. We hope that now the article shows the motivation, opportunity, and application clearly.

Here we respond point by point:

1. We acknowledge that pudu is a relatively young software with limited visibility in its current state. We hope that the changes that we have made in addition to a potential publication, can help the library to gain traction with the target audience. The library comes from a very specific concern regarding research with spectroscopic techniques, but we tried to present it more generically and broadly to expand the target user. However, it is clear now to us, due to your observations, that we didn’t make the mark on building a solid case for such a broad scope. With this in mind, we have decided to narrow the reach of the library to its original motivation regarding spectroscopic data. This is now reflected in the manuscript, and we hope to be clearer on why we believe that pudu can contribute to the field.
2. We recognize that the motivation behind the development of pudu needs to be more explicitly communicated. As mentioned, the initial motivation is very specific, and we tried to make its motivation more generalized to hopefully capture more interest. To this end, we restructured this section to highlight the specific gaps and challenges with existing tools. We include relevant survey articles that observe problems with explainability methods and libraries and also better explain why pudu fits the gap between spectroscopy-related research and explainable methods.

Regarding the relationship with RELIEF: Our intention with pudu was to be a wrapper for the basic principles of RELIEF and to offer extended functionality that addresses the challenges in our field of research. After much deliberation, we decided to remove the reference to RELIEF since it causes more confusion than clarity, and the reference offers no value regardless.

We agree that the terminology of Importance/speed/synergy/reactivation are vague and unclear in the paper. The original idea was for the reader to complement the paper with the documentation where these are better explained. We realize now that we failed to achieve this. Thus, we have improved these explanations in the paper to better define these for the reader in a more clear and accessible way.

Following your advice, to further aid understanding we added a figure to visually represent the primary motivations and benefits of using pudu, giving readers a clear snapshot of its utility.

3. Considering that we have restructured the manuscript to focus on spectroscopy, we believe that now it better shows the context and need on why pudu is beneficial for spectroscopy, using relevant references to support this. We hope this will offer readers a clear understanding of pudu's added value.

Quality of writing:

• We worked on identifying this and other vague statements and transformed them into concrete ones. We are happy to keep improving such vague phrases if needed.
• We have improved the use of acronyms and only making definitions if needed.
• Typos with references were fixed. The problem was spacing between brackets.

[enricgrau]

@hbaniecki Please find the new manuscript in the library's repository here in the meantime. We are not able to re-render with the editorialbot from JOSS for some reason, but all the changes that will shown once this is solved are reflected in this preview generated from the new paper.md.

[hbaniecki]

Hi @enricgrau, thank you for the exhaustive answer. I re-read the revised version of the manuscript, which is much more precise now. I better understand the motivation behind this software and recognize the effort to introduce explainability into spectroscopy-related research.

In my opinion, it would still be useful to reference any other related explainability toolboxes (shap/lime/gradcam could be considered only an implementation of a particular method) so as to let the reader know about the benefits of using pudu in spectroscopic problems.

Figure 1 is mentioned only briefly in the text. Adding more context regarding the predictive task/data is needed for it to be understandable by a broader audience.

other minor:

• Should the {spectrapepper} software (JOSS 2021) be discussed as related work?
• L70: name pudu can be bolded to be consistent
• L71: there is a typo 2x comma
• title: probably shouldn't end with a dot "." as it is then wrongly propagated to BibTeX, GScholar etc.

I believe the above remaining comments are minor/subjective enough to recommend the paper's acceptance.