[PRE REVIEW]: What and How of Machine Learning Transparency: Building Bespoke Explainability Tools with Interoperable Algorithmic Components

[whedon]

Submitting author: @So-Cool (Kacper Sokol) Repository: https://github.com/fat-forensics/Surrogates-Tutorial Version: 2020-ecml-pkdd Editor: @ttimbers Reviewers: @kvarada, @arokem Managing EiC: Jordan Gorzalski

:warning: JOSE reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSE is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://jose.theoj.org/papers/d58625bd4c600da866522c879986b18f"><img src="https://jose.theoj.org/papers/d58625bd4c600da866522c879986b18f/status.svg"></a>
Markdown: [![status](https://jose.theoj.org/papers/d58625bd4c600da866522c879986b18f/status.svg)](https://jose.theoj.org/papers/d58625bd4c600da866522c879986b18f)

Author instructions

Thanks for submitting your paper to JOSE @So-Cool. Currently, there isn't an JOSE editor assigned to your paper.

@So-Cool if you have any suggestions for potential reviewers then please mention them here in this thread (without tagging them with an @). In addition, this list of people have already agreed to review for JOSE and may be suitable for this submission (please start at the bottom of the list).

Editor instructions

The JOSE submission bot @whedon is here to help you find and assign reviewers and start the main review. To find out what @whedon can do for you type:

@whedon commands

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks.

:warning: JOSE reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSE is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Wordcount for paper.md is 1147

[whedon]

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.28 s (49.5 files/s, 61536.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Jupyter Notebook                 5              0          13253           2550
Markdown                         5             98              0            664
Python                           1            145            344            242
TeX                              1              7              0             67
YAML                             2              0              0             40
-------------------------------------------------------------------------------
SUM:                            14            250          13597           3563
-------------------------------------------------------------------------------

Statistical information for the repository '3da1e3c7139522bf7ce1245b' was
gathered on 2022/04/13.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Kacper Sokol                     1           731              0          100.00

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Kacper Sokol                731          100.0          0.0               10.81

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- 10.18653/v1/n16-3020 may be a valid DOI for title: “Why Should I Trust You?”: Explaining the Predictions of Any Classifier
- 10.21105/joss.01904 may be a valid DOI for title: FAT Forensics: A Python Toolbox for Implementing and Deploying Fairness, Accountability and Transparency Algorithms in Predictive Systems

INVALID DOIs

- None

[labarba]

@whedon invite @ttimbers as editor

[labarba]

@whedon invite @ttimbers as editor

[whedon]

@ttimbers has been invited to edit this submission.

[ttimbers]

Happy to act as editor here @labarba !

[labarba]

Great! Here's a neat trick: you can run @whedon assign me as editor

[ttimbers]

@whedon assign me as editor

[whedon]

OK, the editor is @ttimbers

[So-Cool]

@whedon check references

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1145/2939672.2939778 is OK
- 10.48550/arXiv.1710.09412 is OK
- 10.48550/arXiv.1910.13016 is OK
- 10.48550/arXiv.1909.05167 is OK
- 10.21105/joss.01904 is OK
- 10.48550/arXiv.2005.01427 is OK
- 10.48550/arXiv.2008.07007 is OK
- 10.48550/arXiv.2112.14466 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[So-Cool]

@whedon generate pdf

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[ttimbers]

@So-Cool - thank-you for your submission. I have read your manuscript and considered it with the scope of JOSE. Typically, short learning modules are not within scope of the journal. However, short learning modules can be considered if the authors show substantial scholarly effort in other ways. One example of this would be if the tutorial has been given many times and iteratively improved. Another example would be that the module has been subject to methodical evaluation. These scholarly efforts should be reported in the manuscript.

Reading your manuscript, I did note you mentioned this module has been taught more than once, and perhaps by others? If this were further elaborated on, as well as what has been learned from teaching this several times, that could be considered on of the scholarly efforts we described above.

[So-Cool]

@whedon check references

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1145/2939672.2939778 is OK
- 10.48550/arXiv.1710.09412 is OK
- 10.48550/arXiv.1910.13016 is OK
- 10.48550/arXiv.1909.05167 is OK
- 10.21105/joss.01904 is OK
- 10.48550/arXiv.2005.01427 is OK
- 10.48550/arXiv.2008.07007 is OK
- 10.48550/arXiv.2112.14466 is OK
- 10.1023/A:1010933404324 is OK
- 10.1080/10618600.2014.907095 is OK
- 10.1214/aos/1013203451 is OK
- 10.48550/arXiv.2107.06639 is OK
- 10.1038/s42256-019-0048-x is OK

MISSING DOIs

- None

INVALID DOIs

- None

[So-Cool]

@whedon generate pdf

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[So-Cool]

Thank you for your review, @ttimbers. We have revised the manuscript based on your comments. In particular, the last paragraph of the "Modular Surrogate Explainers" section has been updated and extended by a new paragraph explaining the evolution of the training materials and our teaching experience.

[ttimbers]

@joelostblom – hi! 👋 you teach related topics and I wonder if you would consider contributing a review on this manuscript submitted to JOSE. We are in need of some help with this submission:

Title: What and How of Machine Learning Transparency: Building Bespoke Explainability Tools with Interoperable Algorithmic Components View article proof on GitHub

Thanks for considering it!

[ttimbers]

@kbodwin - hi! 👋 you are listed on the JOSE reviewer volunteer list, and that you have domain expertise in machine learning. Thus, I wonder if you would consider contributing a review on this manuscript submitted to JOSE. We are in need of some help with this submission:

Title: What and How of Machine Learning Transparency: Building Bespoke Explainability Tools with Interoperable Algorithmic Components View article proof on GitHub

Thanks for considering it!

[ttimbers]

A gentle ping to @joelostblom and @kbodwin to answer whether or not they could serve as reviewers for this?

[joelostblom]

Thanks for the ping and sorry for my delayed reply! After reading through the manuscript, I must unfortunately decline to review as I don't think I have enough expertise on this specific topic to provide valuable feedback and advice on the submission.

[ttimbers]

Thanks @joelostblom for the reply and completely understandable.

[ttimbers]

Thanks @kvarada for agreeing (over Slack) to serve as a reviewer for this manuscript 🎉

[ttimbers]

@whedon add @kvarada as reviewer

[whedon]

OK, @kvarada is now a reviewer

[ttimbers]

@kbodwin - I am going to assume you don't have the time to review this and search for another reviewer, please correct me in case I am wrongly assuming this.

[ttimbers]

@arokem - hi! 👋 you are listed on the JOSE reviewer volunteer list, and that you have domain expertise in machine learning. Thus, I wonder if you would consider contributing a review on this manuscript submitted to JOSE. We are in need of some help with this submission:

Title: What and How of Machine Learning Transparency: Building Bespoke Explainability Tools with Interoperable Algorithmic Components View article proof on GitHub

Thanks for considering it!

[kbodwin]

@kbodwin - I am going to assume you don't have the time to review this and search for another reviewer, please correct me in case I am wrongly assuming this.

Sorry to miss this!

I don't think the material is quite in my wheelhouse, but if you don't find someone by the end of June, I'll be done with a mountain of deadlines and willing to give it a go.

[arokem]

Hi @ttimbers : I am currently bogged down with other reviews and leaving for a conference at the end of the week. If you don't have the needed reviewers at the end of the month, I'd be happy to jump on it then. Or, if that's an acceptable delay, you can assign me, with the understanding that I will not get to it before the end of June.

[ttimbers]

Thank-you @kbodwin and @arokem. @kbodwin - no worries, if you feel this is outside of your wheelhouse, then we will find someone else. @arokem - thanks for offering if we cannot find anyone else before the end of the month. I will let you know in the event that happens.

[ttimbers]

@mnarayan - hi! 👋 you are listed on the JOSE reviewer volunteer list, and that you have domain expertise in machine learning. Thus, I wonder if you would consider contributing a review on this manuscript submitted to JOSE. We are in need of some help with this submission:

Title: What and How of Machine Learning Transparency: Building Bespoke Explainability Tools with Interoperable Algorithmic Components View article proof on GitHub

Thanks for considering it!

[mnarayan]

I'm not available for the next 2 weeks. If the issue is still open after that, I can review.

On Mon, Jun 13, 2022 at 9:09 AM Tiffany A. Timbers @.***> wrote:

@mnarayan https://github.com/mnarayan - hi! 👋 you are listed on the JOSE reviewer volunteer list, and that you have domain expertise in machine learning. Thus, I wonder if you would consider contributing a review on this manuscript submitted to JOSE. We are in need of some help with this submission:

Title: What and How of Machine Learning Transparency: Building Bespoke Explainability Tools with Interoperable Algorithmic Components View article proof on GitHub https://github.com/openjournals/jose-papers/blob/jose.00171/jose.00171/10.21105.jose.00171.pdf

Thanks for considering it!

— Reply to this email directly, view it on GitHub https://github.com/openjournals/jose-reviews/issues/171#issuecomment-1154110963, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFC754DPVVC3LF4H3EQDNTVO5MLHANCNFSM5TKV2YVA . You are receiving this because you were mentioned.Message ID: @.***>

[ttimbers]

@arokem - I am going to have to take you up on your kind offer to act as a reviewer for this paper. It seems to be a tricky time of year to find folks to review, so we are very happy to work with you on a review timeline that fits with your availability. I will open the review now with the understanding that you cannot start this until after June. Many thanks!

[ttimbers]

@whedon add @arokem as reviewer

[whedon]

OK, @arokem is now a reviewer

[ttimbers]

@mnarayan - thank-you for the offer, however we had another reviewer provide the same offer a bit earlier than you. We will of course keep you in mind for the next machine learning paper we need reviewed at JOSE!

[ttimbers]

@whedon start review

[whedon]

OK, I've started the review over in https://github.com/openjournals/jose-reviews/issues/175.