search

openjournals / jose-reviews

Reviews for the Journal of Open Source Education (JOSE)
http://jose.theoj.org
Creative Commons Zero v1.0 Universal
33 stars 4 forks source link

[REVIEW]: Computational models of human social behavior and neuroscience: An open educational course and Jupyter Book to advance computational training #146

Closed whedon closed 2 years ago

whedon commented 3 years ago

Submitting author: @shawnrhoads (Shawn A. Rhoads) Repository: https://github.com/shawnrhoads/gu-psyc-347 Version: v1.1.0 Editor: @emckiernan Reviewer: @arokem, @djmannion Archive: 10.5281/zenodo.5907205

:warning: JOSE reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSE is currently operating in a "reduced service mode".

Status

status

Status badge code:

HTML: <a href="https://jose.theoj.org/papers/62fca31989cb19be1eb93970ac7554be"><img src="https://jose.theoj.org/papers/62fca31989cb19be1eb93970ac7554be/status.svg"></a>
Markdown: [![status](https://jose.theoj.org/papers/62fca31989cb19be1eb93970ac7554be/status.svg)](https://jose.theoj.org/papers/62fca31989cb19be1eb93970ac7554be)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@arokem & @djmannion, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/jose-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @emckiernan know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @arokem

Conflict of interest

Code of Conduct

General checks

Documentation

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

JOSE paper

Review checklist for @djmannion

Conflict of interest

Code of Conduct

General checks

Documentation

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

JOSE paper

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @arokem, @djmannion it looks like you're currently assigned to review this paper :tada:.

:warning: JOSE reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSE is currently operating in a "reduced service mode".

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/jose-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/jose-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago

Wordcount for paper.md is 1668

whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1111/tops.12404 is OK
- 10.1177/1745691620970585 is OK
- 10.7554/eLife.49547 is OK
- 10.1093/scan/nsaa040 is OK
- 10.1093/scan/nsaa089 is OK
- 10.1016/j.tics.2017.03.011 is OK
- 10.1016/j.neuron.2021.04.001 is OK
- 10.1016/j.tics.2021.03.018 is OK
- 10.1371/journal.pone.0244974 is OK
- 10.1038/s41592-019-0686-2 is OK
- 10.1038/s41586-020-2649-2 is OK
- 10.1109/MCSE.2007.55 is OK
- 10.5281/zenodo.3509134 is OK
- 10.1073/pnas.1603198113 is OK
- 10.3758/s13428-018-01193-y is OK
- 10.5281/zenodo.2561065 is OK
- 10.1016/j.neuron.2013.10.038 is OK
- 10.1002/9781119159193.ch17 is OK
- 10.1186/s13742-016-0121-x is OK
- 10.5281/zenodo.3909717 is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 3 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.21 s (285.1 files/s, 223638.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                           3            727           1282           5141
Jupyter Notebook                33              0          26552           4478
JavaScript                       2            655            503           3247
XML                              2              0              0           1651
Markdown                        11            234              0            820
TeX                              2             37              0            719
YAML                             5             14             26            158
HTML                             1              7              4             29
-------------------------------------------------------------------------------
SUM:                            59           1674          28367          16243
-------------------------------------------------------------------------------

Statistical information for the repository 'ca40fafda596e0be725e9ca8' was
gathered on 2021/10/19.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Shawn Rhoads                     7         26446          14891          100.00

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Shawn Rhoads              11555           43.7          0.0               15.27
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

emckiernan commented 3 years ago

Hi @arokem and @djmannion: Thank you for agreeing to review for JOSE! There are individual checklists for you both at the top of this issue. Please work your way through those, feel free to ask questions or post comments here, and also open issues in the submission repository as needed. I'm here to help, so please let me know if you need anything. Thanks!

djmannion commented 3 years ago

I confirm that I read and will adhere to the JOSE code of conduct.

Sorry, I can't seem to find the JOSE code of conduct - the link on the linked-to website (https://github.com/openjournals/jose/blob/master/CODE_OF_CONDUCT.md) gives a "Page not found" error.

arokem commented 3 years ago

To this question:

"Version: Does the release version given match the repository release (v1.0.0)?"

There are currently no releases on the GitHub repo, although there is a v1.0.0 tag that could easily be released through the GitHub interface, if that's needed.

shawnrhoads commented 3 years ago

To this question:

"Version: Does the release version given match the repository release (v1.0.0)?"

There are currently no releases on the GitHub repo, although there is a v1.0.0 tag that could easily be released through the GitHub interface, if that's needed.

@arokem, thanks for pointing this out! I just changed the repo settings to display the release version. Does it look right now? https://github.com/shawnrhoads/gu-psyc-347

emckiernan commented 3 years ago

Sorry, I can't seem to find the JOSE code of conduct - the link on the linked-to website (https://github.com/openjournals/jose/blob/master/CODE_OF_CONDUCT.md) gives a "Page not found" error.

@djmannion, thanks for flagging this! I get a 404 too, and I can't find the COC in the JOSE repo. I have a message in to the EIC, and will update you ASAP.

labarba commented 3 years ago

Where do you see the broken link to the CoC?

In the reviewer checklists, I see the text linking to https://jose.theoj.org/about#code_of_conduct which resolves properly for me.

emckiernan commented 3 years ago

Where do you see the broken link to the CoC?

In the reviewer checklists, I see the text linking to https://jose.theoj.org/about#code_of_conduct which resolves properly for me.

Hi @labarba: yes, that link does resolve correctly. But that page just describes that there is a CoC and then is supposed to link out to the CoC itself. The embedded link there, which points to https://github.com/openjournals/jose/blob/main/CODE_OF_CONDUCT.md, is what gives the 404.

labarba commented 3 years ago

Thanks for flagging this. We reorganized the Open Journals repositories this Summer, and a few things remain broken. Argh!

This repository is now just a placeholder and has no real content: https://github.com/openjournals/jose/

The JOSE content is now on a jose branch of the joss repo, and the COI is here: https://github.com/openjournals/joss/blob/jose/COI.md

We've made a note to fix this.

labarba commented 3 years ago

Argh! That is the COI, not COC — try again.

https://github.com/openjournals/joss/blob/jose/CODE_OF_CONDUCT.md

emckiernan commented 3 years ago

Thanks, @labarba! Got it now :)

@arokem and @djmannion: The link Lorena shared above will get you to the CoC. Please let me know if you have any other questions, thanks!

whedon commented 2 years ago

:wave: @arokem, please update us on how your review is going (this is an automated reminder).

whedon commented 2 years ago

:wave: @djmannion, please update us on how your review is going (this is an automated reminder).

arokem commented 2 years ago

Clearly, not going fast enough 😄

I am still working through this slowly. I hope to be able to work through everything in the next couple of weeks.

emckiernan commented 2 years ago

No worries, @arokem, thanks for letting us know!

djmannion commented 2 years ago

wave @djmannion, please update us on how your review is going (this is an automated reminder).

Working on it :smile:

emckiernan commented 2 years ago

Super, thanks, @djmannion!

djmannion commented 2 years ago

A general comment about the documentation (which I take to mean the repository documentation, rather than the jupyter book), which affects my response to those criteria, is that it seems to be for students in the course rather than for others who might like to adapt/use the material.

For example, it looks like the repository README.md is copied to become the index page of the jupyter book. That means that it has things like the course logistics that are relevant for students, but not for non-students considering using the material - and that any changes to the README.md to address such topics (e.g., how to build the book) would be not appropriate to show to students.

Might it be better to separate the material for those different audiences? For example, have the repository documentation (README.md) be about the need for the module from an educator perspective, how someone can adapt and build it for their own course, etc., and have all the student-targeted material in docs?

arokem commented 2 years ago

For this point:

Does it describe how it has been used in the classroom or other settings, and how someone might adopt it?

I think that you could add a bit in the paper about your experience . This is also a +1 to @djmannion's comment above that the document could be improved to make this more accessible to other instructors. You might consider adding a section about how to use the materials from the instructor point of view.

This point:

Does the paper tell the "story" of how the authors came to develop it, or what their expertise is?

Also does not currently clearly come through.

arokem commented 2 years ago

Other than these two points, I have read through the materials and the paper and added a couple of issues on the project repo, but have no other comments/suggestions. So, once these issues are all addressed, I would be happy to recommend this paper be accepted.

emckiernan commented 2 years ago

Thanks, @djmannion and @arokem, for your comments above. @shawnrhoads, any questions on the issues raised so far?

shawnrhoads commented 2 years ago

Hi all,

Thank you so much for reviewing these materials! These are great ideas, especially separating the documentation for different audiences (i.e., instructors versus students).

We will work on incorporating this helpful feedback and new issues submitted to the repository in the coming days and keep everyone posted on progress.

Shawn & Lin

shawnrhoads commented 2 years ago

Hi all,

Thank you again for this helpful feedback! We have updated the paper and repository to reflect the following changes. For ease of review, we have separated our updates with two pull requests: #184 updates the repository contents and #186 updates the paper contents. Reviewers can view all changes directly using the hyperlinks above. We also summarize our changes below.

Improving documentation for students, teachers, and contributors

With #184, we have now separated the documentation for students, teachers, and contributors. These changes should also resolve #180.

The repository documentation (README.md) now contains information for mainly for teachers/instructors and contributors. The Jupyter Book homepage (docs/index.md) now contains information targeted at students.

We also added a new resource page to the repository documentation (RESOURCES.md) to provide additional background for interested teachers/instructors (and students).

In addition, the CONTRIBUTING.md documentation now includes additional information on: (1) submitting issues, (2) requesting new content, (3) adding new content, (4) maintaining continuous integration, and (5) building the Jupyter Book locally.

To improve accessibility for students, teachers, and contributors, these updates also include the addition of three new issue template forms:

  1. Issue Reports
  2. Enhancement Requests
  3. Content-Relevant Questions

With #186, we updated the paper to reflect the above changes with a focus on how teachers may adopt the course modules:

Learning Objectives

All course materials were designed to be modular. For example, if students are knowledgeable in Python basics and Jupyter Notebook (Module 01), then teachers could begin with model fitting (Module 02). If students are knowledgeable in model fitting with maximum likelihood estimation (Module 02), teachers could start with reinforcement learning (Module 03). Any tutorial can be adopted, transformed, or built upon for other educational purposes (e.g., courses, single class sessions, workshops) under a Creative Commons Attribution-ShareAlike 4.0 International License. Information on how teachers can use these materials can be found in the README documentation.

Tutorials & Exercises

Students have the option to interact with each tutorial notebook using one of two ways. The first option entails installing Anaconda 3 on their local computers with a custom environment made for this course and then opening each notebook using Jupyter Notebook. Teachers can walk students through the installation steps using the instructions provided on the Getting Started page. However, this can be rather difficult to implement with students' different machines, operating systems, background, etc. As a fail-safe, students also have the option to use Google Colaboratory, which runs the code on Google's cloud servers for free. There is a button at the top of each tutorial page that opens each Jupyter Book page in Google Colaboratory.

Once students are able to run notebooks (locally or on the cloud), they can interact with all of the code in the tutorials. Teachers should guide students through all of the "tutorials", which, for example, will allow them to demonstrate to students how tweaking certain variables in real-time would change the outputs. The "exercises" at the end of each module were created for the students to apply what they have learned. They can work on these by themselves or in groups. Links to example solutions for all exercises are provided on each page. In addition, teachers should familiarize themselves with the recommended readings associated with each module and use them to introduce students to each topic, generate discussions and ideas among students, and provide relevant background for each tutorial. For example, the Wilson & Collins (2019) paper can be assigned for reading and discussion prior to beginning any notebook tutorials for Module 03.

For additional background on the content covered in each tutorial (technical or theoretical), teachers and students are encouraged to review the references and/or resources listed at the beginning of each tutorial or the non-exhaustive list of resources on the Resources page. Content-related questions from students, teachers, and contributors related to the course can also be submitted directly on the GitHub repository. However, we request that individuals make a good faith effort searching for their answer in the relevant readings or resources listed at the beginning of each tutorial before submitting questions.

Future Directions

We appreciate any feedback on errors, omitted credit, or new ideas for this evolving project. These can be communicated on GitHub by either reporting an issue or requesting an enhancement. All contributions will be credited and should follow the community guidelines outlined on the Contribution page.

Experience/Story and Expertise

With #186, we have now updated the following sections in the paper regarding students' expectations and experiences related to the learning objectives as well as the relevant expertise/story of the authors.

Learning Objectives

During the first iteration of the course in the spring 2021, students were 54.5% seniors (4th year), 36.4% juniors (3rd year), and 9.1% sophomores (2nd year). Broadly, students reported that they enrolled in the course to learn how to use Python and computational modeling to accelerate psychological research, to learn how to work with data and research methodology beyond a traditional research methods & statistics course, to learn skills relevant to a career in data science, and to gain a different perspective on topics from other classes. Most students reported spending 3-5 hours per week on readings and assignments (one student reported spending up to 10 hours and one student reported spending less than three hours). On a scale from 1 (strongly disagree) to 5 (strongly agree), students reported that the course was academically challenging and rigorous (M=4.75, SD=0.43) and that the course materials (M=4.00, SD=1.22), course-specific technologies (M=4.00, SD=1.22), and assignments (M=4.25, SD=0.83) helped them meet the stated learning objectives of the course.

Author Contributions

S. A. R. learned to implement the relevant course methods through conducting research and participating in various summer courses (e.g., 2018 MIND Computational Summer School, 2018 Neuroscience School of Advanced Studies in Social Decision-Making, 2019 Computational Psychiatry Summer Course). He also received relevant pedagogical training as a Teaching Assistant (TA) for Neuromatch Academy, as a TA for Dr. Maximilian Risenhuber's Computational Neuroscience course, and as a participant in the Apprenticeship in Teaching Program at Georgetown University. The course materials were created to make the hands-on elements from these experiences as well as smaller, related workshops led by S. A. R. more accessible in university settings and online.

Thanks again and please reach out if you have questions or additional comments.

Best, Shawn & Lin

shawnrhoads commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

emckiernan commented 2 years ago

@shawnrhoads: Great, thanks so much!

@djmannion, @arokem: Could you please look over the revisions, and let us know if you have any questions/comments/concerns, or whether you approve the revised paper? Thanks in advance!

djmannion commented 2 years ago

I'm happy to approve the revised paper. Thanks to the authors for creating this resource and writing it up!

A few other general comments below.

Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?

The documentation on how to build the course site is clear and I was able to build it successfully (though it is missing the Contributing document, which seems to require the build to go via github).

I noticed that there are quite a few locations where the build process is specific to the course. For example, the _config.yml has things like the course URL, a Google analytics ID, the repository URL, etc. - and a lot of Module 0 is specific to this particular course. Given that it is stated in the documentation that: "Instructors can adopt, remix, transform, and build upon these materials for their own course or for other educational purpose", I was going to suggest that the authors add a statement about this specificity and perhaps some instructions on what would need changing for other instructors to be able to convert it to their own course. However, I think it is clear enough from the context that what is presented is for a specific implementation of the material and that some digging would be needed to adapt it to use in a different course.

Miscellaneous

These are just some general thoughts I had when (lightly) reading through the course material. They are outside the scope of the review context, so it isn't necessary to respond to these or make any changes in response - I just thought I'd pass them along.

arokem commented 2 years ago

I have checked all the boxes in my list, and I'm happy to approve the revised paper.

emckiernan commented 2 years ago

Super, thanks @djmannion and @arokem for your reviews! We're very grateful.

@shawnrhoads: Looks like we're almost there! There's just a few minor comments above from @djmannion that I'd suggest looking over. As mentioned, it's not necessary to address all of these, but I think some would definitely be worth thinking about/doing. Just let me know when you've got the final version, and then I think we'll be good to go on final acceptance!

shawnrhoads commented 2 years ago

Sounds great, thank you all so much for these reviews!

I just merged a few updates with #187 addressing the following minor suggestions made by @djmannion:

A couple of typos in the Modules section of "Intro to Python Basics": "python import module_name", "python import module_name as new_module_name".

I wasn't sure why some notebooks show the cell outputs (e.g., "Working with Data", "Linear Modeling") while others do not (e.g., "Intro to Python Basics", "Nonlinear Modeling) - in the built site, that is.

You might not want to add in any extra syntax confusion, but the extraneous cell output that gets produced sometimes by matplotlib (e.g. <matplotlib.legend.Legend at 0x174e572d908>) can be suppressed by adding a semi-colon to the end of the command (e.g., plt.legend();) - in case you weren't aware of that.

The colour scale for the heatmap of the "Least-squares minimization" section makes it difficult to see the minimum (on my screen, anyway) - there are many cells that are black.

emckiernan commented 2 years ago

@shawnrhoads: Super, thanks, we are good for final acceptance! Congrats!

This is the first time I get to this stage as a JOSE editor, so please bear with me :) In the guidelines it says, "When a submission is accepted, we ask that the authors create an archive (on Zenodo, figshare, or other) and post the archive DOI in the REVIEW issue". Could you please do this and post the DOI here? I can then reach out to the EiC for final processing of your article. Thanks in advance!

labarba commented 2 years ago

First make a tagged release in the repository, and post the version number here.

Erin, you will then run @whedon set v1.2.3 as version here (with the correct version number).

shawnrhoads commented 2 years ago

Incredible! Thanks so much!

The latest tagged release is now v1.1.0

shawnrhoads commented 2 years ago

On Zenodo, there is an option that says the following:

Optional. Did your publisher already assign a DOI to your upload? If not, leave the field empty and we will register a new DOI for you. A DOI allows others to easily and unambiguously cite your upload. Please note that it is NOT possible to edit a Zenodo DOI once it has been registered by us, while it is always possible to edit a custom DOI.

Should I provide a DOI from JOSE or have one assigned by Zenodo?

labarba commented 2 years ago

Have Zenodo assign a DOI. This is for the archive of your materials. The JOSE DOI is for the citable paper only.

shawnrhoads commented 2 years ago

Sounds good! Here it is: https://doi.org/10.5281/zenodo.5907205

emckiernan commented 2 years ago

@whedon set v1.1.0 as version

whedon commented 2 years ago

OK. v1.1.0 is the version.

emckiernan commented 2 years ago

@whedon set https://doi.org/10.5281/zenodo.5907205 as archive

whedon commented 2 years ago

OK. 10.5281/zenodo.5907205 is the archive.

emckiernan commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

emckiernan commented 2 years ago

@whedon check references

whedon commented 2 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1111/tops.12404 is OK
- 10.1177/1745691620970585 is OK
- 10.7554/eLife.49547 is OK
- 10.1093/scan/nsaa040 is OK
- 10.1093/scan/nsaa089 is OK
- 10.1016/j.tics.2017.03.011 is OK
- 10.1016/j.neuron.2021.04.001 is OK
- 10.1016/j.tics.2021.03.018 is OK
- 10.1371/journal.pone.0244974 is OK
- 10.1038/s41592-019-0686-2 is OK
- 10.1038/s41586-020-2649-2 is OK
- 10.1109/MCSE.2007.55 is OK
- 10.5281/zenodo.3509134 is OK
- 10.1073/pnas.1603198113 is OK
- 10.3758/s13428-018-01193-y is OK
- 10.5281/zenodo.2561065 is OK
- 10.1016/j.neuron.2013.10.038 is OK
- 10.1002/9781119159193.ch17 is OK
- 10.1186/s13742-016-0121-x is OK
- 10.5281/zenodo.3909717 is OK

MISSING DOIs

- None

INVALID DOIs

- None
emckiernan commented 2 years ago

@labarba Hi Lorena, thanks for the assist :) I think everything is good to go here! I've set the version and archive DOI, checked the proofs and refs, etc. Is there anything else for me to do?

labarba commented 2 years ago

👋 If everything is ready to go, you as handling editor run @whedon recommend-accept — and I think that would automatically do the check on references and generate the pdf ... try it! 😁