[REVIEW]: An Open-Source Active Learning Curriculum for Data Science

whedon commented 3 years ago

Submitting author: @zdelrosario (Zachary del Rosario) Repository: https://github.com/zdelrosario/data-science-curriculum Version: v0.1 Editor: @ttimbers Reviewer: @firasm, @pschloss Archive: 10.5281/zenodo.6314141

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

Status badge code:

HTML: <a href="https://jose.theoj.org/papers/f610a5debae8f7f9f08edd1e2abda774"><img src="https://jose.theoj.org/papers/f610a5debae8f7f9f08edd1e2abda774/status.svg"></a>
Markdown: [![status](https://jose.theoj.org/papers/f610a5debae8f7f9f08edd1e2abda774/status.svg)](https://jose.theoj.org/papers/f610a5debae8f7f9f08edd1e2abda774)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@firasm & @pschloss, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/jose-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @ttimbers know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @firasm

Conflict of interest

[x] As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSE code of conduct.

General checks

[x] Repository: Is the source for this learning module available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of a standard license? (OSI-approved for code, Creative Commons for content)
[x] Version: Does the release version given match the repository release (v0.1)?
[x] Authorship: Has the submitting author (@zdelrosario) made visible contributions to the module? Does the full list of authors seem appropriate and complete?

Documentation

[x] A statement of need: Do the authors clearly state the need for this module and who the target audience is?
[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the module 2) Report issues or problems with the module 3) Seek support

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

[ ] Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
[x] Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
[x] Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
[x] Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
[x] Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

JOSE paper

[x] Authors: Does the paper.md file include a list of authors with their affiliations?
[x] A statement of need: Does the paper clearly state the need for this module and who the target audience is?
[ ] Description: Does the paper describe the learning materials and sequence?
[ ] Does it describe how it has been used in the classroom or other settings, and how someone might adopt it?
[x] Could someone else teach with this module, given the right expertise?
[ ] Does the paper tell the "story" of how the authors came to develop it, or what their expertise is?
[ ] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @pschloss

Conflict of interest

[x] As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSE code of conduct.

General checks

[x] Repository: Is the source for this learning module available at the repository url?
[ ] License: Does the repository contain a plain-text LICENSE file with the contents of a standard license? (OSI-approved for code, Creative Commons for content)
[x] Version: Does the release version given match the repository release (v0.1)?
[x] Authorship: Has the submitting author (@zdelrosario) made visible contributions to the module? Does the full list of authors seem appropriate and complete?

Documentation

[ ] A statement of need: Do the authors clearly state the need for this module and who the target audience is?
[ ] Installation instructions: Is there a clearly stated list of dependencies?
[ ] Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the module 2) Report issues or problems with the module 3) Seek support

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

[x] Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
[x] Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
[x] Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
[x] Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
[x] Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

JOSE paper

[x] Authors: Does the paper.md file include a list of authors with their affiliations?
[ ] A statement of need: Does the paper clearly state the need for this module and who the target audience is?
[x] Description: Does the paper describe the learning materials and sequence?
[x] Does it describe how it has been used in the classroom or other settings, and how someone might adopt it?
[x] Could someone else teach with this module, given the right expertise?
[x] Does the paper tell the "story" of how the authors came to develop it, or what their expertise is?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @firasm & @pschloss it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/jose-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

Set yourself as 'Not watching' https://github.com/openjournals/jose-reviews:

watching

You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

whedon commented 3 years ago

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.12 s (802.6 files/s, 146479.8 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
Rmd                              75           3374           7389           4602
Markdown                         13            260              0            856
HTML                              1              9              1            642
TeX                               1             18              0            140
R                                 1             12              3             67
make                              2             14              3             61
Python                            1              5              4             22
YAML                              1              5              0             14
Bourne Again Shell                1              5              1             13
--------------------------------------------------------------------------------
SUM:                             96           3702           7401           6417
--------------------------------------------------------------------------------

Statistical information for the repository 'fe60f61c653b64ac0e364fa4' was
gathered on 2021/05/26.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Zach                             3            38              7          100.00

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
zdelrosario                  31          100.0          0.0               16.13

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 3 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1073/pnas.1319030111 is OK
- 10.21105/joss.01686 is OK

MISSING DOIs

- 10.18260/1-2--22585 may be a valid DOI for title: The flipped classroom: A survey of the research
- 10.3917/mediu may be a valid DOI for title: Medium
- 10.18637/jss.v064.i04 may be a valid DOI for title: fitdistrplus: An R Package for Fitting Distributions

INVALID DOIs

- None

zdelrosario commented 3 years ago

@whedon generate pdf

(Update DOI for "The Flipped Classroom" and fitdistrplus....)

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 3 years ago

:wave: @firasm, please update us on how your review is going (this is an automated reminder).

ttimbers commented 3 years ago

:wave: @pschloss, please update us on how your review is going (this is an automated reminder).

pschloss commented 3 years ago

I completed my first review and posted issues on the target repository. I like the overall format and goals of the materials and think they represent a unique contribution. The issues I filed concerned...

I look forward to reviewing the revised materials!

firasm commented 3 years ago

Hello @zdelrosario!

Thanks for creating such a great resource! In some cases I've added my comments to existing issues so the reviewer feedback is consolidated in one place:

Links to a few issues:

Let me know when there's something new you'd like me to look at, or if you need help implementing any of my suggestions. If any of the suggestions seem particularly onerous or not rewarding, let's discuss!

zdelrosario commented 3 years ago

Hi @pschloss and @firasm ! Many thanks for the detailed feedback; I am looking forward to addressing these comments, as I believe they will substantially improve the work. Note that I have a few clarifying questions on some of your comments; I'll direct those to the relevant issues.

ttimbers commented 3 years ago

hi @zdelrosario 👋

Checking in to ask you about your timeline to address the feedback with a revision? If you think you cannot get to this in the next few weeks, we should put a paused label on it. Let us know! Many thanks!

zdelrosario commented 3 years ago

Hi @ttimbers !

Yes, a pause would be best. I got a lot of really great feedback from the reviewers, but I have a heavy teaching responsibility right now. I'm hoping to get back to this sometime after my Fall semester.

ttimbers commented 3 years ago

OK, thanks for letting me know @zdelrosario! I will this as so and then ping you at the end of the fall semester.

ttimbers commented 2 years ago

@zdelrosario - any chance you are able to pick this review up again in the near future?

zdelrosario commented 2 years ago

@ttimbers yup! My semester ends this week; aiming to use the winter break to address these reviewer comments....

ttimbers commented 2 years ago

Wonderful, thanks for the update! I will remove the pause label at the end of the week then :)

zdelrosario commented 2 years ago

@whedon generate pdf

(Re-building paper with suggested edits.)

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

zdelrosario commented 2 years ago

@whedon generate pdf

(Re-building paper to correct typos; collect small edits)

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

zdelrosario commented 2 years ago

@ttimbers , @pschloss , @firasm

Many thanks to @pschloss and @firasm for the detailed and helpful comments. Clearly it has taken me a long while to respond to this review. A major obstacle for me was finding the time to teach myself some of the GitHub Action trickery necessary to fully address the reviewer comments. However, I believe I have found a solution that substantially improves the infrastructure of the repository, which will make these materials much easier for both teachers and learners to use the content. Please see links to reviewer comments below, and my responses to those comments:

Comments from @pschloss

The License
- As suggested, I have changed to a CC-SA license
The Statement of Need
- (In the paper) As suggested, I have removed the unproductive dunk on passive learning....
- (In the paper) I have more-clearly articulated the need/niche that these materials fill: This is an active-learning data science curriculum aimed at engineering students
- With that in mind, is is possible to change the title of the paper submission? I think that 'An Open-Source Active Learning Curriculum for Data Science in Engineering' captures the more-focused need.
The Installation Instructions
- I have made substantial changes to the infrastructure and installation instructions; the materials now auto-update to a user-friendly build via GitHub Action. This is reflected in the updated 'How to Use This Repo' instructions in the root readme.
Usage
- Related to above; the instructions now point to a single archive from the (new) build branch, which drastically simplifies the process of setting up the materials.
- As-suggested, I have also added a .Rproj file to the repo (and build). This is no longer necessary to find the correct directory, but it does help with quality-of-life issues (to my knowledge, the Git interface for Rstudio only works when a project is open).

Comments from @firasm

Statement of need
- (In the paper) As suggested, I made my references to existing materials a bit more selective
- (In the paper) Made the suggested edit(s) around the Freeman reference
- (In the paper) Made a note about OER
Installation Instructions
- Made substantial improvement to these instructions: Automated a build branch through GitHub Actions to produce a user-friendly zip archive. This obviates the need for a local run of make, and automatically down-selects to just the challenges/ and exercises_sequenced/ directories.
- These changes are reflected in the use instructions.
- I have also built a web version of the solutions; this is a more handy and accessible version of the solutions. Also, as you suggested, this is a nice way to advertise the content!
Usage
- The need to run make is obviated by the updates to Installation Instructions mentioned above.
- The confusing discrepancies between exercises/ and exercises_sequenced/ is obviated (for learners) by the updates to Installation Instructions mentioned above.
- (In the paper) I also describe how an instructor can adapt these materials.
New Release
- As suggested, I have cut a new release.
Pedagogical Design
- Added a direct link to the learning outcomes in the paper.
- Added the suggested citation for flipped classrooms.
Sequence
- I believe this confusion is obviated (for learners) by the updates to Installation Instructions mentioned above.
Overall Feedback
- As @pschloss suggested I changed to a CC-SA license; I hope this preserves the same level of openness as the MIT license...!
- I have also built a web version of the solutions; as you suggested, this is a nice way to advertise the content!
- Added further description of the materials and sequence in the paper.
- Added a description of how these materials have been used---and how one might adapt them---in the paper.
- Added the "development story" in the paper.
- Key models are listed in the learning outcomes document (now linked from the paper). Note that this also points to specific exercises that implement each learning outcome.
- The learning outcomes is now linked from the paper; this provides an overview of the course content without needing to reference every individual file.

firasm commented 2 years ago

This is fantastic @zdelrosario - well done. It's been a long time but I think you've addressed all my comments, so I'm happy!

The content is much more discoverable now. I except it will also make it easy for others to use your work as well. I'm not sure what your overall goal is with the project, but if you like, you can convert the repo to a "template" repo (this is different from forking) so others can get an exact duplicate of your repo with all the actions and the machinery.

pschloss commented 2 years ago

I'm happy with the changes that were made. Well done!

zdelrosario commented 2 years ago

@firasm Thanks! I'll give the template repository option some thought: Helping others use the action infrastructure definitely sounds useful, but doesn't that also prevent forking? I would also like to allow folks to easily contribute to improving the materials, so it seems like a tradeoff to me.

@pschloss Thanks!

@ttimbers Please let me know if there is anything else I need to do to finish up this revision.

firasm commented 2 years ago

Template repos can still be forked - they are mostly independent.

zdelrosario commented 2 years ago

@firasm good to know! No reason not to make it a template repo then. I'll make sure to add some documentation on this use-case.

ttimbers commented 2 years ago

Thanks @zdelrosario for the revisions in response to @pschloss and @firasm 's very helpful and critical feedback. I will regenerate the PDF and then accept the manuscript (which I just read over again, and enjoyed reading very much!).

ttimbers commented 2 years ago

@whedon generate pdf

whedon commented 2 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

ttimbers commented 2 years ago

@whedon recommend-accept

whedon commented 2 years ago

No archive DOI set. Exiting...

ttimbers commented 2 years ago

@zdelrosario - I see we have not yet archived this. What we need for you to do next step is to archive this on Zenodo, issue a tagged release on the repository and generate a DOI. Please me know if I can assist you in any way on this. Once you have that, I will recommend accept again.

zdelrosario commented 2 years ago

@ttimbers sure thing! I've already created a release on GitHub, and just uploaded an archived copy of this to Zenodo. This has minted the doi: 10.5281/zenodo.6314141

ttimbers commented 2 years ago

@whedon set 10.5281/zenodo.6314141 as archive

whedon commented 2 years ago

OK. 10.5281/zenodo.6314141 is the archive.

ttimbers commented 2 years ago

@whedon recommend-accept

whedon commented 2 years ago

Attempting dry run of processing paper acceptance...

whedon commented 2 years ago

:wave: @openjournals/jose-eics, this paper is ready to be accepted and published.

Check final proof :point_right: https://github.com/openjournals/jose-papers/pull/81

If the paper PDF and Crossref deposit XML look good in https://github.com/openjournals/jose-papers/pull/81, then you can now move forward with accepting the submission by compiling again with the flag deposit=true e.g.

@whedon accept deposit=true

whedon commented 2 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1073/pnas.1319030111 is OK
- 10.21105/joss.01686 is OK
- 10.18637/jss.v064.i04 is OK

MISSING DOIs

- 10.2307/1403699 may be a valid DOI for title: Statistical thinking in empirical enquiry
- 10.1093/oso/9780190222055.003.0002 may be a valid DOI for title: Exploratory data analysis
- 10.2307/1403333 may be a valid DOI for title: New pedagogy and new content: The case of statistics

INVALID DOIs

- 10.18260/1-2–22585 is INVALID

zdelrosario commented 2 years ago

@whedon check references

Hmm, I checked that last invalid DOI, and it appears to be valid in the raw bibtex file, reproduced below.

  doi={10.18260/1-2--22585}

Looks like that double dash -- might be getting converted to a single dash? Removed the offending DOI temporarily for testing....

Also couldn't find a DOI for Tukey's EDA, but that's a rather old reference, so I'm not surprised.

whedon commented 2 years ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1073/pnas.1319030111 is OK
- 10.21105/joss.01686 is OK
- 10.18637/jss.v064.i04 is OK
- 10.2307/1403699 is OK
- 10.2307/1403333 is OK

MISSING DOIs

- 10.18260/1-2--22585 may be a valid DOI for title: The flipped classroom: A survey of the research
- 10.1093/oso/9780190222055.003.0002 may be a valid DOI for title: Exploratory data analysis

INVALID DOIs

- None

ttimbers commented 2 years ago

@zdelrosario - apologies for the delay! Was consulting with more experienced folks on what to do here. 10.18260/1-2--22585 looks like a valid DOI and so we can ignore the check saying that it is missing (the checks are not perfect). And yes, as Tukey's book is from quite a while ago, it has no DOI. So again, we can ignore the missing check on that.

ttimbers commented 2 years ago

@openjournals/jose-eics I think this paper should now be ready for full acceptance. Please let me know if I missed a step with whedon.

labarba commented 2 years ago

@whedon accept deposit=true

whedon commented 2 years ago

Doing it live! Attempting automated processing of paper acceptance...

whedon commented 2 years ago

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSE! 🚨🚨🚨

Here's what you must now do:

Check final PDF and Crossref metadata that was deposited :point_right: https://github.com/openjournals/jose-papers/pull/82
Wait a couple of minutes, then verify that the paper DOI resolves https://doi.org/10.21105/jose.00117
If everything looks good, then close this review issue.
Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

labarba commented 2 years ago

Congratulations, @zdelrosario, your JOSE paper is published! 🚀

Huge thanks to our Editor: @ttimbers, and the Reviewers: @firasm, @pschloss — we couldn't do this without you 🙏

whedon commented 2 years ago

:tada::tada::tada: Congratulations on your paper acceptance! :tada::tada::tada:

If you would like to include a link to your paper from your README use the following code snippets:

Markdown:
[![DOI](https://jose.theoj.org/papers/10.21105/jose.00117/status.svg)](https://doi.org/10.21105/jose.00117)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/jose.00117">
  <img src="https://jose.theoj.org/papers/10.21105/jose.00117/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://jose.theoj.org/papers/10.21105/jose.00117/status.svg
   :target: https://doi.org/10.21105/jose.00117

This is how it will look in your documentation:

We need your help!

Journal of Open Source Education is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: http://jose.theoj.org/reviewer-signup.html
Making a small donation to support our running costs here: https://numfocus.org/donate-to-jose

zdelrosario commented 2 years ago

Thanks @labarba , @ttimbers , @firasm , @pschloss , and everyone at the JOSE team! This whole review process has significantly improved the teaching materials, and was overall a very positive experience!

openjournals / jose-reviews