[REVIEW]: Ecological Forecasting and Dynamics: A graduate course on the fundamentals of time series and forecasting in ecology

whedon commented 1 year ago

Submitting author: !--author-handle-->@ethanwhite@acocac<!--end-editor-- Reviewers: @arokem, @mdietze Archive: 10.5281/zenodo.8250308

Status

Status badge code:

HTML: <a href="https://jose.theoj.org/papers/05268bf72723410e76e9d991ad956db8"><img src="https://jose.theoj.org/papers/05268bf72723410e76e9d991ad956db8/status.svg"></a>
Markdown: [![status](https://jose.theoj.org/papers/05268bf72723410e76e9d991ad956db8/status.svg)](https://jose.theoj.org/papers/05268bf72723410e76e9d991ad956db8)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@arokem & @mdietze, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/jose-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @acocac know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @arokem

Conflict of interest

[x] As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSE code of conduct.

General checks

[x] Repository: Is the source for this learning module available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of a standard license? (OSI-approved for code, Creative Commons for content)
[ ] Version: Does the release version given match the repository release (v2.1.0)?
[x] Authorship: Has the submitting author (@ethanwhite) made visible contributions to the module? Does the full list of authors seem appropriate and complete?

Documentation

[x] A statement of need: Do the authors clearly state the need for this module and who the target audience is?
[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the module 2) Report issues or problems with the module 3) Seek support

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

[x] Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
[x] Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
[x] Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
[x] Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
[x] Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

JOSE paper

[x] Authors: Does the paper.md file include a list of authors with their affiliations?
[x] A statement of need: Does the paper clearly state the need for this module and who the target audience is?
[x] Description: Does the paper describe the learning materials and sequence?
[x] Does it describe how it has been used in the classroom or other settings, and how someone might adopt it?
[x] Could someone else teach with this module, given the right expertise?
[x] Does the paper tell the "story" of how the authors came to develop it, or what their expertise is?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @mdietze

Conflict of interest

[x] As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSE code of conduct.

General checks

[x] Repository: Is the source for this learning module available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of a standard license? (OSI-approved for code, Creative Commons for content)
[x] Version: Does the release version given match the repository release (v2.1.0)?
[x] Authorship: Has the submitting author (@ethanwhite) made visible contributions to the module? Does the full list of authors seem appropriate and complete?

Documentation

[x] A statement of need: Do the authors clearly state the need for this module and who the target audience is?
[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the module 2) Report issues or problems with the module 3) Seek support

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

[x] Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
[x] Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
[x] Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
[x] Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
[x] Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

JOSE paper

[x] Authors: Does the paper.md file include a list of authors with their affiliations?
[x] A statement of need: Does the paper clearly state the need for this module and who the target audience is?
[x] Description: Does the paper describe the learning materials and sequence?
[x] Does it describe how it has been used in the classroom or other settings, and how someone might adopt it?
[x] Could someone else teach with this module, given the right expertise?
[x] Does the paper tell the "story" of how the authors came to develop it, or what their expertise is?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)

whedon commented 1 year ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @arokem, @gvdr it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/jose-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

Set yourself as 'Not watching' https://github.com/openjournals/jose-reviews:

watching

You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

whedon commented 1 year ago

Wordcount for paper.md is 1574

whedon commented 1 year ago

Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.09 s (1516.5 files/s, 91614.1 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Markdown                       112           1370              0           4128
JSON                             2              0              0            977
HTML                             3             33              0            194
TeX                              1             17              0            180
YAML                             5             25             52            161
Rmd                              3            123            263            118
TOML                             2             37             65            100
R                                1             11              2             39
Bourne Shell                     1              0             14              3
SVG                              1              0              0              2
-------------------------------------------------------------------------------
SUM:                           131           1616            396           5902
-------------------------------------------------------------------------------

Statistical information for the repository '1090112a3505ce460c8c55e3' was
gathered on 2023/02/27.
No commited files with the specified extensions were found.

whedon commented 1 year ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1002/ecy.3709 is OK
- 10.1002/ecy.2373 is OK
- 10.1073/pnas.1710231115 is OK
- 10.1101/332783 is OK
- 10.1093/biosci/bix025 is OK
- 10.7717/peerj.4278 is OK
- 10.1111/oik.03726 is OK
- 10.1177/002248718703800308 is OK
- 10.1002/ecy.3431 is OK
- 10.2218/ijdc.v10i1.351 is OK
- 10.1111/2041-210X.13104 is OK
- 10.21105/jose.00139 is OK
- 10.12688/f1000research.3-62.v2 is OK
- 10.1111/ele.12353 is OK
- 10.1371/journal.pbio.3000125 is OK

MISSING DOIs

- None

INVALID DOIs

- 10.9790/7388-05616670 is INVALID
- 10.1890/0012-9658(2003)084[1412:TETTWU]2.0.CO;2 is INVALID

whedon commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

acocac commented 1 year ago

👋 @arokem & @gvdr we will conduct the review in this issue.

Please read through the above information and let me know if you have any questions about process.

Thank you 🙏

whedon commented 1 year ago

:wave: @arokem, please update us on how your review is going (this is an automated reminder).

whedon commented 1 year ago

:wave: @gvdr, please update us on how your review is going (this is an automated reminder).

acocac commented 1 year ago

@arokem @gvdr hope you're doing well. The above messages are automated reminders. I encourage you both to revisit the submission and your individual review checklist.

arokem commented 1 year ago

Hello! I am a bit stuck on the first checklist item "Is the source for this learning module available at the repository url?" This is because I am not sure what exactly I should be reviewing. In browsing through the course website, I am finding many links to "materials" pages that don't have any content:

https://course.naturecast.org/lessons/intro-time-series-data/material/ https://course.naturecast.org/lessons/regime-shifts/material/

Or contain some links to external readings:

https://course.naturecast.org/lessons/paleo-dynamics/material/ https://course.naturecast.org/lessons/phenology/material/ https://course.naturecast.org/lessons/community-dynamics/material/

(and several others).

There are also many pages that contain links to R tutorials about the materials, which seem to be the meat of it. I think that the structure of this website may just be confusing. What a "materials" page means, for example. I think that it would be helpful to provide a "how to use this course" page that explains the structure of the site/repo and guides potential users/students a bit more. At any rate, I would like to have some clarification, before I proceed to review the materials.

acocac commented 1 year ago

@arokem thanks for having a look at the resource. I agree a how to use tab or section would be very beneficial to facilitate users/contributors navigation through the e-learning resource. You are alro right there are some emtpy sections. Feel free to point them in the repository issues tab. They should be removed or completed by @ethanwhite and collaborators as part of the version submitted to JOSE.

If you go to our EiC @labarba comment on this PREVIEW submission, JOSE considers videos as supplementary material that we do not review. In general, we do not review videos, or slide presentations, or other forms of OpenCourseWare (OCW), because they cannot be modified or easily reused. JOSE reviews open-source material that is reusable and allows derived works. Also, when you make an archive of the repository, the videos will not be included.

acocac commented 1 year ago

@ethanwhite I would appreciate if you have any comments about empty pages in the version submitted to JOSE. @arokem already opened an issue (#21) to highlight this. Thanks for your collaboration.

ethanwhite commented 1 year ago

@acocac - we'll definitely get those cleaned up in revision. I'd been waiting to see what else the review turned up before fixing them, but if you'd like them done immediately just let me know and I can add it to my To Do list.

ethanwhite commented 1 year ago

From a review perspective the Material tabs are not the core material and so maybe need to be renamed. It is basically links to reading and background information for students. The course "source" is the Discussion Questions, Instructor Notes, and R Tutorial sections. While the beginning of the R tutorials is often video - if you scroll down there is a full set of written R tutorial material on each page. Hope this helps.

acocac commented 1 year ago

@arokem please read the comments above. Please continue the revision and checklist according to the course source suggested by @ethanwhite. Note you should follow the guidelines for JOSE reviewers.

arokem commented 1 year ago

That helps. Will do. Thanks!

acocac commented 1 year ago

Hi @gvdr, since we have not hard from you in several weeks, we are now looking for a new reviewer. Thank you for your original willingness to contribute a review.

acocac commented 1 year ago

@whedon remove @gvdr as reviewer

whedon commented 1 year ago

OK, @gvdr is no longer a reviewer

acocac commented 1 year ago

@whedon remind @arokem in two days

whedon commented 1 year ago

Reminder set for @arokem in two days

whedon commented 1 year ago

:wave: @arokem, please update us on how your review is going (this is an automated reminder).

arokem commented 1 year ago

Hello! Slowly, but I will have some time on Friday to work through this.

arokem commented 1 year ago

Version: Does the release version given match the repository release (v2.1.0)?

I see that the repo is now at v2.1.1, but I assume that's not a problem for the review. Leaving unchecked for now until that's confirmed.

arokem commented 1 year ago

Installation instructions: Is there a clearly stated list of dependencies?

The only list or instructions for installation that I could find is in the README of the repository. Unfortunately, these instructions are not for the learners/instructors, but for folks interested in building the (rather sophisticated...) website that currently serves these materials. I would suggest that these instructions should go into someplace more obscure and the README should provide a more welcoming entry point for potential learners and teachers of these materials, that would facilitate the reuse of the materials.

arokem commented 1 year ago

Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?

This might be in there somewhere, but I couldn't find it. In general, I think that the structure of this repo and the associated website are really hard to understand and are not currently designed to easily facilitate reuse.

arokem commented 1 year ago

Regarding content: I think that the content covered and pedagogy are both really great! Clearly, the authors have a lot of experience teaching this kind of material and the coverage is really ambitious. Relatedly: I don't think that I know who the target audience for this material is. Is it designed for undergraduates or graduate students? What should they have learned as pre-requisites? Maybe this is stated somewhere, but I couldn't find it.

One suggestion for an advanced topic that you might want to cover is the use of generalized additive models for time-series modeling with mgcv. Gavin Simpson wrote a great introductory paper (and lots of blog posts) on this topic that you can use as a starting point. Admittedly, it's not a great forecasting model, though, so possibly out of scope for this course.

arokem commented 1 year ago

The paper manuscript is really well-written, motivates the work well, and provides a clear story of how these materials came about, the design principles underlying the materials' organization, and how they might be used (with the caveat that it would be good to specify the target audience more clearly, see above).

arokem commented 1 year ago

OK - I am done with a review of this work. Based on my feedback, I've left the following items unchecked:

[ ] Version: Does the release version given match the repository release (v2.1.0)?
[ ] Installation instructions: Is there a clearly stated list of dependencies?
[x] Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?
[x] Could someone else teach with this module, given the right expertise?

Arguably, the last one could be checked, but I think that the materials are not currently designed to facilitate this kind of usage. I think that the addition of some documentation on how this is all put together into a course might be sufficient to check this box as well, but just wanted to keep this in mind as you work on revisions.

acocac commented 1 year ago

@ethanwhite may I ask if you can address @arokem review? I'm intensely hunting an additional reviewer to replace the initial one we had in the PRE-REVIEW issue.

ethanwhite commented 1 year ago

@acocac - yes, I will put it on the To Do list

acocac commented 1 year ago

@acocac - yes, I will put it on the To Do list

Great - Thanks for let us know. I'll add soon updates for the additional reviewer.

acocac commented 1 year ago

@whedon add @mdietze as reviewer

whedon commented 1 year ago

OK, @mdietze is now a reviewer

acocac commented 1 year ago

Hi @mdietze, thanks for your interest and availability to review the submission in this REVIEW issue for JOSE.

Before starting, I appreciate to declare any relevant conflict of interest here. Please note JOSE reviewers are asked to provide constructive review with the purpose of improving the quality of the submission. We generally do not ask reviewers to “accept/reject” the submission. You should instead recommend publication after all your concerns and suggestions have been addressed. Please update your checklist available at the first comment of this REVIEW issue.

Please feel free to email back or contact in GitHub if you have any questions.

mdietze commented 1 year ago

While I will do my best to provide an objective, constructive review, in the interest of transparency I wanted to disclose the following potential conflicts of interest:

Ethan what a coauthor on this 2018 paper that I led:

Dietze MC, A Fox, L Beck-Johnson, JL Betancourt, MB Hooten, CS Jarnevich, TH Keitt, MA Kenney, CM Laney, LG Larsen, HW Loescher, CK Lunch, B Pijanowski, JT Randerson, EK Read, AT Tredennick, R Vargas, KC Weathers, EP White. 2018. Iterative near-term ecological forecasting: Needs, opportunities, and challenges. Proceedings of the National Academy of Sciences 115 (7) 1424-1432 https://doi.org/10.1073/pnas.1710231115

Ethan and Morgan are coauthors on this "perspective" paper in review:

Dietze M, EP White, A Abeyta, C Boettiger, N Bueno Watts, C Carey, R Chaplin-Kramer, R Emanuel, SKM Ernest, R Figueiredo, Renato MD Gerst, LR Johnson, MA Kenney, J McLachlan, IC Paschalidis, JA Peters, CR Rollinson, J Simonis, K Sullivan-Wiley, RQ Thomas, GM Wardle, A Willson, J Zwart. 2023. “Forecasting the Field of Ecological Forecasting” Nature Climate Change

Ethan and Morgan are Co-I and Senior Personnel (respectively) on the NSF STC proposal “National Center for Ecological Forecasting” that I’m lead PI on. We have not received a formal “reject”, but site visits occurred in Jan/Feb 2023 and we were not invited.
Ethan was previously on the Ecological Forecasting Initiative steering committee, though he rotated off over a year ago (March 2022). I was then, and still am now, the chair.

acocac commented 1 year ago

While I will do my best to provide an objective, constructive review, in the interest of transparency I wanted to disclose the following potential conflicts of interest:

Ethan what a coauthor on this 2018 paper that I led:

Dietze MC, A Fox, L Beck-Johnson, JL Betancourt, MB Hooten, CS Jarnevich, TH Keitt, MA Kenney, CM Laney, LG Larsen, HW Loescher, CK Lunch, B Pijanowski, JT Randerson, EK Read, AT Tredennick, R Vargas, KC Weathers, EP White. 2018. Iterative near-term ecological forecasting: Needs, opportunities, and challenges. Proceedings of the National Academy of Sciences 115 (7) 1424-1432 https://doi.org/10.1073/pnas.1710231115

Ethan and Morgan are coauthors on this "perspective" paper in review:

Dietze M, EP White, A Abeyta, C Boettiger, N Bueno Watts, C Carey, R Chaplin-Kramer, R Emanuel, SKM Ernest, R Figueiredo, Renato MD Gerst, LR Johnson, MA Kenney, J McLachlan, IC Paschalidis, JA Peters, CR Rollinson, J Simonis, K Sullivan-Wiley, RQ Thomas, GM Wardle, A Willson, J Zwart. 2023. “Forecasting the Field of Ecological Forecasting” Nature Climate Change

Ethan and Morgan are Co-I and Senior Personnel (respectively) on the NSF STC proposal “National Center for Ecological Forecasting” that I’m lead PI on. We have not received a formal “reject”, but site visits occurred in Jan/Feb 2023 and we were not invited.

Ethan was previously on the Ecological Forecasting Initiative steering committee, though he rotated off over a year ago (March 2022). I was then, and still am now, the chair.

Dear @mdietze, I really appreciate the transparency for making public previous work with @ethanwhite.

Following the instructions by JOSE EiC @labarba, please go ahead with a constructive review to improve the quality of the submission.

Thanks for volunteering!

acocac commented 1 year ago

@whedon remind @mdietze in one days

whedon commented 1 year ago

Reminder set for @mdietze in one days

whedon commented 1 year ago

:wave: @mdietze, please update us on how your review is going (this is an automated reminder).

mdietze commented 1 year ago

Installation instructions were clear and easy. I was able to fork the repo and launch in netlify.com in only a couple minutes
I skimmed through all of the lessons, both those that are “science” focused and those that are “code” focused and found the learning objectives to be clear, the discussion questions to be great, the materials to be appropriate, the quality to be high, and the pedagogy to be better than most (all?) of my own work.
I was particularly happy to see the inclusion of “non-technical” sections, such as the lesson on forecast ethics and the paleoecological context
Paper describing the course was straightforward and easy to read
Overall, I have no requested changes that might prevent publication.

acocac commented 1 year ago

@ethanwhite may I ask your attention to the remaining items and comments left by @arokem? @mdietze also provided a constructive and positive feedback of the submission.

acocac commented 1 year ago

Reminder set for @ethanwhite in two days

ethanwhite commented 1 year ago

Thanks everyone for the reviews and apology for the delays (a combination of planned and unplanned travel has kept me running all over the place this summer). Hoping to get this turned around to all of you in the next couple of weeks.

acocac commented 1 year ago

Thanks everyone for the reviews and apology for the delays (a combination of planned and unplanned travel has kept me running all over the place this summer). Hoping to get this turned around to all of you in the next couple of weeks.

@ethanwhite thanks for the reply. Feel free to reach us here in case of any additional updates.

ethanwhite commented 1 year ago

A big thanks to @arokem & @mdietze for the helpful and generous reviews and @acocac for herding all of us cats.

We have implemented changes designed to address all of the comments both in this review issue and in the individual issues submitted by @arokem in the course repository. This includes the following changes to address @arokem's main suggestions related to providing documentation on how to adopt the lesson material:

We have added an audience description (including prerequisite knowledge) to both the paper (see the new Audience section) and the website.
We have added a detailed Getting Started page to the website that explains how both students and teachers should start interfacing the material and linking to this in both the 'Self-guided online learning' and 'Reuse and remixing' sections of the paper.
We have revised the README to start with a brief Getting Started section that outlines the basics of how to adopt/adapt the material and links to the Getting Started page on the website for more details. The additional technical detail related to building the site follows below for users who want to fully leverage the infrastructure.
We have rename pages titled "Material" to clearly reference the specific things on those pages using the following more specific titles: "Reading", "Reading & Video", and "Data & Software".
Combining (3) & (4) there are now two sets of installation instructions: 1) the infrastructure installation instructions provided in the README; and 2) The "Data & Software" pages on the website which provide information on the data to download and packages to install for working through the R tutorials. These are the only two installation components to the course.
Regarding @arokem's suggestion of adding GAMs to the class - it's been on our To Do list for a while and we'll hopefully have time to implement it when we teach the course later this year.

Since @skmorgane & I have just completed this work we're waiting on @ha0ye to make sure he agrees with the changes, but I think given the fluid nature of JOSE review that it would be fine for @arokem to take a look at the current changes and see if they address the associated comments. Once @ha0ye and @arokem are both happy with things I'll make a new release with all of these changes (and any more that they suggest).

ha0ye commented 1 year ago

No major objections from me. I'm going to do some copy-editing and submit a PR off of my fork.

ethanwhite commented 1 year ago

@acocac - I've now merged @ha0ye's edits so we're all done on our end for the moment

arokem commented 1 year ago

@whedon generate pdf

editorialbot commented 1 year ago

My name is now @editorialbot

openjournals / jose-reviews