[REVIEW]: ClimateEstimate.net: A tutorial on climate econometrics

[whedon]

Submitting author: !--author-handle-->@jrising<!--end-author-handle-- (James Rising) Repository: https://github.com/atrisovic/weather-panel.github.io Branch with paper.md (empty if default branch): Version: v2.0 Editor: !--editor-->@kyleniemeyer<!--end-editor-- Reviewers: @jwagemann, @kls2177 Archive: 10.5281/zenodo.10833818

Status

Status badge code:

HTML: <a href="https://jose.theoj.org/papers/b8083032c189d1d472dc228b55ccd086"><img src="https://jose.theoj.org/papers/b8083032c189d1d472dc228b55ccd086/status.svg"></a>
Markdown: [![status](https://jose.theoj.org/papers/b8083032c189d1d472dc228b55ccd086/status.svg)](https://jose.theoj.org/papers/b8083032c189d1d472dc228b55ccd086)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@jwagemann & @kls2177, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

1. Make sure you're logged in to your GitHub account
2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @kyleniemeyer know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @jwagemann

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSE code of conduct.

General checks

• [x] Repository: Is the source for this learning module available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of a standard license? (OSI-approved for code, Creative Commons for content)
• [x] Version: Does the release version given match the repository release (v1.2)?
• [x] Authorship: Has the submitting author (@jrising) made visible contributions to the module? Does the full list of authors seem appropriate and complete?

Documentation

• [x] A statement of need: Do the authors clearly state the need for this module and who the target audience is?
• [x] Installation instructions: Is there a clearly stated list of dependencies?
• [x] Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the module 2) Report issues or problems with the module 3) Seek support

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

• [x] Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
• [x] Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
• [x] Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
• [x] Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
• [x] Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

JOSE paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Does the paper clearly state the need for this module and who the target audience is?
• [x] Description: Does the paper describe the learning materials and sequence?
• [ ] Does it describe how it has been used in the classroom or other settings, and how someone might adopt it?
• [x] Could someone else teach with this module, given the right expertise?
• [x] Does the paper tell the "story" of how the authors came to develop it, or what their expertise is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @kls2177

Conflict of interest

• [x] As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSE code of conduct.

General checks

• [x] Repository: Is the source for this learning module available at the repository url?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of a standard license? (OSI-approved for code, Creative Commons for content)
• [x] Version: Does the release version given match the repository release (v1.2)?
• [x] Authorship: Has the submitting author (@jrising) made visible contributions to the module? Does the full list of authors seem appropriate and complete?

Documentation

• [x] A statement of need: Do the authors clearly state the need for this module and who the target audience is?
• [x] Installation instructions: Is there a clearly stated list of dependencies?
• [x] Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the module 2) Report issues or problems with the module 3) Seek support

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

• [x] Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
• [x] Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
• [x] Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
• [x] Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
• [x] Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

JOSE paper

• [x] Authors: Does the paper.md file include a list of authors with their affiliations?
• [x] A statement of need: Does the paper clearly state the need for this module and who the target audience is?
• [x] Description: Does the paper describe the learning materials and sequence?
• [x] Does it describe how it has been used in the classroom or other settings, and how someone might adopt it?
• [x] Could someone else teach with this module, given the right expertise?
• [x] Does the paper tell the "story" of how the authors came to develop it, or what their expertise is?
• [x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

[whedon]

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @howardjp, @jwagemann it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/jose-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

1. Set yourself as 'Not watching' https://github.com/openjournals/jose-reviews:

2. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf

[whedon]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1126/science.aad9837 is OK
- 10.3386/w25189 is OK
- 10.3386/w22181 is OK
- 10.1088/1748-9326/ab281e is OK
- 10.1002/wcc.579 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[kyleniemeyer]

👋 @jrising @howardjp @jwagemann the actual review for this submission will take place in this issue. Please note the review instructions and checklists at the top of the issue ☝️

[atrisovic]

@whedon generate pdf

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[atrisovic]

We have a new title 😄

[howardjp]

We have a new title 😄

That's a good title.

[jwagemann]

Hi @kyleniemeyer It seems that I cannot trigger the checklist. Have I accepted the invitation to review? If I follow the link now, I get an error message.

[atrisovic]

Hi @jwagemann, how are you doing? 👋 I had the same problem when I was reviewing another submission, but I somehow solved it by making a change in the URL. I don't exactly remember what I changed 😄 but maybe this thread could help.

[jwagemann]

Hi @atrisovic . As far as I understand it this brings me to the error message. Maybe @kyleniemeyer can re-assign me? Thanks.

[kyleniemeyer]

@whedon re-invite @jwagemann as reviewer

[whedon]

The reviewer already has a pending invite.

@jwagemann please accept the invite by clicking this link: https://github.com/openjournals/jose-reviews/invitations

[kyleniemeyer]

@jwagemann please try that link ☝️

[jwagemann]

@kyleniemeyer - I get 'Sorry, we couldn't find that repository invitation. It is possible that the invitation was revoked or that you are not logged into the invited account.'

Before it was saying that the invite expired.

[atrisovic]

I think for me it worked when I viewed invitations from my profile, ie, something along the lines: https://github.com/<username>/<reponame>/invitations 🤔

[jwagemann]

It would be something like this in my case

https://github.com/jwagemann/openjournals/jose-reviews/invitations? This brings me to a 404.

[kyleniemeyer]

@openjournals/dev can you help here? not sure what is going on

[arfon]

@whedon re-invite @jwagemann as reviewer

[whedon]

I'm sorry @arfon, I'm afraid I can't do that. That's something only editors are allowed to do.

[arfon]

Can you try this URL again please @jwagemann: https://github.com/openjournals/jose-reviews/invitations

[jwagemann]

Great. Thanks @arfon . It works now.

[jwagemann]

I thank the authors for their time to develop the tutorial ‘ClimateEstimate.net - A tutorial on climate econometrics’ and giving me the opportunity to review it. The tutorial claims to be a ‘step-by-step guidance on carrying out a climatic econometric analysis’. My recommendation is to accept with major revisions for the following reasons:

• at the current state, it fails to meet the aim to be a step-by-step guide on how to carry out climatic econometrics. Sections often seem arbitrary and for a learner or external person it is often hard to understand what the section aims to describe or to do
• Many parts are not easy to follow, as authors try to capture too many aspects. Also many headlines seem a bit out of place, e.g the headline ‘Geographic Information Systems’ is used when the shapefile format is explained
• at the current state, the tutorial cannot be used by anyone other than the authors in order to teach students on the important topic of climatic econometrics
• The smart use of images and explanatory media is missing. The tutorial is heavily based on text and hardly works with additional images that help to understand the complex content
• The general writing needs to be spell-checked and the general tone feels somehow a bit sloppy: e.g. ‘here are some suggestions for dealing with the mess that is political geography’ . I would not use the word ‘mess’ in any type of tutorial.
• The tutorial does not follow a consistent instructional design. Some sections are accompanied with code examples, others explain the content only in a very abstract way.
• Coding examples are either in R or try to show examples in multiple programming languages, which is quite confusing for the learner. I recommend to focus only on one programming language and provide only links to similar packages in other programming languages
• The tutorial misses an accompanying coding script in one of the programming languages. Without a coding script, it is very hard that a learner will be able to implement the content explained

I opened a series of issues with recommendations / comments for each section in the tutorial:

• General comments
• Comments to Introduction section
• Comments to Weather and Climate Data section
• Comments to Developing a reduced-form specification section
• Comments to Weighting Schemes section
• Comments to Geographical Unit Data section
• Comments to Work Organisation section

I am happy to review a new submission.

[atrisovic]

Hi @jwagemann, thank you very much for your time, thorough review, and great comments! We will start working on this right away and follow up once we go through all the comments :)

[labarba]

Hi @jrising 👋 — do let us know how you are getting along with revising your submission! We are all digging out of the rubble of piled up work, and overall world chaos, so just give as a quick update when you can.

[jrising]

Thank you @labarba for the prompt! We have been going through the suggestions and thinking broadly about how to improve the structure of the tutorial in light of these comments. I have just replied to the full list of helpful comments. Here are responses to the points above from @jwagemann.

at the current state, it fails to meet the aim to be a step-by-step guide on how to carry out climatic econometrics. Sections often seem arbitrary and for a learner or external person it is often hard to understand what the section aims to describe or to do

We have made several improvements to address this. We added "learning objectives and decision points" boxes to the top of each main section of the tutorial. We added a consistent "hands-on exercise" to show how the various points can be put together. And we have made a number of improvements in the organization of sections, splitting up sections to make the key points clearer.

Many parts are not easy to follow, as authors try to capture too many aspects. Also many headlines seem a bit out of place, e.g the headline ‘Geographic Information Systems’ is used when the shapefile format is explained

This comment drove us to reconsider the structure of each section. In many cases,we have split up a large section (such as the one relating to GIS concepts) into multiple smaller sections. For example, we first introduce GIS concepts, then have a section on how to get shapefiles, and then a section on how to use them.

at the current state, the tutorial cannot be used by anyone other than the authors in order to teach students on the important topic of climatic econometrics

We have tried to make the tutorial much more accessible, and have shared it with more people. Crucially, we have also been more explicit about the background knowledge (mainly on econometrics) that we are expecting readers to have (in a section "Who is this tutorial for?"), while also including more references to other resources that will help readers get needed knowledge throughout the tutorial.

The smart use of images and explanatory media is missing. The tutorial is heavily based on text and hardly works with additional images that help to understand the complex content

While the tutorial is still quite "wordy", we have tried to make it more accessible in a few ways. First, we have made better use of Jupyter Book features like tabs (used now to split out different programming languages) and colored boxes (used to reference additional resources and highlight notes). We have also included a "Hands-On Exercise" that will allow readers to engage directly with the concepts in the tutorial. Finally, we have added more explanatory material to go with our images.

The general writing needs to be spell-checked and the general tone feels somehow a bit sloppy: e.g. ‘here are some suggestions for dealing with the mess that is political geography’ . I would not use the word ‘mess’ in any type of tutorial.

We apologize for these errors. We have done a more thorough editing process, and we have also passed the tutorial content through spell-checking and readability tools. We have also gotten comments back from other users, and adjusted the language to address their confusion.

The tutorial does not follow a consistent instructional design. Some sections are accompanied with code examples, others explain the content only in a very abstract way.

We hope that our inclusion of a common example will help make even the very general parts of the text more concrete. We have not used the same level of coding examples throughout the text, because of the wide range of challenges we hope that our tutorial can help students address: some challenges are conceptual, some informational, some mathematical, and some coding. We have worked to make the tutorial as a whole more consistent however.

Coding examples are either in R or try to show examples in multiple programming languages, which is quite confusing for the learner. I recommend to focus only on one programming language and provide only links to similar packages in other programming languages

It is quite common in this field to use multiple languages within a single project, and R, python, and Stata are all in common use amongst our audience. For this reason, we have kept the multiple languages, but we have improved how they are presented. In the previous version, they were included in the text in different ways. Now, we consistently use Jupyter Book tabs, to allow readers to quickly get the example in the language they need.

The tutorial misses an accompanying coding script in one of the programming languages. Without a coding script, it is very hard that a learner will be able to implement the content explained

We have now added this, in the form of a common hands-on exercise. The exercise is split into four steps, following the structure of the tutorial, and takes readers from downloading the necessary data through to getting out a dose-response function similar to the ones shown on the first page of the tutorial.

[atrisovic]

Hi @howardjp and @jwagemann, I just wanted to check if you've seen the updates in this submission. I'm looking forward to your thoughts! Of course, no pressure! Thanks! 😄

[labarba]

hi @howardjp, @jwagemann 👋 — It looks like the author of this JOSE submission made improvements and they are waiting for you to take a second look. Can you let us know when you might be able to do so? Thanks!

[atrisovic]

Hi @labarba 👋 , our reviewers don't seem to be active any more. Can we look for new reviewers? Maybe one of these people @/ali-ramadhan, @/marda-science, @/angela-li, @/leouieda can help? :)

[jwagemann]

Hi - it's on my list and I will get to it first week in November. Apologies for the delay - busy period.

[kyleniemeyer]

Hi @jrising @atrisovic, sorry for the major delay here. I have checked in with the reviewers, and so far @jwagemann has confirmed being able to complete a second review in a few weeks.

[jrising]

@kyleniemeyer Thanks for the update.

[kyleniemeyer]

@whedon generate pdf

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[jwagemann]

Hi @kyleniemeyer , I finally had a chance to review the reviewed tutorial and I congratulate @jrising , @atrisovic and the other authors for producing this great resource. It is now well-structured and guides learners nicely through the content.

I recommend to accept. One small proposal of change: Under ERA5, instead of linking to UCAR’s ERA5 description, link to the description on the Climate Data Store: https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview

[atrisovic]

Hi @jwagemann and All, Thanks for the review! We have now incorporated the change! :)

[kyleniemeyer]

@whedon generate pdf

[whedon]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[atrisovic]

Hi @kyleniemeyer, @labarba, what are the next steps for us? Do we need a second review at this stage (@howardjp)? Maybe someone else can step in if needed (ie, @/ali-ramadhan, @/marda-science, @/angela-li, @/leouieda). Thank you all!

[kyleniemeyer]

Hi all, sorry for the (continued) delays here. @leouieda has agreed to serve as a second reviewer, and will take a look next week—but I do think this submission is in good shape.

[kyleniemeyer]

@whedon add @leouieda as reviewer

[whedon]

OK, @leouieda is now a reviewer

[kyleniemeyer]

@whedon remove @howardjp as reviewer

[whedon]

OK, @howardjp is no longer a reviewer

[jrising]

@kyleniemeyer Thanks for moving this forward!

[jrising]

@leouieda @kyleniemeyer Would it be possible to get an update on the review for this? At this point (as of some months ago), we completed the revision process for one reviewer, but we are still waiting on comments from another reviewer. It would be great to get this out!

[ks905383]

@labarba @jgorzalski @kyleniemeyer (possibly @acocac @kls2177 ; my apologies, not sure who's responsible for this) Would it be possible to get an update on the review process for this? Thank you!

[kyleniemeyer]

Hi @ks905383 @jrising unfortunately it looks like we lost a second reviewer. I will do a (non-expert) review myself and try to get this wrapped up soon.

[kls2177]

@kyleniemeyer, I can provide a review (if needed). It is not directly in my area, but I may be able to help. I would probably need until the end of May to complete.

[kyleniemeyer]

@kls2177 that would be great, thank you!