[REVIEW]: ClimateEstimate.net: A tutorial on climate econometrics

whedon commented 4 years ago

Submitting author: !--author-handle-->@jrising@kyleniemeyer<!--end-editor-- Reviewers: @jwagemann, @kls2177 Archive: 10.5281/zenodo.10833818

Status

Status badge code:

HTML: <a href="https://jose.theoj.org/papers/b8083032c189d1d472dc228b55ccd086"><img src="https://jose.theoj.org/papers/b8083032c189d1d472dc228b55ccd086/status.svg"></a>
Markdown: [![status](https://jose.theoj.org/papers/b8083032c189d1d472dc228b55ccd086/status.svg)](https://jose.theoj.org/papers/b8083032c189d1d472dc228b55ccd086)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@jwagemann & @kls2177, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

Make sure you're logged in to your GitHub account
Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @kyleniemeyer know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @jwagemann

Conflict of interest

[x] As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSE code of conduct.

General checks

[x] Repository: Is the source for this learning module available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of a standard license? (OSI-approved for code, Creative Commons for content)
[x] Version: Does the release version given match the repository release (v1.2)?
[x] Authorship: Has the submitting author (@jrising) made visible contributions to the module? Does the full list of authors seem appropriate and complete?

Documentation

[x] A statement of need: Do the authors clearly state the need for this module and who the target audience is?
[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the module 2) Report issues or problems with the module 3) Seek support

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

[x] Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
[x] Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
[x] Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
[x] Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
[x] Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

JOSE paper

[x] Authors: Does the paper.md file include a list of authors with their affiliations?
[x] A statement of need: Does the paper clearly state the need for this module and who the target audience is?
[x] Description: Does the paper describe the learning materials and sequence?
[ ] Does it describe how it has been used in the classroom or other settings, and how someone might adopt it?
[x] Could someone else teach with this module, given the right expertise?
[x] Does the paper tell the "story" of how the authors came to develop it, or what their expertise is?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

Review checklist for @kls2177

Conflict of interest

[x] As the reviewer I confirm that I have read the JOSE conflict of interest policy and that there are no conflicts of interest for me to review this work.

Code of Conduct

[x] I confirm that I read and will adhere to the JOSE code of conduct.

General checks

[x] Repository: Is the source for this learning module available at the repository url?
[x] License: Does the repository contain a plain-text LICENSE file with the contents of a standard license? (OSI-approved for code, Creative Commons for content)
[x] Version: Does the release version given match the repository release (v1.2)?
[x] Authorship: Has the submitting author (@jrising) made visible contributions to the module? Does the full list of authors seem appropriate and complete?

Documentation

[x] A statement of need: Do the authors clearly state the need for this module and who the target audience is?
[x] Installation instructions: Is there a clearly stated list of dependencies?
[x] Usage: Does the documentation explain how someone would adopt the module, and include examples of how to use it?
[x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the module 2) Report issues or problems with the module 3) Seek support

Pedagogy / Instructional design (Work-in-progress: reviewers, please comment!)

[x] Learning objectives: Does the module make the learning objectives plainly clear? (We don't require explicitly written learning objectives; only that they be evident from content and design.)
[x] Content scope and length: Is the content substantial for learning a given topic? Is the length of the module appropriate?
[x] Pedagogy: Does the module seem easy to follow? Does it observe guidance on cognitive load? (working memory limits of 7 +/- 2 chunks of information)
[x] Content quality: Is the writing of good quality, concise, engaging? Are the code components well crafted? Does the module seem complete?
[x] Instructional design: Is the instructional design deliberate and apparent? For example, exploit worked-example effects; effective multi-media use; low extraneous cognitive load.

JOSE paper

[x] Authors: Does the paper.md file include a list of authors with their affiliations?
[x] A statement of need: Does the paper clearly state the need for this module and who the target audience is?
[x] Description: Does the paper describe the learning materials and sequence?
[x] Does it describe how it has been used in the classroom or other settings, and how someone might adopt it?
[x] Could someone else teach with this module, given the right expertise?
[x] Does the paper tell the "story" of how the authors came to develop it, or what their expertise is?
[x] References: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

kyleniemeyer commented 1 year ago

@whedon remove @leouieda as reviewer

whedon commented 1 year ago

OK, @leouieda is no longer a reviewer

kyleniemeyer commented 1 year ago

@whedon add @kls2177 as reviewer

whedon commented 1 year ago

OK, @kls2177 is now a reviewer

jrising commented 1 year ago

@kls2177 Thank you so much for stepping up, and to @kyleniemeyer for helping get us reviewers!

kls2177 commented 1 year ago

Just getting to this now. Sorry for the delay. @kyleniemeyer, can you create a checklist for me? My plan is to complete my review by the end of the week. Thanks!

kyleniemeyer commented 1 year ago

@kls2177 I've updated the top of issue header to include a checklist for you. Thank you!

kls2177 commented 1 year ago

Hello @kyleniemeyer, I apologize for the delay with my review.

The tutorial, "ClimateEstimate.net - A practical guide to climate econometrics", is a valuable contribution to the econometrics community. Thank you to the authors - I learned a lot while reading it and feel that most relevant topics are covered. I was able to work through the most of the Hands-On Exercises, but had some trouble in Step 3 (see the links to my issues below) and consequently, Step 4.

I feel that this tutorial is suitable for publication after some revision. I recommend that there be more examples and/or activities in order to make it a more valuable learning tool. I also agree with the previous reviewer that more visuals would help students better engage with certain concepts and would also help to break up the text. Also, as a python-user, given that python is used quite a bit in the tutorial, I feel that providing python code examples for ALL the steps would be very helpful. I am not that familiar with R.

Finally, although the tutorial seems relatively complete for someone who already has a background in econometrics (this is key!), I feel that the guide would benefit from integrating the Hands-On Exercises into the narrative of the content to a greater extent. Let the Exercises tell the story. This is just a suggestion.

Major Comments:

The first section of Chapter 1 is much too long and needs to be broken up
The narrative around the Hands-On Exercises need to be spelled out in Step 1. Otherwise, Step 1 just seems like a "make-work" exercise. Let students know that the exercises are scaffolded and have a particular end goal in mind.
The first Section of Chapter 2 is mostly a list of variables and their descriptions, most of which could be put in an appendix (or this section needs to be revised)
Hands-On Exercise, Step 2 does not align well with the Chapter content
I had problems with Hands-On Exercise, Step 3 (see below)
Hands-On Exercise, Step 4 could benefit from exploring sensitivity to reduced form specification, FEs, etc.

Here are my detailed comments:

Introduction Chapter 1 Chapter 2 Chapter 3 Chapter 4

Thank you very much to the authors for your patience! I would be happy to review a revised version of the tutorial.

kyleniemeyer commented 1 year ago

Thank you @kls2177!

jrising commented 1 year ago

Thank you @kls2177 for the extensive and detailed comments! These will be very useful for improving the tutorial. We will work through them over the course of the next few weeks, and we will respond first within each github Issue.

jrising commented 1 year ago

@kls2177 We have now made changes in response to all of the items you identified. Thank you again for all of your time on the tutorial-- we believe that it has improved significantly with your suggestions.

Responding to your major comments here:

The first section of chapter 1 is now split into more manageable chunks.
We have now added an introduction to the exercise, at the beginning of step 1. This lays out the goal, and some important considerations that will inform our decisions throughout the exercise.
Section 2 has been split up to highlight the most important variables, putting the rest after the main content.
We appreciate the challenge creating a sensible introduction across the chapters while also including relevant steps of the exercise. The main element "work" of step 2 of the exercise is actually the first part of that section, "Thinking about the data-generating process", which matches the goal of chapter 2. However, since this is the point where we think about the dependent variable, it makes sense to prepare that data. Unfortunately, this auxiliary work ends up dominating the length of step 2. However, we think it is still useful to break it up, rather than include this in step 3.
Thank you so much for letting us know about the problems with step 3 and 4. We have now fixed bugs in the underlying library and provided an alternative implementation using a recently-developed R package.
We have now added a discussion of the main considerations that go into our regression (and what we miss), and provide a list of ideas for robustness checks.

More information is available in the issues. If anything is unclear, do not hesitate to let us know!

kls2177 commented 1 year ago

Thanks very much @jrising for responding to my comments. I am just ramping up with the first week of classes, so I will likely not be able to take another look over everything for a week or two. Thanks for your patience.

atrisovic commented 9 months ago

Hi @kls2177, I hope all is well! It seems we are almost there with the review. I wonder if you had a chance to give it another look and if you could send us an update. Thank you!

kls2177 commented 9 months ago

Thanks for checking-in. I apologize for the delay. I will take a look tomorrow afternoon.

Karen

On Mon, Jan 29, 2024 at 11:42 AM Ana Trisovic @.***> wrote:

Hi @kls2177 https://github.com/kls2177, I hope all is well! It seems we are almost there with the review. I wonder if you had a chance to give it another look and if you could send us an update. Thank you!

— Reply to this email directly, view it on GitHub https://github.com/openjournals/jose-reviews/issues/90#issuecomment-1915115332, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADKDR7VJVDZG2IYBHKKPMHDYQ7GPDAVCNFSM4RV7NFVKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOJRGUYTCNJTGMZA . You are receiving this because you were mentioned.Message ID: @.***>

-- ~Please help keep this email paper free

kls2177 commented 9 months ago

Hi @atrisovic, @jrising, I have added some new comments. Overall, I think the updates have addressed most of my questions.

However, I am still have technical problems with the python code. I have suggested some edits, but I am still stuck. I am getting a RunTimeError in Step 3 and I think this error is causing issues in Step 4.

Please see my comments for Step 4 here

and Step 3 here

I have also added new comments to my other issues for the other section here, here and here

jrising commented 9 months ago

@kls2177 Thank you so much for the comments and going through the example code again! It's so strange that you are still getting errors-- we will try with the most recent versions and offer datasets so people can try out later steps even if they run into problems.

We will get back to you pretty quickly on this.

kls2177 commented 8 months ago

@kyleniemeyer, @jrising, @atrisovic, I have gone through the revisions and tested the code and all is working well. I recommend this resource for publication in JOSE after all the revisions are complete.

Thanks for your patience!

atrisovic commented 8 months ago

Hi @kls2177 wonderful news! Thank you very much for your help!

A question for my co-authors: what do you think about removing .net from the title in case we want to move the domain in the next few years? I opened an issue where we could quickly revisit it.

jrising commented 7 months ago

@kls2177 We have now responded to all of your incredibly helpful comments. Please let us know if you have any remaining concerns or what the next step is.

kls2177 commented 7 months ago

Hi @kyleniemeyer, @jrising, @atrisovic: I recommend "accept".

In terms of changing the paper title, I like this idea and I like your second option as well, but I will let @kyleniemeyer make a final decision on this aspect of the process.

kyleniemeyer commented 7 months ago

Thanks @kls2177!

@jrising @atrisovic I think making that minor change to the title ("ClimateEstimate.net" -> "ClimateEstimate") seems reasonable. Please go ahead, and then I'll do a final read-through of the paper.

jrising commented 7 months ago

@kyleniemeyer We realized that we actually already did this, shortly after the initial submission. The paper title is "A practical guide to climate econometrics: Navigating key decision points in weather and climate data analysis", which is perfect. It's ready for you!

kyleniemeyer commented 7 months ago

@editorialbot generate pdf

editorialbot commented 7 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

kyleniemeyer commented 7 months ago

@jrising at this point, please make an archive of the materials and archive it (e.g., on Zenodo) and report the DOI back here.

atrisovic commented 7 months ago

Hi @kyleniemeyer I just created a new release for JOSE with the DOI https://doi.org/10.5281/zenodo.10833818 =)

kyleniemeyer commented 7 months ago

Thanks @atrisovic! Can you make sure the title and authors match the paper?

kyleniemeyer commented 7 months ago

(@atrisovic also: this is minor, but it looks like the GitHub repo name doesn't match?)

atrisovic commented 7 months ago

Hi @kyleniemeyer, sounds good! I updated the title name, the order of authors, and added the repository URL on Zenodo. Is that the GitHub repo name you are referring to?

kyleniemeyer commented 7 months ago

@editorialbot set 10.5281/zenodo.10833818 as archive

editorialbot commented 7 months ago

Done! archive is now 10.5281/zenodo.10833818

kyleniemeyer commented 7 months ago

@atrisovic yeah, it was the GitHub repo name that looked off. Perhaps that is older? It doesn't matter, just caught my attention.

kyleniemeyer commented 7 months ago

@editorialbot recommend-accept

editorialbot commented 7 months ago

Attempting dry run of processing paper acceptance...

editorialbot commented 7 months ago

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1126/science.aad9837 is OK
- 10.3386/w25189 is OK
- 10.3386/w22181 is OK
- 10.1088/1748-9326/ab281e is OK
- 10.1002/wcc.579 is OK

MISSING DOIs

- None

INVALID DOIs

- None

editorialbot commented 7 months ago

:wave: @openjournals/jose-eics, this paper is ready to be accepted and published.

Check final proof :point_right::page_facing_up: Download article

If the paper PDF and the deposit XML files look good in https://github.com/openjournals/jose-papers/pull/141, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

kyleniemeyer commented 7 months ago

@jrising @atrisovic this is now in the hands of @openjournals/jose-eics to do the final checks and then accept. I'm sorry this took so long to complete, but we are at the finish line now!

jrising commented 7 months ago

@kyleniemeyer We're just happy for the positive result! Thank you for helping move this along.

atrisovic commented 7 months ago

Thank you so much @kyleniemeyer 🙏

atrisovic commented 6 months ago

Hi @openjournals/jose-eics, I hope all is well! This is a gentle reminder, we are now in the final stage of the review process.

labarba commented 5 months ago

I see that the version number is v2.0 in the repository, but we see v1.2 at the top of this issue. As I don't see a version command was run here, I will run it now.

labarba commented 5 months ago

@editorialbot set v2.0 as version

editorialbot commented 5 months ago

Done! version is now v2.0

labarba commented 5 months ago

I found a broken link to the JupyterBook landing pages, and in fixing it, added a couple of tiny edits. https://github.com/atrisovic/weather-panel.github.io/pull/93

labarba commented 5 months ago

I notice that the LICENSE on the repository is CC-BY-SA. The footer of the JupyterBook, however, has a copyright symbol and year only. Would you edit the _config.yml to include the license information in the footer?

labarba commented 5 months ago

PR here: https://github.com/atrisovic/weather-panel.github.io/pull/94

atrisovic commented 5 months ago

Both PRs merged! Thank you very much!

labarba commented 5 months ago

@editorialbot accept

editorialbot commented 5 months ago

Doing it live! Attempting automated processing of paper acceptance...

editorialbot commented 5 months ago

Ensure proper citation by uploading a plain text CITATION.cff file to the default branch of your repository.

If using GitHub, a Cite this repository menu will appear in the About section, containing both APA and BibTeX formats. When exported to Zotero using a browser plugin, Zotero will automatically create an entry using the information contained in the .cff file.

You can copy the contents for your CITATION.cff file here:

CITATION.cff

``` cff-version: "1.2.0" authors: - family-names: Rising given-names: James A. orcid: "https://orcid.org/0000-0001-8514-4748" - family-names: Hussain given-names: Azhar orcid: "https://orcid.org/0000-0002-6475-1052" - family-names: Schwarzwald given-names: Kevin orcid: "https://orcid.org/0000-0001-8309-7124" - family-names: Trisovic given-names: Ana orcid: "https://orcid.org/0000-0003-1991-0533" doi: 10.5281/zenodo.10833818 message: If you use this software, please cite our article in the Journal of Open Source Software. preferred-citation: authors: - family-names: Rising given-names: James A. orcid: "https://orcid.org/0000-0001-8514-4748" - family-names: Hussain given-names: Azhar orcid: "https://orcid.org/0000-0002-6475-1052" - family-names: Schwarzwald given-names: Kevin orcid: "https://orcid.org/0000-0001-8309-7124" - family-names: Trisovic given-names: Ana orcid: "https://orcid.org/0000-0003-1991-0533" date-published: 2024-05-23 doi: 10.21105/jose.00090 issn: 2577-3569 issue: 75 journal: Journal of Open Source Education publisher: name: Open Journals start: 90 title: "A practical guide to climate econometrics: Navigating key decision points in weather and climate data analysis" type: article url: "https://jose.theoj.org/papers/10.21105/jose.00090" volume: 7 title: "A practical guide to climate econometrics: Navigating key decision points in weather and climate data analysis" ```

If the repository is not hosted on GitHub, a .cff file can still be uploaded to set your preferred citation. Users will be able to manually copy and paste the citation.

Find more information on .cff files here and here.

openjournals / jose-reviews