search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
725 stars 38 forks source link

[REVIEW]: Comet Time Series (CometTS) Visualizer #1047

Closed whedon closed 5 years ago

whedon commented 6 years ago

Submitting author: @jshermeyer (Jacob Shermeyer) Repository: https://github.com/CosmiQ/CometTS Version: <v1.2.0> Editor: @Kevin-Mattheus-Moerman Reviewers: @zhampel, @rmsare, @jjmcnelis Archive: 10.5281/zenodo.3418091

Status

status

Status badge code:

HTML: <a href="http://joss.theoj.org/papers/2e38056740935063b95551d23334a1c1"><img src="http://joss.theoj.org/papers/2e38056740935063b95551d23334a1c1/status.svg"></a>
Markdown: [![status](http://joss.theoj.org/papers/2e38056740935063b95551d23334a1c1/status.svg)](http://joss.theoj.org/papers/2e38056740935063b95551d23334a1c1)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@zhampel, and @rmsare please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.theoj.org/about#reviewer_guidelines. Any questions/concerns please let @Kevin-Mattheus-Moerman know.

✨ Please try and complete your review in the next two weeks ✨

Review checklist for @zhampel

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @rmsare

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @jjmcnelis

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

danielskatz commented 5 years ago

And @chrismattmann replied to me by email that he doesn't have time currently

danielskatz commented 5 years ago

@turmon, @darth-pr - would either of you be able to step in and help with a review of this submission & this software for JOSS? (Or perhaps suggest someone else who might?)

See JOSS review criteria in case you are not aware of how JOSS works...

turmon commented 5 years ago

Maybe Kyo Lee in my group? He's a climate/remote sensing model analysis person, who contributes to an open source regional climate model system.

https://orcid.org/0000-0003-3754-3204

Huikyo.Lee@jpl.nasa.gov

Mike

On May 20, 2019, at 6:24 PM, Daniel S. Katz notifications@github.com wrote:

@turmon, @darth-pr - would either of you be able to step in and help with a review of this submission & this software for JOSS? (Or perhaps suggest someone else who might?)

See JOSS review criteria in case you are not aware of how JOSS works...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

danielskatz commented 5 years ago

Thanks Mike - Does Kyo has a github account? I'll let @Kevin-Mattheus-Moerman follow-up with him, as the editor.

turmon commented 5 years ago

I was not able to find a GH account, but I can't rule it out. Mike

On May 21, 2019, at 8:44 AM, Daniel S. Katz notifications@github.com wrote:

Thanks Mike - Does Kyo has a github account? I'll let @Kevin-Mattheus-Moerman follow-up with him, as the editor.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Kevin-Mattheus-Moerman commented 5 years ago

:wave: @lubiavinhas @mloning @TonyBagnall @gilbertocamara @rolfsimoes @pedro-andrade-inpe @ammaciel @jgrss @slandersson @mmann1123 @avanetten @aftimg would you be interested in reviewing this submission (about time-series analysis of satellite image data) for the Journal of Open Source Software (JOSS)? The review process, which takes place in this GitHub issue, focuses largely on the software and a short paper. Let me know if you have questions.

Kevin-Mattheus-Moerman commented 5 years ago

@whedon remove @lewismc as reviewer

whedon commented 5 years ago

OK, @lewismc is no longer a reviewer

Kevin-Mattheus-Moerman commented 5 years ago

:wave: @ceholden @PontusOlofsson @ZachTRice @dgketchum @rmsare would you be interested in reviewing this submission (about time-series analysis of satellite image data) for the Journal of Open Source Software (JOSS)? The review process, which takes place in this GitHub issue, focuses largely on the software and a short paper. Let me know if you have questions.

usethedata commented 5 years ago

@whedon generate pdf

whedon commented 5 years ago 
Attempting PDF compilation. Reticulating splines etc...
whedon commented 5 years ago

:point_right: Check article proof :page_facing_up: :point_left:

rmsare commented 5 years ago

@Kevin-Mattheus-Moerman I can complete this review within the next 2 weeks. Is that okay?

Kevin-Mattheus-Moerman commented 5 years ago

@rmsare yes thanks!

Kevin-Mattheus-Moerman commented 5 years ago

@whedon add @rmsare as reviewer

whedon commented 5 years ago

OK, @rmsare is now a reviewer

TonyBagnall commented 5 years ago

hi, don't have time or expertise to review this, sorry.

Kevin-Mattheus-Moerman commented 5 years ago

@rmsare thanks for your help. I've created a set of tick boxes at the top of this issue which will guide you through the review process. Please also check out our review guidelines. Let me know if you have any questions.

Kevin-Mattheus-Moerman commented 5 years ago

@TonyBagnall okay, thanks for your reply.

usethedata commented 5 years ago

@whedon add @jjmcnelis as reviewer

whedon commented 5 years ago

OK, @jjmcnelis is now a reviewer

Kevin-Mattheus-Moerman commented 5 years ago

@jjmcnelis thanks for joining as reviewer. Would you be able to update your GitHub profile e.g. such that it lists your affiliation (and/or a link ORCID profile if possible). Thanks. Let me know if you have questions.

rmsare commented 5 years ago

@jshermeyer I am reviewing your submission, and I had a question about your changes to the project.

I would like to suggest some reorganization, but it looks like a bit of that has been implemented in @zhampel's fork of CometTS. Do you plan to incorporate any of these changes, e.g. to directory structure?

Thanks for clarifying - it will help us to provide more manageable reviews.

rmsare commented 5 years ago

Thanks for submitting this, @jshermeyer! Here is my initial review (based on the master branch of CosmiQ/CometTS).

Overall, it looks like a useful set of tools for quick time series analysis of satellite imagery that is appropriate for JOSS. Nice work.

While the code functions as advertised, the codebase and documentation could be improved with substantial changes that will make it easier to use. I believe implementing many of these changes will ensure that CometTS is a usable, extensible software package rather than an analysis utility.

I was able to install in a conda environment with PyPI.

Documentation

The code is well-commented, but function-level documentation is not quite sufficient (see review criteria). I also think the README could be streamlined and the example notebooks improved with some instructions. Someone interested in CometTS should be able to scan the example notebooks and documentation to quickly see how things work.

Once you add docstrings you might consider auto-generating API documentation with Sphinx and ReadtheDocs (file an issue with the docs tag?).

Functionality

Paper

The manuscript is written well, but describes the CometTS workflow rather than making a case for the research software contribution. Consider adding two or three sentences to introduction providing more context on satellite timeseries analysis and why CometTS is a good alternative to other open-source offerings.

Many of the details in the final paragraph could be summarized in a shorter sentence like “CometTS output includes user-specified statistics such as mean, median, and quartiles, and the package offers masking functionality to remove clouds and snow from the area of interest.” Then the remaining space could outline key research use cases in more detail.

The statement of need should state the intended audience, which appears to include social scientists, earth scientists, and/or geospatial intelligence analysts.

Minor issues

Recommendation: Could be accepted after moderate enhancements

Kevin-Mattheus-Moerman commented 5 years ago

@rmsare thanks for this very thorough review!

Kevin-Mattheus-Moerman commented 5 years ago

@jshermeyer can you comment on the points raised by @rmsare. Thanks.

jjmcnelis commented 5 years ago

@jjmcnelis thanks for joining as reviewer. Would you be able to update your GitHub profile e.g. such that it lists your affiliation (and/or a link ORCID profile if possible). Thanks. Let me know if you have questions.

@Kevin-Mattheus-Moerman Happy to help. ORCID and Org are now listed on my profile. I'll complete my initial review before the business week opens on Monday. Please let me know if that doesn't agree with your expected schedule for reviewers and I'll up the urgency.

jshermeyer commented 5 years ago

Thanks @rmsare for the thorough review. Good recommendations that I'm going to gradually start addressing. I don't have as much time for this repo as I used to as this project was more of a focus about a year ago. Will see how it goes, may take some time.

-Jake

jjmcnelis commented 5 years ago

@jshermeyer I enjoyed working through your example notebooks. I had a few hangups but nothing major. Please give me a day or so to open issues with specific recommendations. I will update this comment with specifics.

Key recommendations:

  1. Acknowledge in the documentation whether CometTS is a. a workflow for processing/visualizing raster time series, or b. tools for workflows; i.e. it's unclear how many of the functions are reliant on outputs from other pieces.
  2. Acknowledge the key features that distinguish CometTS from other processing/visualization tools for raster time series, e.g. TIMESAT.

@Kevin-Mattheus-Moerman Is 1a is compatible with JOSS submission requirements?

General

Functionality

Documentation

Paper

My feelings about the content of the paper are pretty well covered above. No problems with the grammar/writing style but I can send a marked up copy if you need that level of detail.

Kevin-Mattheus-Moerman commented 5 years ago

@jshermeyer apologies this review has taken so long. We initially had great difficulty finding suitable (and suitably active) reviewers.

Since you've indicated that addressing the points raised may take a while, I've paused this review for now. We can resume it again at any point, when you are ready.

Would you be able to give me your best (and most realistic) estimate as to when you think you'll have addressed the points raised?

Let me know if you think you will not be able to implement these changes. If that is the case we could instead retract this submission (and resubmision in the future s also an option).

Thanks.

Kevin-Mattheus-Moerman commented 5 years ago

@zhampel, @rmsare, @jjmcnelis I've paused this review for now since @jshermeyer has indicated that addressing the issue raised may take a significant amount of time. @jjmcnelis thanks also for your review comments. You mentioned you might be able to open specific issues related to your comments. Please do, so @jshermeyer can work on these too.

Once @jshermeyer indicates they are ready to resume the review I'll remove the paused label and will call upon the reviewers again.

jshermeyer commented 5 years ago

Thanks @jjmcnelis for another strong review. I think this process will make a much stronger toolset when it's all said and done.

I've update the docstrings which I think was the most cumbersome thing to add. Next on my list is documenting the notebooks a bit better. After that's adjusting the readme and responding to some of the more minor comments as proposed by reviewers. Seems like it could be done in a day or two if I can steal a few hours.

My timing on this is quite ambiguous and my availability to work on this will decline rapidly starting next week. If I can get to it over the next 2 weeks I have a feeling I could get it done by then. Otherwise this may bleed for awhile.

@Kevin-Mattheus-Moerman Am I correct in assuming no further reviewers will be added? I'd hope to address everything in one shot and then move forward to a decision.

Kevin-Mattheus-Moerman commented 5 years ago

@jshermeyer thanks for the update. Correct, no more reviewers will be added (I had to recruit new/more reviewers halfway through the review process because one was no longer responsive and one indicated a (minor) conflict of interest). So yes it looks like you are looking at the full set of changes/comments now. Thanks for working on this now. Looking forward to the next updates.

jshermeyer commented 5 years ago

@Kevin-Mattheus-Moerman, @zhampel, @rmsare, @jjmcnelis Thanks for the reviews and constructive feedback. I've worked to incorporate all of your suggestions throughout this process into the repository in one way or another. As I stated previously I think the process has been worthwhile and I’m appreciative for your time to make CometTS a better and more usable package.

What I've worked on Specific items that I've changed in the code, readme, or paper are all tracked in the issues: https://github.com/CosmiQ/CometTS/issues

Specifically issue 15 tracks the minor comments from all reviewers and aggregates them in one location.

Specific responses A few things throughout the review likely require a direct response from me, I’ll run through them here.

  1. Acknowledge in the documentation whether CometTS is a. a workflow for processing/visualizing raster time series, or b. tools for workflows; i.e. it's unclear how many of the functions are reliant on outputs from other pieces.

I think CometTS is both a package with unique functionality, a workflow, and provides tools for workflows for processing and visualizing raster time series. So all of the above.

  1. Acknowledge the key features that distinguish CometTS from other processing/visualization tools for raster time series, e.g. TIMESAT.

Key differences are: 1) the ability to handle arbitrarily sized areas of interest (polygons) vs. individual pixels for visualization.
2) Anomaly detection and auto-regressive integrated moving average forecasting 3) Python / jupyter notebook deployment

Are raster formats other than GeoTIFF supported? I don't see any reference to other formats than GeoTIFF. Display a list of supported formats prominently on the README. Support for other GDAL-supported formats will be straightforward to implement if you choose to refactor as an API as zhampel and rmsare recommend. The metadata stored internally by HDF and others eliminates the need for a config file like the one generated by CSV_lt.py. • Test and indicate in the README which imagery sources and formats are compatible with CometTS. The VIIRS example is cool, but I think it represents an edge case for most VIIRS users. VIIRS is distributed in HDF for most standard products, not GeoTIFF like in the example. The documentation suggests that users of "overhead imagery" are the intended audience, but many of the relevant datasets are distributed in other formats.

Have updated the readme to address this. Any GDAL format raster is supported. Any OGR format vector file is supported. All metadata unfortunately is not created equally and no standard exists for what is contained in each file (although one is being worked on). I think an ad-hoc approach to extracting at least the date information from each raster file is still the best method and lends flexibility to an end user. CometTS will auto-extract extent and projection if it is appropriately defined in the rasters. HDF has no standard metadata format, and is really a poor choice by NOAA to use. In fact, some of their recent data has no projection or extent information embedded in each file. When you don’t have that from the start, you’re setting people up to fail. If you have experience getting these data to play nicely out of the box with a GIS, I’d love to hear about it.

• Provide examples for more data sources. It's not clear how transferable the VIIRS workflow example is to other data sources.

This I really can’t do. I’ve built the package, now it’s up to end users to run with it, and get creative with their scientific pursuits. I’ve written a full paper regarding tracking electrification rates in Puerto Rico following Hurricane Maria using CometTS (now cited in the paper). Additionally, I’ve shown separate examples with Landsat (transferable to any multi-spectral imagery with some tweaking) and using VIIRS for estimating seasonal population migrations.

Future Things If/when the package is approved for JOSS, I intend to update the zenodo code as well as the code on pypi/conda. Want to make sure there are no more modifications or recommendations before I do this.

rmsare commented 5 years ago

@jshermeyer Thanks for your response. I will be able to look through your changes in more detail this weekend.

@Kevin-Mattheus-Moerman, I will add a comment here when that's done and follow up in review issues where appropriate.

rmsare commented 5 years ago

@Kevin-Mattheus-Moerman Looking through the changes, my major comments have been addressed by @jshermeyer and I think the notebooks and code have been made much more accessible.

Right now, I feel like the code and packaging meets the standards of JOSS, but it is up to the author to justify CometTS as a novel research tool. As @jjmcnelis asked above, could editors clarify when a workflow-oriented tool like this submission is defined as a "minor utility"?

The paper should be edited a bit for clarity (#16). In particular, it claims CometTS provides "new" functionality (averaging, cloud masking, etc.) that is available in other open source software like SITS.

Kevin-Mattheus-Moerman commented 5 years ago

@jjmcnelis can you review the latest changes and @jshermeyer 's response to your points.

@jjmcnelis @rmsare To answer your question yes this package is considered more than a "minor utility", and it is clear it may be valuable for scientific research applications. A typical "minor utility" is a package doing something quite trivial or simple and which does not influence scientific research and the findings much or at all. Therefore one would likely not feel the need to cite such a minor utility. In this case however the package does influence the science as it provides both visualization and processing tools and workflows.

Kevin-Mattheus-Moerman commented 5 years ago

@zhampel can you finalize the review on your end? Are you satisfied with the changes made? Thanks

Kevin-Mattheus-Moerman commented 5 years ago

@rmsare @jjmcnelis you can tick the version box (this tick box will be removed in future review issues as it often leads to confusion), the reviewed version will be set appropriately before the acceptance stage.

jjmcnelis commented 5 years ago

@Kevin-Mattheus-Moerman Apologies for the radio silence over the last week.

@jshermeyer I'm satisfied with the improvements to the repo. My main critiques about the README content have all been addressed:

  1. Outlined basic steps for the primary use case (check - displayed prominently atop README.)
  2. Supported formats (check - would've been fine with "All GDAL/OGR compatible formats are supported.", but it's nice to be reminded that I need to update my GDAL bookmarks :)
  3. Replacing the details in the README with the three (?) additional time series examples better communicates your intent in developing this software. And the plots are fantastic. Clear and information rich, with concise descriptions of the phenomena captured by the data and depicted through CometTS.

Your updates to the paper help to clarify CometTS's place among the other tools that provide some similar functionality.

I apologize for not giving you more specific feedback during my initial review. Several commitments came calling at once. @rmsare Thanks for your diligence in documenting the review comments in the issues on CometTS repo!

Kevin-Mattheus-Moerman commented 5 years ago

@zhampel can you finalize the review on your end? Are you satisfied with the changes made? Thanks

zhampel commented 5 years ago

@Kevin-Mattheus-Moerman I see that the author has addressed my earlier comments as well as those of the other reviewers. The documentation has been improved, and I think users will appreciate the addition of sample data for testing purposes. The final two boxes have been checked. Thanks.

Kevin-Mattheus-Moerman commented 5 years ago

@zhampel thanks a lot!

Kevin-Mattheus-Moerman commented 5 years ago

@jjmcnelis thanks for the update too

Kevin-Mattheus-Moerman commented 5 years ago

@jshermeyer @rmsare all boxes are ticked. However is it correct to say some work remains related to https://github.com/CosmiQ/CometTS/issues/16 raised by @rmsare?

Kevin-Mattheus-Moerman commented 5 years ago

@whedon generate pdf

whedon commented 5 years ago 
Attempting PDF compilation. Reticulating splines etc...
whedon commented 5 years ago

:point_right: Check article proof :page_facing_up: :point_left:

rmsare commented 5 years ago

@Kevin-Mattheus-Moerman @jshermeyer Yes, I think the claims in the paper need to be revised a bit to describe what's new in CometTS, and some small edits remain. Otherwise the package is in good shape, and I'd be happy to recommend an updated submission for acceptance.

Kevin-Mattheus-Moerman commented 5 years ago

@jshermeyer can you give an update on progress in terms of replying/dealing with @rmsare comments? It looks like we are nearly there.