search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
725 stars 38 forks source link

[REVIEW]: WhaleMap: a tool to collate and display whale survey results in near real-time #3094

Closed whedon closed 3 years ago

whedon commented 3 years ago

Submitting author: @hansenjohnson (Hansen Johnson) Repository: https://github.com/hansenjohnson/WhaleMap Version: v1.0.0 Editor: @KristinaRiemer Reviewer: @pjbouchet, @mcsiple Archive: 10.5281/zenodo.4913270

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/60546bbc8ffdf18e856c8c269f846a2e"><img src="https://joss.theoj.org/papers/60546bbc8ffdf18e856c8c269f846a2e/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/60546bbc8ffdf18e856c8c269f846a2e/status.svg)](https://joss.theoj.org/papers/60546bbc8ffdf18e856c8c269f846a2e)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@pjbouchet & @mcsiple, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @KristinaRiemer know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @pjbouchet

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @mcsiple

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @pjbouchet, @mcsiple it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1121/10.0001811 is OK
- 10.1111/csp2.267 is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 3 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.18 s (646.0 files/s, 79319.5 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
R                               99           3379           3328           5961
make                             1             97            105            446
Bourne Shell                    12             84             93            240
Markdown                         3             76              0            229
Rmd                              1             71             89            144
TeX                              1              1              0             22
-------------------------------------------------------------------------------
SUM:                           117           3708           3615           7042
-------------------------------------------------------------------------------

Statistical information for the repository '45d5b31b85593a03c45255fb' was
gathered on 2021/03/09.
No commited files with the specified extensions were found.
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

whedon commented 3 years ago

:wave: @pjbouchet, please update us on how your review is going (this is an automated reminder).

whedon commented 3 years ago

:wave: @mcsiple, please update us on how your review is going (this is an automated reminder).

KristinaRiemer commented 3 years ago

@Kevin-Mattheus-Moerman could you re-send @pjbouchet the reviewer invitation? (Or is that something that I'm able to do?)

Kevin-Mattheus-Moerman commented 3 years ago

@KristinaRiemer I think you can, but I can help if it doesn't work. I think you can run: @whedon re-invite @pjbouchet as reviewer

KristinaRiemer commented 3 years ago

@whedon re-invite @pjbouchet as reviewer

whedon commented 3 years ago

OK, the reviewer has been re-invited.

@pjbouchet please accept the invite by clicking this link: https://github.com/openjournals/joss-reviews/invitations

KristinaRiemer commented 3 years ago

Looks like it worked, thanks @Kevin-Mattheus-Moerman! So @pjbouchet, you should be able to accept there.

pjbouchet commented 3 years ago

Found a typo in the README file, where src/get_get_remote_data.sh should be src/get_remote_data.sh

pjbouchet commented 3 years ago

WhaleMap is cited/mentioned in more papers than currently reported in the reference list, though some authors only provide the app URL rather than include an actual citation which could be more easily tracked. You may want to add the following to your list of references:

pjbouchet commented 3 years ago

I don't have any specific comments, other than the above. The app is well designed and works well when I tested it online. A couple of thoughts and suggestions for future improvements:

pjbouchet commented 3 years ago

You may want to add a sentence to the paper indicating the lack of equivalent software in existence.

hansenjohnson commented 3 years ago

Thank you very much for your detailed feedback, @pjbouchet, it is much appreciated! @KristinaRiemer, is it ok for me to start addressing the suggested changes, or should I wait until all reviews are complete?

KristinaRiemer commented 3 years ago

Hi @hansenjohnson, you can start addressing comments from @pjbouchet right away!

hansenjohnson commented 3 years ago

Great! Thanks for the clarification @KristinaRiemer, I will start addressing these suggestions shortly.

hansenjohnson commented 3 years ago

Hi @pjbouchet, thanks again for your helpful comments! I have taken the following steps to address them:

Found a typo in the README file, where src/get_get_remote_data.sh should be src/get_remote_data.sh

Fixed

WhaleMap is cited/mentioned in more papers than currently reported in the reference list, though some authors only provide the app URL rather than include an actual citation which could be more easily tracked. You may want to add the following to your list of references:

  • Koubrak et al. 2021. Saving the North Atlantic right whale in a changing ocean: Gauging scientific and law and policy responses.
  • Gervaise et al. 2021. Optimizing passive acoustic systems for marine mammal detection and localization: Application to real-time monitoring north Atlantic right whales in Gulf of St. Lawrence.
  • Baumgartner et al. 2020. Slocum Gliders Provide Accurate Near Real-Time Estimates of Baleen Whale Presence From Human-Reviewed Passive Acoustic Detection Information.

Added

I don't have any specific comments, other than the above. The app is well designed and works well when I tested it online. A couple of thoughts and suggestions for future improvements:

  • I wonder about the sustainability of the workflow given the need to have custom scripts written by the curator for each survey team. In my experience, "allowing any team to submit data in nearly any format" (as stated in the article) is a recipe for headaches. What happens if the curator is suddenly unavailable, changes jobs etc.? Are there systems in place to ensure that the running of the app remains unaffected? I would argue that encouraging data standardisation during collection would be worthwhile.

This is an excellent point. Our choice to accept data in nearly any format was one of necessity. As stated in the manuscript: "[this method] is essential for rapid data collection, as survey teams in the field typically lack the time and resources to coerce their data into a common format.". In our experience the only way to get these data quickly was to accept them in any way, shape or form. Fortunately we are converging on a relatively small number of different data formats that remain stable from one year to the next. Where possible we recommend the use of a common format (e.g., simple excel table for sightings and a standard gpx file for tracklines) when first incorporating data from a new group. We are also taking several steps to seek long term support for the system and its curator(s).

  • Are you thinking / have you thought about adding environmental layers to the map (and the possibility to query environmental data at the locations where whales were detected)? This would be interesting and could support spatial modelling efforts to better understand how whale distribution is affected by climate change etc.

Great idea! There are several efforts underway that are interested in pursuing this kind of work (e.g., combining WhaleMap detections and operational ocean model output to forecast whale distribution). We are often approached about adding these and other layers to WhaleMap. In general we are very cautious about adding layers out of concern that they would clutter the display, overwhelm general users, and distract from the core goal of WhaleMap (display Whale survey results). An alternative that we are pursuing is to provide the WhaleMap data feed to other groups (with permission) who could then build a similar viewer and add layers that are specific to their project.

  • It would be nice to have the map automatically zoom in to the extent of the layers currently being shown (but that's a very minor suggestion).

We have pursued this in the past. It is very nice in some instances, but becomes jarring and undesirable when the user wishes to keep the map focused on a specific region while they adjust the data filters. This is a very common use case, so I think we will keep the current behaviour.

  • The article doesn't mention data accessibility - I imagine that there are various hurdles there with confidentiality and the fact that this is an endangered species. A statement about this may be worthwhile in the text though.

This is an important point. The last bullet in the Statement of Need reads: "[Critically, WhaleMap does not:] Allow access to raw or processed data without approval from the data originator". We feel this adequately covers the issue of data accessibility, but can elaborate if necessary.

You may want to add a sentence to the paper indicating the lack of equivalent software in existence.

Added

Please let me know if anything needs clarification. Many thanks again for your help, it has certainly improved the manuscript!

pjbouchet commented 3 years ago

This is great - thanks for your thorough responses @hansenjohnson. I have no further queries and am happy to recommend this paper for publication. Congratulations on a great piece of work!

KristinaRiemer commented 3 years ago

Great, thanks for the review @pjbouchet! @mcsiple let me know if you have any questions or need any help with starting your review.

mcsiple commented 3 years ago

Sorry for the delay in my comments; I think I wrote the date down incorrectly. I am working on it and will have all my comments done by this Friday (4/30/21).

mcsiple commented 3 years ago

Thank you all for your patience-- below are my review notes.

WhaleMap review

This is a great tool and both the code and ui are simple and clean. An excellent use of leaflet for displaying several data layers. I can see how it is already a useful tool for survey teams, but could also be great for teaching about surveys for cetacean abundance.

Substantial scholarly effort

I know that the app is not intended to provide raw data outputs without approval from the data originators. Technically the app would be more likely to be used by scientists like myself if there were some way to either:

1) Direct the user to the pathway for getting raw data if they need it. For example, including contact information where people might dig further (as I'm sure data exploration in this tool will lead to research questions). Or

2) Include, if possible, an option for the app to show summaries of sightings data at a coarse enough grain that it would be permissible by the data sources.

These are not deficiencies in the app but would lead to more citations of the tool. I do note that it has been cited by other papers before (@pjbouchet caught the full list) so will leave this detail to the editor to decide.

Functionality

The app works great and is quite fast. I really only have minor comments on it.

server comments:

I am impressed with how clean and polished both the server and ui are. Well done; the code is very easy to read and interpret. I had some of the same workflow concerns as @pjbouchet but I think you've addressed them!

ui comments:

Documentation:

@KristinaRiemer, most of my comments in this section are just editor discretion. There are a few listed things on JOSS's checklist that are absent but I think there is good reason those absences.

General comments

Software paper

General quality

My only writing suggestions are to the Statement of Need -- everything else is great. I would like to see a "use case" here.

Statement of need

hansenjohnson commented 3 years ago

Hi @mcsiple! Thank you very much for your clear and constructive comments. Please see my responses below:

Substantial scholarly effort

I know that the app is not intended to provide raw data outputs without approval from the data originators. Technically the app would be more likely to be used by scientists like myself if there were some way to either:

  1. Direct the user to the pathway for getting raw data if they need it. For example, including contact information where people might dig further (as I'm sure data exploration in this tool will lead to research questions). Or

We have considered adding contact information for each survey group, but have found that doing so ends up being less efficient for a variety of reasons. For example, it makes it more difficult for a WhaleMap user to compile a comprehensive data request, and adds to the workload of each survey team. We have found it preferable to have all requests for WhaleMap data submitted via email to me (hansen.johnson@dal.ca). That way I can work with my team to review the requests, gather additional information, identify duplication of effort, solicit the appropriate permissions, and pull the requested data.

  1. Include, if possible, an option for the app to show summaries of sightings data at a coarse enough grain that it would be permissible by the data sources.

This is an excellent suggestion. Many users currently take screenshots of the app for survey summary purposes, but in the future perhaps a more elegant solution would be to add a feature to either take a screenshot automatically (see this issue), or even capture the main features of the map, timeseries and summary table in a small summary report. I recently filed an issue with a very similar feature request. I would not consider this essential to implement prior to publication, but I leave that to the discretion of the editor.

Functionality

ui comments:

  • If this tool is used by survey teams to coordinate, do they have access to finer-grain data or raw data if they have a password? That seems like it would be useful for their purposes.

Currently, entering a password reveals unverified data such as opportunistic sightings reports or survey platforms that are still in development. It does not provide access to raw data or data at a different resolution. I work quite closely with a number of survey teams that use WhaleMap regularly. In my experience most teams use the measurement tool (at the bottom left of the map), graticules layer, management layers, and (in US waters) NOAA charts, in addition to the data filters, to plan their survey activities. None have suggested to me that access to the raw data would be helpful for survey planning purposes. I also suspect that some teams would not be comfortable allowing their raw data to be accessed by other groups without their knowledge. Any teams requiring raw data for more detailed analyses are welcome to request access.

  • blue line on upper plot (Acoustic detection events per day) is below zero-- I think this just needs to be moved up to 0. In general it seems safe to fix yaxis start at 0 across all sightings plots.

The problem with place this effort line at zero is that it becomes very difficult to see when the y axis range gets large (i.e., when displaying large numbers of observations). The only way I can think to mitigate this issue is to add conditions to change the line position depending on the numbers of observations. I would like to avoid that if possible, as I suspect it would reduce the performance of the application and potentially confuse the viewer. Let me know if I am misunderstanding your suggestion, or if there are obvious alternatives that I may be overlooking.

  • "Go!" button is small and hard to see on my monitor -- suggest moving to the sidebar

I absolutely agree. In the past I had the "Go!" button on the sidebar, but many users became frustrated because they would either forget to click "Go!" after changing the filters, or because they constantly had to scroll up and down between the map, filters, and "Go!" button. It's also important to consider multiple devices / screen sizes for this kind of change, as WhaleMap is often used via smartphone or other small mobile devices. I have made a couple changes to increase the size of the button, which I already find is a substantial improvement. I hope that resolves your issue as well.

  • might be good to move incomplete data warning (yellow box at bottom right when selected start date is earlier than first complete data year) up to sidebar too (or make larger, or fix, or both).

I have had several conversations recently that make it clear to me that these data warnings need to be made more prominent, so thank you for these suggestions. I have made a few changes including increasing the text size, changing the color (to red), and also increasing the length of time the warnings last before they expire.

  • If the "Range among years" option shows a multi-year range between the earliest and latest year in the selectizeInput() widget, maybe it is more streamlined to make this input a double-ended range slider?

I prefer the selectizeInput() because it allows the comparison of non-sequential years. For example, the selectizeInput() allows you to select data specifically from 2015 and 2017 (skipping 2016), rather than 2015-2017 (including 2016). It does make it slightly more cumbersome for users to select many years, but I think this is acceptable, particularly when the "Date range" option makes it very easy to show all data over a large time range.

  • when I changed the "choose date(s)" option to "Range among years," I got an error when I first tried it (about a week ago; "Error: Evaluation error: wrong sign in 'by' argument") - I haven't been able to get it again but if that happens with other errors I suggest using safeError() to build more informative errors into the ui. You can also use tags$head(tags$style(".shiny-output-error{color: mycolor;}")) as an argument to fluidPage() to change the color of the errors (my experience is that people panic when they get red error messages)

I am not sure what precipitated this error. Perhaps some bad data were briefly uploaded in the last few weeks as I have been bringing some new survey platforms online. I am glad it seems to be resolved now, but I will be watchful for it in the future. I like your suggestion to aspire to provide more descriptive error messaging. While these messages probably will not be relevant to most users, they would certainly help me in reproducing and debugging. I think this is a relatively minor change so have filed it as a feature request and leave the urgency of the implementation up to the editor.

  • tiny suggestion: change legend in leaflet to show points for observations instead of plain squares. Or change the grey color for 'buoy' vs 'definite visual'

Thanks a lot for this. I am slightly colorblind so occasionally struggle with color conflicts like this. I have changed the default track colour for 'buoy' to better distinguish it from the 'definite visual' detections. Note that the default colours can be adjusted further in the 'Colors' tab of the UI.

Documentation:

General comments

  • Usage examples are also NA. Would like to see a short blurb about a real-time usage example for a survey team or an educational example

Excellent idea! I have added a few sentences to the end of the "Statement of Need" that hopefully provide suitable usage examples.

Software paper

General quality

My only writing suggestions are to the Statement of Need -- everything else is great. I would like to see a "use case" here.

See previous comment.

Statement of need

  • "attempt" seems the wrong term here. Maybe "[baleen whales] have recovered slowly from commercial whaling"

I appreciate that "attempt" is perhaps an anthropomorphism. I used it to convey that none of these species, as far as I am aware, have actually recovered to their pre-whaling abundances. I have made new edit that reads: "[Baleen whale] recovery from commercial whaling is impeded by..."

  • by 'infrastructure' do you mean shipping, fishing, or both?

I used 'infrastructure' as something of a catch-all term that applies to all manner of our physical occupation of the ocean. It is not exclusive to shipping and fishing but also relates to other activities like energy extraction (e.g., wind, petroleum, tide, mining, aquaculture, etc.). I have replaced with "industry", which hopefully captures these ideas more clearly.

  • line 2: suggest to just change this to 'shifting environmental baseline' or explicitly say human-driven climate change ('exploitation' in a marine context is almost exlusively used in the context of fisheries)

Thank you for pointing out that 'exploitation' may have specific connotations and convey an incomplete message. However, I think using only "shifting environmental baseline" or "human-driven climate change" omits the very significant risks posed by industry (e.g., fishing gear entanglement, ship strike) and pollution (e.g., noise, chemicals, plastic). I have edited so the sentence now reads: "[Baleen whale] recovery from commercial whaling is impeded by anthropogenic risks from ocean industry, pollution, and climate change".

  • It is hard [for me] to identify a broader audience in the scientific community. Anyone besides the survey teams? Vessel captains? The public? Perhaps the intended user groups should be mentioned here.

Hopefully this has been addressed in the new usage examples included at the end of the "Statement of Need".

Thank you again for your thoughtful review. It has greatly improved the manuscript and application!

hansenjohnson commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

hansenjohnson commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

arfon commented 3 years ago

:wave: @KristinaRiemer – just checking what the next steps are with this submission?

KristinaRiemer commented 3 years ago

Thanks @arfon, and to everyone for their patience. It looks like @hansenjohnson has addressed all of the review comments by @mcsiple. Are there any additional followup questions about those, or do you confirm your acceptance of this submission @mcsiple?

mcsiple commented 3 years ago

I think all my comments are all addressed! The only lingering one is that dotted line at zero sightings on the y-axis-- I suggest just bringing that line to the front and making it a contrasting color if it's hard to see, or making the bars in that plot lighter so it's easier to see them at the same time. From a data viz perspective, I think it is confusing to put a possible number of sightings that could be < 0.

I like the other ui changes you made.

Everything looks great! Congratulations @hansenjohnson et al. for making this great tool available. I accept it!

KristinaRiemer commented 3 years ago

Great, thanks @mcsiple!

@hansenjohnson if you want to implement a change based on @mcsiple's comment in the previous message, feel free to do that. The next steps then are to review the paper for typos, etc., and then generate archives.

KristinaRiemer commented 3 years ago

@whedon generate pdf

KristinaRiemer commented 3 years ago

@whedon check references

whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1121/10.0001811 is OK
- 10.1111/csp2.267 is OK
- 10.3389/fmars.2020.00100 is OK
- 10.1016/j.ocecoaman.2020.105109 is OK
- 10.1016/j.apacoust.2021.107949 is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

KristinaRiemer commented 3 years ago

@hansenjohnson the only change to make to the paper would be to change "html" to "HTML" in line 53. Otherwise the paper and its references look good!

hansenjohnson commented 3 years ago

Good catch @KristinaRiemer! I have made that change here. What are the next steps?

KristinaRiemer commented 3 years ago

Next steps:

  1. Create a tagged release of the GitHub repo (instructions here) and post the version number here
  2. Generate a DOI by archiving the code on Zenodo, figshare, an institutional repository, etc. and provide that DOI. Make sure the archive has the same title as the paper title, and the same authors.
hansenjohnson commented 3 years ago
  1. The tagged release is version 1.0.0
  2. The DOI is 10.5281/zenodo.4913270

Let me know if there are any issues!

KristinaRiemer commented 3 years ago

@whedon set v1.0.0 as version

whedon commented 3 years ago

OK. v1.0.0 is the version.

KristinaRiemer commented 3 years ago

For the Zenodo archive, could you add the other authors as authors there? And the title should match the submission's title of "WhaleMap: a tool to collate and display whale survey1results in near real-time". Could you let me know when those are updated?

hansenjohnson commented 3 years ago

I just made those changes, sorry for not doing that the first time around! Let me know if any others are necessary

KristinaRiemer commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

KristinaRiemer commented 3 years ago

@whedon set 10.5281/zenodo.4913270 as archive

whedon commented 3 years ago

OK. 10.5281/zenodo.4913270 is the archive.

KristinaRiemer commented 3 years ago

That all looks good @hansenjohnson! I'm going to recommend acceptance, an EiC will double check everything, and then finalize submission acceptance.

KristinaRiemer commented 3 years ago

@whedon recommend-accept