search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
721 stars 38 forks source link

[REVIEW]: Leafmap: A Python package for interactive mapping and geospatial analysis with minimal coding in a Jupyter environment #3414

Closed whedon closed 3 years ago

whedon commented 3 years ago

Submitting author: @giswqs (Qiusheng Wu) Repository: https://github.com/giswqs/leafmap Version: v0.3.5 Editor: @ajstewartlang Reviewers: @martinfleis, @TomasBeuzen Archive: 10.5281/zenodo.5137372

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/b081afbd9bdaa4386475c787db8e370c"><img src="https://joss.theoj.org/papers/b081afbd9bdaa4386475c787db8e370c/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/b081afbd9bdaa4386475c787db8e370c/status.svg)](https://joss.theoj.org/papers/b081afbd9bdaa4386475c787db8e370c)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@TomasBeuzen & @martinfleis, please carry out your review in this issue by updating the checklist below. If you cannot edit the checklist please:

  1. Make sure you're logged in to your GitHub account
  2. Be sure to accept the invite at this URL: https://github.com/openjournals/joss-reviews/invitations

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @ajstewartlang know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Review checklist for @TomasBeuzen

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

Review checklist for @martinfleis

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

whedon commented 3 years ago

Hello human, I'm @whedon, a robot that can help you with some common editorial tasks. @TomasBeuzen, @TomasBeuzen, @martinfleis it looks like you're currently assigned to review this paper :tada:.

:warning: JOSS reduced service mode :warning:

Due to the challenges of the COVID-19 pandemic, JOSS is currently operating in a "reduced service mode". You can read more about what that means in our blog post.

:star: Important :star:

If you haven't already, you should seriously consider unsubscribing from GitHub notifications for this (https://github.com/openjournals/joss-reviews) repository. As a reviewer, you're probably currently watching this repository which means for GitHub's default behaviour you will receive notifications (emails) for all reviews 😿

To fix this do the following two things:

  1. Set yourself as 'Not watching' https://github.com/openjournals/joss-reviews:

watching

  1. You may also like to change your default settings for this watching repositories in your GitHub profile here: https://github.com/settings/notifications

notifications

For a list of things I can do to help you, just type:

@whedon commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@whedon generate pdf
whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.rse.2017.06.031 is OK
- 10.21105/joss.02272 is OK
- 10.5334/jors.148 is OK
- 10.5281/zenodo.4569086 is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 3 years ago 
Software report (experimental):

github.com/AlDanial/cloc v 1.88  T=0.47 s (174.0 files/s, 143442.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
JSON                             6              0              0          50166
Python                          15           1573           1782           6213
XML                              2              0              0           1636
Markdown                        23            253              0            676
Jupyter Notebook                24              0           4369            479
YAML                             9             25             10            278
HTML                             2              3              0             65
TeX                              1              4              0             59
-------------------------------------------------------------------------------
SUM:                            82           1858           6161          59572
-------------------------------------------------------------------------------

Statistical information for the repository '4dcb6ecfa0ee11557c92d6e1' was
gathered on 2021/06/25.
The following historical commit information, by author, was found:

Author                     Commits    Insertions      Deletions    % of changes
Kharude, Sachin                  7          1145            131           11.60
Nahid Pervez                     3            32              4            0.33
Qiusheng Wu                     45          9258            433           88.08

Below are the number of rows from each author that have survived and are still
intact in the current revision:

Author                     Rows      Stability          Age       % in comments
Kharude, Sachin             707           61.7          0.4                3.54
Nahid Pervez                 29           90.6          0.4                6.90
Qiusheng Wu                8832           95.4          0.8                6.13
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

giswqs commented 3 years ago

@ajstewartlang I noticed TomasBeuzen is listed twice under the Reviewers section in the PDF. Is it possible to correct this?

danielskatz commented 3 years ago

👋 @ajstewartlang - FYI, this is because in https://github.com/openjournals/joss-reviews/issues/3396 you both assigned him and then added him - you only need to do one of these: typically, use assign for the first reviewer, and then add for any subsequent ones. You can probably use @whedon remove @TomasBeuzen as reviewer to remove one of the duplicates here, then regenerate the PDF. Let me know if you have problems...

whedon commented 3 years ago

:wave: @martinfleis, please update us on how your review is going (this is an automated reminder).

whedon commented 3 years ago

:wave: @TomasBeuzen, please update us on how your review is going (this is an automated reminder).

whedon commented 3 years ago

:wave: @TomasBeuzen, please update us on how your review is going (this is an automated reminder).

martinfleis commented 3 years ago

Dear @giswqs and @ajstewartlang,

I have finished the review of the package and its paper. Even though the package is an advanced piece of software helpful on many occasions, some issues need to be resolved before publication in JOSS. Please find a detailed explanation below.

Review

leafmap will undoubtedly find its audience. It does what it promises to do and does it well. I have only a few remarks, mostly minor, but let me start with the only major issue.

The software, in its current state, can be considered untested. The repository contains a few test files, but those do not sufficiently check the functionality. As reported by pytest-cov, overall coverage is only 12% (see the full report below). I would argue that even those 12% do not reflect the actual state as most of the tests are essentially checking if it fails or not. Test suite needs to check whether the output of each class, method and function returns what is expected of it. The current test suite does not do that. I understand that it may not be as straightforward in interactive mapping, but there are ways based on string (HTML) outputs - look at folium, for example. 

Name                   Stmts   Miss  Cover
------------------------------------------
leafmap/__init__.py       16      3    81%
leafmap/basemaps.py       28      4    86%
leafmap/colormaps.py      74     74     0%
leafmap/common.py       1038    839    19%
leafmap/foliumap.py      470    470     0%
leafmap/heremap.py       153    153     0%
leafmap/leafmap.py       932    809    13%
leafmap/legends.py        27     24    11%
leafmap/osm.py            88     50    43%
leafmap/toolbar.py       549    549     0%
------------------------------------------
TOTAL                   3375   2975    12%

Now on to minor issues (in no particular order):

AttributeError: module 'leafmap.foliumap' has no attribute 'linked_maps'`

The mental model leafmap use is very different from the PyData stack I am mostly used to. It tries to solve everything, which is even taken into extremes with functions like csv_to_pandas which is just pd.read_csv(in_csv, **kwargs) or read_postgis which is a wrapper of gpd.read_postgis(sql, con, geom_col, crs, **kwargs). It is not a problem that they exist; I am just struggling to figure out whether they should be a part of the public API as they are now or whether they are considered private. The same applies to a series of functions I would not expect in the API of a package like leafmap, supporting the dev-level installation of packages (update_package, install_from_github...). Again, should it be part of a user-facing API? I am not asking for these to be changed; I just wanted to point out that it is not a very common practice.

Paper

A few notes regarding the paper:

giswqs commented 3 years ago

@martinfleis Thank you very much for the thorough review and constructive comments. I will revise the repo and paper as suggested over the next few days.

Regarding your major comment on test coverage, I will look into the folium package to see how to increase the test coverage in interactive mapping. As you already pointed out, test coverage in interactive mapping is not very straightforward. I followed the ipyleaflet package for testing coverage in interactive mapping. The ipyleaflet repo itself does not contain the tests folder. All the tests are actually performed when the documentation website is being built and deployed to https://ipyleaflet.readthedocs.io, which is similar to what I did for leafmap. All the notebook examples demonstrating key functions of leamap are executed and tested using GitHub workflow (docs.yml) when deployed to GitHub Pages, e.g., https://leafmap.org/notebooks/01_leafmap_intro. Users can see the interactive map. The downside is that pytest coverage can't capture these automated tests passed by GitHub workflow.

Similarly, the here-map-widget-for-jupyter package repo does not contain a tests folder. All the tests are performed when the documentation website is being built and deployed to https://here-map-widget-for-jupyter.readthedocs.io

ajstewartlang commented 3 years ago

👋 @ajstewartlang - FYI, this is because in #3396 you both assigned him and then added him - you only need to do one of these: typically, use assign for the first reviewer, and then add for any subsequent ones. You can probably use @whedon remove @TomasBeuzen as reviewer to remove one of the duplicates here, then regenerate the PDF. Let me know if you have problems...

Thanks @danielskatz - yes, looks like I used assign twice rather than add the second time - I will remove the duplicate and regenerate the PDF....

ajstewartlang commented 3 years ago

@whedon remove @TomasBeuzen as reviewer

whedon commented 3 years ago

OK, @TomasBeuzen is no longer a reviewer

ajstewartlang commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

ajstewartlang commented 3 years ago

@whedon add @TomasBeuzen

whedon commented 3 years ago

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@whedon commands
ajstewartlang commented 3 years ago

@whedon add @TomasBeuzen as reviewer

whedon commented 3 years ago

OK, @TomasBeuzen is now a reviewer

ajstewartlang commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

martinfleis commented 3 years ago

All the tests are actually performed when the documentation website is being built

That is interesting, I did not realise that. But that again only checks if the map is generated, not what it shows, right? The point of CI is to ensure the correctness of the output.

A lot of your methods can be tested independently even before they're added to the map itself (e.g. everything in colormaps.py), which can make it a bit easier. I've been implementing folium-based plotting to geopandas recently so you may even check how we deal with tests there (we essentially get the HTML string and check if expected strings are part of it) - https://github.com/geopandas/geopandas/pull/1953.

giswqs commented 3 years ago

@martinfleis It can check if the map is generated as well as functions within the Map class. Take a look at this notebook https://leafmap.org/notebooks/10_add_vector as an example. It tests functions such as add_shp, add_geojson, add_kml. One can verify whether data are being loaded correctly on the map. Leafmap also has a add_gdf function for loading GeoPandas GeoDataFrame. I will add it to the notebook example as well.

Thank you for providing the reference to your geopandas folium example. I will learn from it to increase the test coverage.

martinfleis commented 3 years ago

I know that the example uses those methods and they're executed during the docs creation but there's no automatic check if the result is correct, right? Just that it didn't fail. You can always check manually but that is not what CI is about. Correct me if I misunderstood what is going on there.

giswqs commented 3 years ago

@martinfleis Yes, I totally agree with you. There is no automatic check if the result is expected during the docs creation. I will learn from your geopands/test_explore.py and try my best to increase the test coverage. Getting the HTML string and checking if expected strings are part of it sounds great. One thing to keep in mind is that checking parting of the HTML string does not guarantee that the map will be created successfully or the data will be loaded properly. Leafmap has multiple plotting backends (i.e., ipyleaflet, folium, heremap). The HTML string of the map created using different backends (and data loaded using the same function, e.g., add_geojson, add_shp) might have some common strings in the HTML, but there are not interchangeable. I will use CI as the automatic check and use notebooks during docs creation as a supplemental check to make sure the generated results are as expected.

giswqs commented 3 years ago

Just to add a reference to the point I raised earlier - checking the HTML string does not guarantee the map/data will be rendered successfully. Here is a relevant ipyleaflet issue (https://github.com/jupyter-widgets/ipyleaflet/issues/841) I opened last month. The map can be exported to HTML without any problem (i.e., you can get the HTML string), but the map still fails to render due to other issues. Therefore, CI would not be able to capture this issue. In this case, using both CI and notebooks executed during docs creation would be very helpful, even though we have to manually check the map/data shown on notebooks.

TomasBeuzen commented 3 years ago

Hi @giswqs and @ajstewartlang 👋

I have finished the review of the Leafmap package and its paper. The package and its documentation are comprehensive. I feel it's particularly suited to less-technical users, or those just getting started, but that's not to say that it's not also a useful tool for more seasoned programmers/developers. Visualizing geospatial data can certainly be a daunting task for novice users so I think this is a nice contribution to the open source world, well done @giswqs. I believe there is a bit of room for improvement before publication, and my comments mostly echo what Martin has said before me.

Software

Paper

Overall I felt the paper could be expanded on a little:

ajstewartlang commented 3 years ago

Many thanks for your detailed and helpful reviews @martinfleis and @TomasBeuzen. This looks like a really nice package @giswqs and a contribution that will be really useful for the community. The comments by the reviewers look very helpful and hopefully easy to address. If you could let us know when you've been able to incorporate those suggestions that would be great.

giswqs commented 3 years ago

@TomasBeuzen Thank you for the thorough review and encouraging comments. I will be addressing your comments in this issue https://github.com/giswqs/leafmap/issues/83. Thank you for your understanding of some of the simple wrapper functions (e.g., csv_to_pdf, read_postgis). Those functions were designed for novice users who might not be familiar with pandas or geopandas. They can use leafmap without having to import pandas or geopandas. Other less commonly used functions have been changed to private functions as suggested by @martinfleis. See https://github.com/giswqs/leafmap/commit/a28e84a5989344a99ce93f0a978f89c1e557ddf1. I prefer to keep the leafmap.update_package() function, which is essentially pip install git+https://github.com/giswqs/leafmap. Typing a function with 2-3 characters and then use autocompletion in Jupyter is much more convenient than memorizing and typing (or copying and pasting) a long string with ~50 characters.

@ajstewartlang I expect to finish the revision by early next week. I will give an update here when it is ready for re-review.

giswqs commented 3 years ago

@whedon generate pdf

whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

giswqs commented 3 years ago

@martinfleis @TomasBeuzen Thank you again for your constructive comments. I have revised the repo, documentation and paper based on your comments. You can check out the commit history referenced in https://github.com/giswqs/leafmap/issues/83. Each commit addresses one of your comments, in no particular order. Please review it and let me know if you have any further concerns. Thank you very much for your time.

martinfleis commented 3 years ago

@giswqs Thanks a lot for the changes, great job! Though I could imagine higher code coverage I don't think it an issue blocking the publication. All the other comments have been resolved.

I have ticked remaining boxes in the review form above and recommend the paper for publication.

giswqs commented 3 years ago

@martinfleis Thank you very much for taking the time to re-review the package. I agree with you that the code coverage can be improved. I tried my best. There are some interactive custom widgets that are a bit challenging for unit tests. I will continue to improve the code coverage in the future when unit tests for these interactive custom widgets become possible.

@TomasBeuzen Please let me know if there is anything else you would like me to address. Thank you.

TomasBeuzen commented 3 years ago

@giswqs (cc @ajstewartlang) - you've (comprehensive!) addressed all my comments, and thank you for laying everything out in easy-to-navigate commits (apologies for the delay, I was moving house the last few days so was reviewing the changes in stages). I'm satisfied that the submissions meets all the JOSS criteria and have ticked all my boxes. Excellent work again @giswqs for this excellent submission - I look forward to following how the software develops and is used by the community into the future.

ajstewartlang commented 3 years ago

@whedon generate pdf

ajstewartlang commented 3 years ago

@whedon check references

giswqs commented 3 years ago

@whedon generate pdf

ajstewartlang commented 3 years ago

@whedon generate pdf

@giswqs I've reached out to the rest of the JOSS team to find out why whedon isn't generating the pdf...

giswqs commented 3 years ago

@ajstewartlang Thank you for fixing the typo. I have made a tag version that includes all changes made to the paper.

danielskatz commented 3 years ago

@whedon check references

whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.rse.2017.06.031 is OK
- 10.21105/joss.02272 is OK
- 10.5334/jors.148 is OK
- 10.5281/zenodo.4569086 is OK
- 10.5281/zenodo.4447642 is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 3 years ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1016/j.rse.2017.06.031 is OK
- 10.21105/joss.02272 is OK
- 10.5334/jors.148 is OK
- 10.5281/zenodo.4569086 is OK
- 10.5281/zenodo.4447642 is OK

MISSING DOIs

- None

INVALID DOIs

- None
whedon commented 3 years ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

ajstewartlang commented 3 years ago

@whedon set 10.5281/zenodo.5137372 as archive

whedon commented 3 years ago

OK. 10.5281/zenodo.5137372 is the archive.

ajstewartlang commented 3 years ago

Many thanks @martinfleis and @TomasBeuzen for your helpful and detailed reviews - thank you @giswqs for addressing all the comments - I've read through myself and agree that this is a great submission that will be very useful to many people! Congrats and thank you for submitting to JOSS!

ajstewartlang commented 3 years ago

@whedon set v0.3.5 as version

whedon commented 3 years ago

OK. v0.3.5 is the version.