search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
697 stars 36 forks source link

[REVIEW]: Mobilkit: A Python Toolkit for Urban Resilience and Disaster Risk Management Analytics using High Frequency Human Mobility Data #5201

Closed editorialbot closed 4 months ago

editorialbot commented 1 year ago

Submitting author: !--author-handle-->@takayabe0505<!--end-author-handle-- (Takahiro Yabe) Repository: https://github.com/mindearth/mobilkit Branch with paper.md (empty if default branch): Version: v0.2.8 Editor: !--editor-->@crvernon<!--end-editor-- Reviewers: @ifthompson, @levisweetbreu Archive: 10.6084/m9.figshare.24707115

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/aee7d69db418d0e47105ce41c91a096d"><img src="https://joss.theoj.org/papers/aee7d69db418d0e47105ce41c91a096d/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/aee7d69db418d0e47105ce41c91a096d/status.svg)](https://joss.theoj.org/papers/aee7d69db418d0e47105ce41c91a096d)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@ssujit & @jlevente, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @crvernon know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @levisweetbreu

📝 Checklist for @ifthompson

editorialbot commented 1 year ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
crvernon commented 1 year ago

👋 @takayabe0505 @ssujit @jlevente This is the review thread for the paper. All of our communications will happen here from now on.

Please read the "Reviewer instructions & questions" in the first comment above.

Both reviewers have checklists at the top of this thread (in that first comment) with the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention https://github.com/openjournals/joss-reviews/issues/5201 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for the review process to be completed within about 4-6 weeks but please make a start well ahead of this as JOSS reviews are by their nature iterative and any early feedback you may be able to provide to the author will be very helpful in meeting this schedule.

editorialbot commented 1 year ago 
Software report:

github.com/AlDanial/cloc v 1.88  T=1.54 s (27.8 files/s, 20969.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Jupyter Notebook                11              0          20418           3844
Python                          10            849           2334           2972
XML                              9              0              0           1401
Markdown                         4             61              0            187
reStructuredText                 4             84             54             84
YAML                             2              5             11             29
TeX                              1              0              0             24
make                             1              4              7              9
TOML                             1              0              0              6
-------------------------------------------------------------------------------
SUM:                            43           1003          22824           8556
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository
editorialbot commented 1 year ago

Wordcount for paper.md is 728

editorialbot commented 1 year ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- None

MISSING DOIs

- None

INVALID DOIs

- None
editorialbot commented 1 year ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

jlevente commented 1 year ago

Review checklist for @jlevente

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

crvernon commented 1 year ago

:wave: @ssujit, @jlevente could you give me a short update as to how things are going with your reviews? Thanks!

crvernon commented 1 year ago

:mega: Mid-week Rally! :mega:

@ssujit and @jlevente could you give me a short update as to how things are going with your reviews?

Thanks!

crvernon commented 1 year ago

:wave: @ssujit and @jlevente could you give me a short update as to how things are going with your reviews?

@ssujit let me know if you are having trouble creating your reviewer checklist.

crvernon commented 1 year ago

Please update 🙏 ...

👋 @ssujit and @jlevente could you give me a short update as to how things are going with your reviews?

@ssujit let me know if you are having trouble creating your reviewer checklist.

crvernon commented 1 year ago

🚨 MIDWEEK RALLY 🚨

@ssujit and @jlevente - Could you please provide an update to how things are going here? Please do let me know if you are unable to review this article so I may move it forward. As always, let me know if you have any questions!

Have a great day!

crvernon commented 1 year ago

❗ Last call for a response before I have to assign different reviewers @ssujit and @jlevente! Please respond in this thread if you are able to conduct this review within the very near future. Thanks!

crvernon commented 1 year ago

@takayabe0505 I am removing your current reviewers due to no response. I will find you two more to keep this review going. Apologies.

crvernon commented 1 year ago

@editorialbot remove @ssujit as reviewer

editorialbot commented 1 year ago

@ssujit removed from the reviewers list!

crvernon commented 1 year ago

@editorialbot remove @jlevente as reviewer

editorialbot commented 1 year ago

@jlevente removed from the reviewers list!

crvernon commented 1 year ago

:wave: @ifthompson can you help out with this one?

crvernon commented 1 year ago

:wave: @levisweetbreu let me know if you are able to help out with this one. Thanks!

ifthompson commented 1 year ago

@crvernon Yes, I can review this in the near future.

crvernon commented 1 year ago

@editorialbot assign @ifthompson as reviewer

editorialbot commented 1 year ago

I'm sorry human, I don't understand that. You can see what commands I support by typing:

@editorialbot commands

crvernon commented 1 year ago

@editorialbot add @ifthompson as reviewer

editorialbot commented 1 year ago

@ifthompson added to the reviewers list!

levisweetbreu commented 1 year ago

@crvernon I would be happy to help out!

crvernon commented 1 year ago

@editorialbot add @levisweetbreu as reviewer

editorialbot commented 1 year ago

@levisweetbreu added to the reviewers list!

crvernon commented 1 year ago

👋 @ifthompson and @levisweetbreu

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions @ifthompson & @levisweetbreu , your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let me know.

crvernon commented 1 year ago

@ifthompson and @levisweetbreu

This is the review thread for the paper. All of our communications will happen here from now on.

As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention https://github.com/openjournals/joss-reviews/issues/5201 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for the review process to be completed within about 3-4 weeks but please make a start well ahead of this as JOSS reviews are by their nature iterative and any early feedback you may be able to provide to the author will be very helpful in meeting this schedule.

levisweetbreu commented 1 year ago

Review checklist for @levisweetbreu

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

ifthompson commented 1 year ago

Review checklist for @ifthompson

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

crvernon commented 1 year ago

👋 @ifthompson and @levisweetbreu how are we doing on this submission? Are there any other outstanding issues?

crvernon commented 1 year ago

:wave: @takayabe0505 - I see there are a few outstanding issues. Do you know when these will be resolved so that we can move forward? Thanks!

levisweetbreu commented 1 year ago

@crvernon those issues have been left open by me, there are two outstanding concerns I have.

The first is probably semantic, but the checklist asks if the submitting author has made major contributions to the software. I am confident that significant scholarly work has gone into this software, but the only work I can see in the commit history is from @ubi15 (who is the lead author on the paper), not from the submitting author @takayabe0505. I wanted to know your opinion on this before moving forward.

The second concern is with the proprietary dataset mentioned in the paper and used in the software tutorial. Although the author's don't make specific claims or results in the paper, the project they mention does not have open-source data and thus cannot be replicated. The authors do supply some open-source data in the tutorial, but it is difficult to verify the functionality of the software when most of the data used is proprietary, especially because the software does not use automated tests.

@crvernon those are the only items I wanted to check with you and see JOSS's opinion on, otherwise I am comfortable giving the OK on this submission.

ifthompson commented 1 year ago

@crvernon aside from what @levisweetbreu has mentioned, I think the community guidelines just need to explicitly mention that users are welcome to raise an issue to 2) Report issues or problems with the software 3) Seek support, as per the review checklist, not just to contribute. @takayabe0505 let me know if I have missed this somewhere in the documentation, otherwise it should be a quick add to the README.

Not a blocking issue, but non-specialists reading the paper might benefit from the term "mobility data" being defined early on. I think from context it's clear enough to understand, but for those outside of this field, "mobility" can be more associated with as the ability to physically move (e.g., move your body's joints), rather than high frequency location data.

@crvernon One last thing: JOSS's Human participants research policy primarily relates to medical data, rather than privacy/ethics concerns with this kind of location data, such as determining home/work location. The documentation mentions that mobility data often has noise added near the user's home in order to preserve their privacy, so home location cannot be determined with higher resolution than this noise if present, but does JOSS have any guidelines relating to software that works with this kind of data?

Once these are addressed I think this is good.

takayabe0505 commented 1 year ago

@levisweetbreu Hello, regarding the contribution from my side, I was the author of the first version of most of the functions that were implemented as part of the work for the World Bank. @ubi15 then cleaned the code and formatted them into production level code, and then committed to the github repo. I believe this is sufficient as a significant contribution on the software.

takayabe0505 commented 11 months ago

@ifthompson We have addressed the following comment

One comment is: I think the community guidelines just need to explicitly mention that users are welcome to raise an issue to 2) Report issues or problems with the software 3) Seek support, as per the review checklist, not just to contribute. @takayabe0505 let me know if I have missed this somewhere in the documentation, otherwise it should be a quick add to the README.

Please see the 7b50b0e commit, thanks!

crvernon commented 11 months ago

👋 @takayabe0505 @ifthompson and @levisweetbreu

Could you provide an update in this thread to how things are going and what is left to complete for this review? Thanks!

levisweetbreu commented 11 months ago

@crvernon I need input from you on the issues I mentioned in my previous comment, namely the authors' use of proprietary data in the paper and for verification of the software.

arfon commented 10 months ago

:wave: @crvernon – I think the author is seeking your feedback here.

crvernon commented 10 months ago

Thank you! I'll reply in full here later today. I am on business travel but will have time tonight.

crvernon commented 9 months ago

@levisweetbreu ... to address your questions:

Question 1:

The first is probably semantic, but the checklist asks if the submitting author has made major contributions to the software. I am confident that significant scholarly work has gone into this software, but the only work I can see in the commit history is from @ubi15 (who is the lead author on the paper), not from the submitting author @takayabe0505. I wanted to know your opinion on this before moving forward.

I believe the response from the author is sufficient to meet this requirement. See:

@levisweetbreu Hello, regarding the contribution from my side, I was the author of the first version of most of the functions that were implemented as part of the work for the World Bank. @ubi15 then cleaned the code and formatted them into production level code, and then committed to the github repo. I believe this is sufficient as a significant contribution on the software.

Question 2:

The second concern is with the proprietary dataset mentioned in the paper and used in the software tutorial. Although the author's don't make specific claims or results in the paper, the project they mention does not have open-source data and thus cannot be replicated. The authors do supply some open-source data in the tutorial, but it is difficult to verify the functionality of the software when most of the data used is proprietary, especially because the software does not use automated tests.

I think this is a valid concern. Though the requirement is that the software provided be open-source, the need to actually ensure that it is working as claimed is a part of this review. @takayabe0505 is there a way to ensure the functionality claims of this software without the use of proprietary data?

crvernon commented 9 months ago

@takayabe0505 just coming back the the question above ⬆️

crvernon commented 8 months ago

@takayabe0505 following up again on the aforementioned. Let me know if you have any questions! Thanks!

takayabe0505 commented 8 months ago

Hi @crvernon sorry for the late response. We used open source data (there's only 1 publicly available open dataset we know of - the GeoLife data; https://www.microsoft.com/en-us/research/publication/geolife-gps-trajectory-dataset-user-guide/) to make sure our functions work properly, in the 'Quickstart' section in the documentation: https://mobilkit.readthedocs.io/en/latest/examples/mobilkit_tutorial.html#

I hope this satisfies the reviewers' concerns. Thanks!

crvernon commented 8 months ago

@levisweetbreu note:

Hi @crvernon sorry for the late response. We used open source data (there's only 1 publicly available open dataset we know of - the GeoLife data; https://www.microsoft.com/en-us/research/publication/geolife-gps-trajectory-dataset-user-guide/) to make sure our functions work properly, in the 'Quickstart' section in the documentation: https://mobilkit.readthedocs.io/en/latest/examples/mobilkit_tutorial.html# I hope this satisfies the reviewers' concerns. Thanks!

levisweetbreu commented 8 months ago

@crvernon If this meets the JOSS requirements, then my concerns are satisfied, and I approve this submission for publication.

crvernon commented 8 months ago

@ifthompson are we all good from your side of things?

crvernon commented 8 months ago

@takayabe0505 there are still two open issues related to this thread:

https://github.com/mindearth/mobilkit/issues/5

https://github.com/mindearth/mobilkit/issues/6

Have these been satisfied?