search

openjournals / joss-reviews

Reviews for the Journal of Open Source Software
Creative Commons Zero v1.0 Universal
712 stars 38 forks source link

[REVIEW]: DiffeRT2d: A Differentiable Ray Tracing Python Framework for Radio Propagation #6915

Closed editorialbot closed 3 months ago

editorialbot commented 3 months ago

Submitting author: !--author-handle-->@jeertmans<!--end-author-handle-- (Jérome Eertmans) Repository: https://github.com/jeertmans/DiffeRT2d Branch with paper.md (empty if default branch): Version: v0.3.4 Editor: !--editor-->@danielskatz<!--end-editor-- Reviewers: @idoby, @roth-jakob Archive: 10.5281/zenodo.12600658

Status

status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/5a48c7901e2d9de3d9e13dc0e39615a0"><img src="https://joss.theoj.org/papers/5a48c7901e2d9de3d9e13dc0e39615a0/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/5a48c7901e2d9de3d9e13dc0e39615a0/status.svg)](https://joss.theoj.org/papers/5a48c7901e2d9de3d9e13dc0e39615a0)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@idoby & @roth-jakob, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @danielskatz know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @idoby

📝 Checklist for @roth-jakob

editorialbot commented 3 months ago

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf
editorialbot commented 3 months ago

Software report:

github.com/AlDanial/cloc v 1.90  T=0.07 s (1152.6 files/s, 182987.8 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          31           1346           1631           4323
Markdown                        18            443              0           1197
Jupyter Notebook                 1              0           1341            572
YAML                             8             43             10            256
TeX                              2             18              0            185
TOML                             1             23              5            170
JSON                             1              0              0            154
SVG                              1              0              0             54
DOS Batch                        1              8              1             26
reStructuredText                 9             11             33             20
make                             1              6              7             14
CSS                              1              1              0              9
-------------------------------------------------------------------------------
SUM:                            75           1899           3028           6980
-------------------------------------------------------------------------------

Commit count by author:

   277  Jérome Eertmans
    36  pre-commit-ci[bot]
     9  dependabot[bot]
     2  ImgBotApp
editorialbot commented 3 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.23919/EuCAP60739.2024.10501570 is OK
- 10.1109/TAP.2014.2323961 is OK
- 10.1109/ACCESS.2015.2453991 is OK
- 10.23919/EuCAP57121.2023.10132934 is OK
- 10.1371/journal.pone.0260060 is OK

MISSING DOIs

- No DOI given, and none found for title: Adam: A Method for Stochastic Optimization
- No DOI given, and none found for title: The DeepMind JAX Ecosystem
- No DOI given, and none found for title: JAX: composable transformations of Python+NumPy pr...
- No DOI given, and none found for title: jaxtyping: type annotations and runtime checking f...
- No DOI given, and none found for title: Equinox: neural networks in JAX via callable PyTre...
- No DOI given, and none found for title: Advanced Radio Channel Simulator
- 10.1109/gcwkshps58843.2023.10465179 may be a valid DOI for title: Sionna RT: Differentiable Ray Tracing for Radio Pr...
- No DOI given, and none found for title: TensorFlow: Large-Scale Machine Learning on Hetero...

INVALID DOIs

- None
editorialbot commented 3 months ago

Paper file info:

📄 Wordcount for paper.md is 1243

✅ The paper includes a Statement of need section

editorialbot commented 3 months ago

License info:

✅ License found: MIT License (Valid open source OSI approved license)

editorialbot commented 3 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

danielskatz commented 3 months ago

@idoby and @roth-jakob - Thanks for agreeing to review this submission. This is the review thread for the paper. All of our communications will happen here from now on.

As you can see above, you each should use the command @editorialbot generate my checklist to create your review checklist. @editorialbot commands need to be the first thing in a new comment.

As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

As you both know, the JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention openjournals/joss-reviews#6915 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for reviews to be completed within about 2-4 weeks. Please let me know if either of you require some more time. We can also use editorialbot (our bot) to set automatic reminders if you know you'll be away for a known period of time.

Please feel free to ping me (@danielskatz) if you have any questions/concerns.

idoby commented 3 months ago

Review checklist for @idoby

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

roth-jakob commented 3 months ago

Review checklist for @roth-jakob

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

idoby commented 3 months ago

Hey @jeertmans, thanks for submitting your package to JOSS! 📦 It seems very cool!

A few comments about the paper:

  1. Lines 50-51: "such as the number of multipath components and angle of arrival."
  2. Line 68: Please spell out the acronym MPT for people who don't know what it means.
  3. Line 69: "...we can easily accomodate these surfaces." Referring to objects here can be confusing since the next sentence refers to software objects rather than physical simulation objects.
  4. Figure 2: "it is evident that..." It isn't evident. Please assume the reader is not well versed in the specific subdiscipline, explain in more detail what is being shown in the figure. Point out what the RIS is and where it is depicted in the image. I also advise to plot a specific path to illustrate clearly the 45-degree angle you're referring to. re: the code, I suggest to provide the code sample as a complete, runnable script with the package and to link externally to the relevant Python file. As it stands, the code is (admittedly) incomplete and not runnable because it omits the actual drawing of the figure. While understandable, this takes away substantial value from a potential user. Consider linking to a stable repository (e.g., Zenodo) as well as GitHub once the paper is accepted and your software archive is created. Besides this minor detail, I think the code illustrates the basic usage of the package well. As a final point, if you do decide to link to the complete example as an external file, consider pointing this out in the text and not as a footnote. If you do not want to include a link, it's enough to state that the example is available as a sample in the package repository.
  5. Figure 3: Likewise, I suggest providing full runnable code for this example as well, though there is indeed no need to include the code in the paper itself.
  6. "Machine Learning" section: It is unclear what "this COST meeting" means and why it is relevant to the paper. I suggest rewriting this part and including a representative figure from the linked page. I also strongly suggest linking to a stable URL and not your personal website.

Other than these minor comments, good work! I will take a look at the software itself soon.

roth-jakob commented 3 months ago

Hi, thanks for your effort in writing the paper! Just a follow-up on the 4. above comment. I fully agree with that. Giving a little bit more explanation for readers not familiar with the subdiscipline could make the paper much more accessible.

jeertmans commented 3 months ago

Hey @jeertmans, thanks for submitting your package to JOSS! 📦 It seems very cool!

Hello @idoby, thanks for your first review comments, which I try to address in https://github.com/jeertmans/DiffeRT2d/pull/69.

Let me add a few comments here:

A few comments about the paper:

  1. Lines 50-51: "such as the number of multipath components and angle of arrival."
  2. Line 68: Please spell out the acronym MPT for people who don't know what it means.
  3. Line 69: "...we can easily accomodate these surfaces." Referring to objects here can be confusing since the next sentence refers to software objects rather than physical simulation objects.

Thanks for catching! I have rephrased those sentences.

  1. Figure 2: "it is evident that..." It isn't evident. Please assume the reader is not well versed in the specific subdiscipline, explain in more detail what is being shown in the figure. Point out what the RIS is and where it is depicted in the image. I also advise to plot a specific path to illustrate clearly the 45-degree angle you're referring to. re: the code, I suggest to provide the code sample as a complete, runnable script with the package and to link externally to the relevant Python file. As it stands, the code is (admittedly) incomplete and not runnable because it omits the actual drawing of the figure. While understandable, this takes away substantial value from a potential user. Consider linking to a stable repository (e.g., Zenodo) as well as GitHub once the paper is accepted and your software archive is created. Besides this minor detail, I think the code illustrates the basic usage of the package well. As a final point, if you do decide to link to the complete example as an external file, consider pointing this out in the text and not as a footnote. If you do not want to include a link, it's enough to state that the example is available as a sample in the package repository.

I think I have addressed this issue by (1) putting the complete code and (2) rephrasing the caption.

  1. Figure 3: Likewise, I suggest providing full runnable code for this example as well, though there is indeed no need to include the code in the paper itself.

Done.

  1. "Machine Learning" section: It is unclear what "this COST meeting" means and why it is relevant to the paper. I suggest rewriting this part and including a representative figure from the linked page. I also strongly suggest linking to a stable URL and not your personal website.

I hope this is now clearer (I forgot to update that part before submitting, this is my bad).

I always hesitate between putting links from Zenodo, GitHub, or else. The motivation to use a link to my website was that, I am more likely to keep my eertmans.be up for a long time, than keeping the absolute path to the rendered notebook the same. On my website, I just set up a permalink redirect that I can update anytime I want. The permalink is also checked by tests, so hopefully I will always point to a valid url. Zenodo (or web archives) can be nice to have a permalink, but then you cannot update the code (e.g., to fix typos etc.).

Other than these minor comments, good work! I will take a look at the software itself soon.

Looking forward to the software review! :D

jeertmans commented 3 months ago

Hi, thanks for your effort in writing the paper! Just a follow-up on the 4. above comment. I fully agree with that. Giving a little bit more explanation for readers not familiar with the subdiscipline could make the paper much more accessible.

Hello! I have (I think) addressed this comment in https://github.com/jeertmans/DiffeRT2d/pull/69. Can you review it and tell me if it needs more details?

idoby commented 3 months ago

The issue with the links is that Zenodo etc are meant to be truly permanent in the sense that they are meant to be archived forever. This is why JOSS requests that upon acceptance, you archive your software with one of these services for posterity. The JOSS guidelines do not seem to address the issue of including external links in the paper. @danielskatz what do you think?

The same applies to the link you added to https://dial.uclouvain.be/pr/boreal/object/boreal:288635 I'm not sure what this is exactly. Is it a publication? If so, it should be cited properly. Is it a preprint? If so, it should be placed on a preprint server and cited properly as a preprint paper.

Other than these points, the paper seems good and my comments have been addressed as far as I'm concerned. Will take a look at the software soon.

danielskatz commented 3 months ago

Personally, I'm ok with content on website being linked to, but I would prefer that an archive.org copy of it was linked, as this is more "permanent" if possible. e.g., rather than referencing https://www.ncsa.illinois.edu, one could reference https://web.archive.org/web/20240620232147/https://www.ncsa.illinois.edu/ for the version as of a few days ago.

For the link above, using the hdl (http://hdl.handle.net/2078/288635) would be better than using the direct link, even though it points to the same thing. If https://dial.uclouvain.be is an institutional repository, which I think it is, then this is enough. If not, then doing what @idoby suggests would be good, putting it on a "permanent" archive like Zenodo, arXiv, ...

jeertmans commented 3 months ago

Indeed, https://dial.uclouvain.be/ is an institutional repository, and should be a permalink that will always be valid. This is a preliminary work, so not really published, but I transformed the URL-reference to a citation that points to the "preprint", available on our inst. website.

For the other internet links, I will convert them later (today or this week) to web archives :-)

jeertmans commented 3 months ago

@editorialbot generate pdf

editorialbot commented 3 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

roth-jakob commented 3 months ago

Thanks for fixing the links and adding the full code for the figure in jeertmans/DiffeRT2d#69. I was able to reproduce the figure. I will have a look at the software in the next few days.

idoby commented 3 months ago

Looks like the new paper proof did not pick up the changes

jeertmans commented 3 months ago

Looks like the new paper proof did not pick up the changes

Yes, I forgot to first merge the PR ^^'. I'll fix that soon!

idoby commented 3 months ago

The software itself looks good to me, pretty cool work. I'm not a huge fan of a fully object-oriented style with abstract classes but that's really a matter of preference/design choices. I have no comments on the software other than good job, it seems well crafted and easy to read. 👍

I did find some typos in the documentation, I will open a PR to fix them soon. Other than that, l think the doc intro page deserves a little more exposition wrt what this tool is and how it's different than other similar tools (what JOSS would call a "statement of need"). I suggest to take some prose from the paper and graduate the footnote on the intro page into normal text. However, I leave this up to the author and will not block the review over this.

Once the typos are fixed, as far as I'm concerned this piece of software is ready to be included in JOSS.

jeertmans commented 3 months ago

@editorialbot commands

editorialbot commented 3 months ago

Hello @jeertmans, here are the things you can ask me to do:


# List all available commands
@editorialbot commands

# Get a list of all editors's GitHub handles
@editorialbot list editors

# Adds a checklist for the reviewer using this command
@editorialbot generate my checklist

# Set a value for branch
@editorialbot set joss-paper as branch

# Run checks and provide information on the repository and the paper file
@editorialbot check repository

# Check the references of the paper for missing DOIs
@editorialbot check references

# Generates the pdf paper
@editorialbot generate pdf

# Generates a LaTeX preprint file
@editorialbot generate preprint

# Get a link to the complete list of reviewers
@editorialbot list reviewers
jeertmans commented 3 months ago

The software itself looks good to me, pretty cool work. I'm not a huge fan of a fully object-oriented style with abstract classes but that's really a matter of preference/design choices. I have no comments on the software other than good job, it seems well crafted and easy to read. 👍

Thanks!

I did find some typos in the documentation, I will open a PR to fix them soon.

I have reviewed and merge all PRs, thanks for your help!

Other than that, l think the doc intro page deserves a little more exposition wrt what this tool is and how it's different than other similar tools (what JOSS would call a "statement of need"). I suggest to take some prose from the paper and graduate the footnote on the intro page into normal text. However, I leave this up to the author and will not block the review over this.

Good remark, I will wait for the second reviewer (and the editor?) to give their feedback first, then I think I might improve a bit the documentation according to your suggestions.

Other than that, I will re-generate a paper with the latest changes.

jeertmans commented 3 months ago

@editorialbot generate pdf

editorialbot commented 3 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

jeertmans commented 3 months ago

My TODO list:

I put that here to keep track of important changes to make, and I won't have access to a computer from July 2nd to July 16th, so this doesn't get lost :-)

roth-jakob commented 3 months ago

@jeertmans, nice software package! I had a closer look at it. I have only one minor suggestion, along the lines of @idoby's comment. I personally like it if there are a few sentences at the top of the README explaining what the package is doing and for whom it is targeted. All this information is in the paper, and you could consider summarizing it at the top of the README.

danielskatz commented 3 months ago

👋 @idoby & @roth-jakob - You both have checked off all your items - but I also see comments about things to do from you and @jeertmans. So I'm a little uncertain about if you think this is done, or if we need to wait for @jeertmans to do some more things.

idoby commented 3 months ago

@danielskatz Jerome still wants to improve the documentation page, but as far as I'm concerned the review is done and is up to @jeertmans

danielskatz commented 3 months ago

👋 @jeertmans

My TODO list:

  • [ ] Improve the main index.md documentation page, as suggested by @idoby
  • [ ] Create a new Zenodo release to point to the latest version (and update in README.md)
  • [ ] Change URLs in paper to webarchive links (or Zenodo)
  • [ ] Change package version in JOSS paper (to include latest changes)

I put that here to keep track of important changes to make, and I won't have access to a computer from July 2nd to July 16th, so this doesn't get lost :-)

At the end of the process, once the review is complete, I will proofread the paper, and perhaps suggest changes, then ask you to make an archival deposit (e.g., in Zenodo) and tell me that DOI and the version associated with it, so you might want to wait on the 2nd and 4th items above until then.

roth-jakob commented 3 months ago

Yes, for me, it's the same. I proposed to @jeertmans to expand the readme a bit, but otherwise, I'm done with the review.

danielskatz commented 3 months ago

@jeertmans - Also, I think we can complete this before you go away on July 2nd if you want, since we are quite close.

jeertmans commented 3 months ago

Sure @jeertmans, I will do that in the next few hours :-)

jeertmans commented 3 months ago

@danielskatz done, the latest version is v0.3.3 (https://github.com/jeertmans/DiffeRT2d/releases/tag/v0.3.3). I'll update the Zenodo after you have proofread the paper, as you suggested :-)

danielskatz commented 3 months ago

My institution's summer picnic is starting soon, but I'll do this after it and some meetings, in the 4-8 hours...

danielskatz commented 3 months ago

@editorialbot generate pdf

danielskatz commented 3 months ago

@editorialbot check references

editorialbot commented 3 months ago 
Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.23919/EuCAP60739.2024.10501570 is OK
- 10.1109/TAP.2014.2323961 is OK
- 10.1109/ACCESS.2015.2453991 is OK
- 10.23919/EuCAP57121.2023.10132934 is OK
- 10.1371/journal.pone.0260060 is OK

MISSING DOIs

- No DOI given, and none found for title: Adam: A Method for Stochastic Optimization
- No DOI given, and none found for title: The DeepMind JAX Ecosystem
- No DOI given, and none found for title: JAX: composable transformations of Python+NumPy pr...
- No DOI given, and none found for title: jaxtyping: type annotations and runtime checking f...
- No DOI given, and none found for title: Equinox: neural networks in JAX via callable PyTre...
- No DOI given, and none found for title: Learning to Sample Ray Paths for Faster Point-to-P...
- No DOI given, and none found for title: Advanced Radio Channel Simulator
- 10.1109/gcwkshps58843.2023.10465179 may be a valid DOI for title: Sionna RT: Differentiable Ray Tracing for Radio Pr...
- No DOI given, and none found for title: TensorFlow: Large-Scale Machine Learning on Hetero...

INVALID DOIs

- None
editorialbot commented 3 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

danielskatz commented 3 months ago

@jeertmans - I've suggested some changes in https://github.com/jeertmans/DiffeRT2d/pull/78. Please merge this or let me know what you disagree with.

jeertmans commented 3 months ago

Thanks, @danielskatz! It looked great, and I also remove one footnote that was no longer needed (as I now include the full code snippet).

jeertmans commented 3 months ago

@editorialbot generate pdf

editorialbot commented 3 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

danielskatz commented 3 months ago

@jeertmans - I seem to have introduced a bug - sorry. https://github.com/jeertmans/DiffeRT2d/pull/79 should fix it. Once you merge this, please check the proof to make it's ok, and once it is, please:

I can then move forward with accepting the submission.

danielskatz commented 3 months ago

@editorialbot generate pdf

editorialbot commented 3 months ago

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

jeertmans commented 3 months ago

There you go @danielskatz: 10.5281/zenodo.12600658.

I am just not sure: should the Zenodo metadata also contain the paper title and authors?

danielskatz commented 3 months ago

The title and authors of the Zenodo deposit should be the same as those of the paper. You can change these manually without making a new deposit by editing the metadata.