JOSS Review Comment #1

[paudetseis]

Hi @akinremisa,

Thank you for making your code public. It looks like an interesting and useful application, and I would likely end up using it myself or recommending it to collaborators. However, I found the package to be deficient in several regards, and I strongly encourage you to address the following comments and suggestions:

• [x] Installing rfsed using pip works fine, but it isn't clear what to do once the package is installed this way. One can then import it using import rfsed, but then what? Because the software is not clearly developed as an API (although it could be, see below) but rather a collection of example notebooks that import the rfsed modules and run the functions, installing with pip makes little sense to me because those scripts are not available by importing rfsed, and the user does not have a clear idea where the examples folder is located (on my computer they are in /opt/anaconda3/envs/rfsedtest/lib/python3.10/site-packages/rfsed/examples). If you wish to make rfsed available via PyPI, you should have a function that installs the notebooks locally (e.g., import rfsed; rfsed.install_notebooks()), and there should be documentation to explain this and how to run the tests, etc. Alternatively, you could remove rfsed from PyPI and only provide installation instructions through cloning.
• [x] Installing the package by cloning the repository and mirroring the conda environment did not work for me. It appears like the environment.yml file is too restrictive with specific package versions, and this install is not flexible enough. Most users will give up trying to fix the package versions, or you will end up spending lots of time trying to debug other people's environments. I recommend specifying which external packages are minimally required and installing them when creating the conda environment, e.g., conda create -n rfsed python=3.9 obspyh5=0.6 ....
• [x] The Documentation link appears to link back to the README file and shows identical content. Proper documentation should describe how to use the code and not simply restate the README and point to the examples folder. In particular, this doesn't work when installing using PyPI. Please either make a full documentation or explain how to install and run the notebooks. However, even the notebooks should be edited to improve clarity (the code is difficult to read and contains a single comment per code cell). Writing proper documentation is essential; it improves the user experience and prevents you from receiving countless requests for help.
• [x] The src/rfsed folder contains the modules developed (each module being a collection of functions), but these are not listed anywhere, and the user doesn't know what they do. They appear to contain minimal documentation that can be consulted when typing, e.g., from rfsed import ExtractEq; help(ExtractEq); help(ExtractEq.ExtractEq). Because the API information is available, it should be provided on the Documentation page without having to type help() on the command line. I recommend expanding the docstring to ensure the user knows exactly what the function does and how to call it properly. You could even add basic usage or specific examples directly in the docstring.
• [x] I highly recommend linking the documentation in a docs folder and building it using tools like Sphinx (https://www.sphinx-doc.org/en/master/) or Read the Docs (https://readthedocs.com). Note: I see that sphinx is required in the environment.yml file, but it's unclear how it's been used here.
• [x] I recommend improving the readability of your code by adhering to the pep8 style guide (https://peps.python.org/pep-0008/). You could consider using Black for strict Python code formatting (https://github.com/psf/black)
• [x] Finally, you could check out other similar repositories to guide the restructuring of your code and address some of these comments. I can point you to my own repository Telewavesim, which is also a collection of modules that contain functions with example notebooks. This is by no means the only or best example, but it will give you some guidance for a package published in JOSS.

This issue relates to https://github.com/openjournals/joss-reviews/issues/6612

[akinremisa]

Hi @paudetseis

Thank you for your review of our repository. Your comments are very helpful in improving the accessibility of the software.

1. A local installation of the notebook examples is added after installation of rfsed via PyPI. The notebooks are now more detailed step-by-step of how to use the modules.
2. The environment.yml has been excluded from the repository. And a guide on how to create an environment and install the packages is described in the documentation.
3. A proper and extended documentation is now done for rfsed. Although the html files are still in the repository
4. The docstring for the modules are now extended to include what the function does, the parameters, how to initialize/import the function, and a simple demonstration of how to use it. This is also included in the documentation.
5. The readability of the code has been improved using pep8

[paudetseis]

Thank you, @akinremisa, for making those revisions to the software. I can now install rfsed without a glitch using both pip and cloning. The documentation is also much improved, and the html doc is highly appreciated. I still have a few suggestions for improving the user's experience:

• [x] The automated tests are appreciated (although they produce weird results, probably because they are meant to quickly test the code), but the corresponding documentation is still a bit misleading. Simply typing pytest after installation will produce very different results whether the software is installed via pip or through cloning, because the source code is not in the current directory when installed via pip. I suggest you either tell the user where to go to run the tests depending on the install method, or you specify that tests are only available for those who clone the code.
• [x] The installation of the notebooks works as advertised in the documentation. However, when I tried running Example1 in Jupyter, I got a message saying that tqdm is unavailable (not installed). This should be fixed in the installation (setup.cfg).
• [x] Some notebooks (e.g., Example6) have the output of the last jupyter run saved and displayed. I suggest saving "clean" versions for all notebooks (the successfully executed versions are online anyways).
• [x] I am a bit worried about the quality of the notebook results. For example, Examples 6 to 8 show a poor estimation of Vp/Vs and Depth, where the best-fit solutions are at the edges of the model. None of them appear to be very satisfactory. Is this on purpose? Maybe to show that the waveform fitting is superior? I am surprised to see that most methods fail to estimate model parameters for this station. It is not a great selling point of the software if the examples don't give any decent results. Can you either change the example data or be more explicit in the notebook about these data and what you observe and interpret from the results? Describe the experiment in more detail and the motivation for choosing poor examples? Or, I'm guessing, the example data are the same as the ones used in testing, and are much more limited than a typical dataset?
• [x] When I started paying attention to the actual waveform data used in the examples, I noticed a mix-up in stations between Example 1 (station HGN) and Examples 3-5 (station NE301) and then Examples 6-8,11-12 (station OPLO). This is quite confusing to the user, and I recommend cleaning this up to offer a complete analysis for a single station. Perhaps use the example highlighted in the JOSS paper?

[paudetseis]

Note, I am leaving a couple of checkboxes open in my review, that I will check once these comments have been addressed.

[elbeejay]

Perfect, thanks @paudetseis

[akinremisa]

Hi @paudetseis

Thank you for the new comments. I now addressed them in the repository and also provide the details of the change made below.

1. The results of the automated tests (waveform fitting) look as such because they use smaller discretization to make them run faster. On running the tests, this is now more clearly described in the documentation. The pytest can be used in the package directory when rfsed is installed by cloning the repository. There is an option to install the automated files locally and then run the tests. This is described in the documentation.

2. 'tqdm' is a dependency of rf and should be installed automatically along side other dependencies. However, it is also now included in the setup.cfg file.

3. Because I use Jupyter notebook viewer that is directly linked to the example files in the repository, the same version (executed) is kept for both the online viewing and the repository itself.

4. I now use a single station data for all the examples (Station OPLO). The station is overlying sedimentary rock and the estimation of the Moho depth and Vp/Vs is poorly constrained when using one-layer H-k stacking, Sequential H-k stacking and the resonance filtering methods. This effect of sediment on the estimation in the examples is not clearly described in the notebook. Also, the waveform fitting approach estimate a more reliable Moho depth and Vp/Vs estimates.

5. Now, same data (single station - OPLO) is used for all the examples and tests.

[paudetseis]

Tahnk you @akinremisa I appreciate these changes and can now recommend publication at JOSS. I will now close this comment and mark it as complete. Good work!