[REVIEW]: Olaf: a lightweight, portable audio search system

[editorialbot]

Submitting author: !--author-handle-->@JorenSix<!--end-author-handle-- (Joren Six) Repository: https://github.com/JorenSix/Olaf Branch with paper.md (empty if default branch): Version: v1.1.0 Editor: !--editor-->@bmcfee<!--end-editor-- Reviewers: @liscio, @ebezzam Archive: 10.5281/zenodo.8093527

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/f5b4572fd51939c2ab8363e561bdb2b7"><img src="https://joss.theoj.org/papers/f5b4572fd51939c2ab8363e561bdb2b7/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/f5b4572fd51939c2ab8363e561bdb2b7/status.svg)](https://joss.theoj.org/papers/f5b4572fd51939c2ab8363e561bdb2b7)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@liscio & @ebezzam, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review. First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @bmcfee know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @liscio

📝 Checklist for @ebezzam

[editorialbot]

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

[editorialbot]

Software report:

github.com/AlDanial/cloc v 1.88  T=0.15 s (443.0 files/s, 342772.0 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C/C++ Header                    22            490           2742          27735
C                               25           2256           2955          12901
Ruby                             7            250            105            901
SVG                              3              4              0            537
TeX                              1             47              0            322
make                             1             14             11            136
Arduino Sketch                   1             35              7             85
HTML                             1             13              0             60
Markdown                         1             22              0             43
JavaScript                       1             14              2             35
R                                1              4              0             34
YAML                             2              5              8             32
Dockerfile                       1              8             22             12
-------------------------------------------------------------------------------
SUM:                            67           3162           5852          42833
-------------------------------------------------------------------------------

gitinspector failed to run statistical information for the repository

[editorialbot]

Wordcount for paper.md is 1040

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1109/MMSP.2002.1203273 is OK
- 10.5281/zenodo.1416340 is OK
- 10.5281/zenodo.1417593 is OK
- 10.1109/TASL.2012.2234114 is OK
- 10.1007/978-3-319-73165-0_10 is OK
- 10.1007/s11265-005-4151-3 is OK
- 10.5281/zenodo.1414728 is OK
- 10.1121/1.400476 is OK
- 10.1121/1.404385 is OK
- 10.1109/CDMA47397.2020.00010 is OK
- 10.1145/3144749.3144755 is OK
- 10.1109/ICASSP39728.2021.9414337 is OK
- 10.5281/zenodo.1417008 is OK
- 10.5281/zenodo.1416190 is OK
- 10.1007/s12193-015-0196-1 is OK
- 10.48550/arXiv.2302.12258 is OK
- 10.5281/zenodo.1422385 is OK
- 10.5281/zenodo.1422385 is OK
- 10.5281/zenodo.1417973 is OK
- 10.5281/zenodo.4245544 is OK
- 10.21105/joss.04554 is OK
- 10.1007/s12193-015-0196-1 is OK
- 10.5281/zenodo.7343030 is OK
- 10.1007/978-3-031-18444-4_16 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[liscio]

Review checklist for @liscio

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the https://github.com/JorenSix/Olaf?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@JorenSix) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• [x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• [x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• [x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[JorenSix]

Just an update from here, I have fixed and closed the reported issues. They were mainly about fixing instructions and offering some proper error messages when things (tests) go wrong.

I have also updated the readme. The text grew over time and is now more properly structured. This is needed since there are instructions for running Olaf on traditional computers, embedded devices and in the browser. I have also added an index which helps to navigate the (perhaps long) document.

[liscio]

Thanks for addressing those bits. Everything looks good from where I stand. 👍

[JorenSix]

Thanks @liscio for the review, I am looking forward to the comments by @ebezzam. Thanks!

[bmcfee]

Checking in - @ebezzam, will you have a chance to look at this soon?

[ebezzam]

Review checklist for @ebezzam

Conflict of interest

• [x] I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

• [x] I confirm that I read and will adhere to the JOSS code of conduct.

General checks

• [x] Repository: Is the source code for this software available at the https://github.com/JorenSix/Olaf?
• [x] License: Does the repository contain a plain-text LICENSE file with the contents of an OSI approved software license?
• [x] Contribution and authorship: Has the submitting author (@JorenSix) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
• [x] Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
• [x] Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
• [x] Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
• [x] Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

• [x] Installation: Does installation proceed as outlined in the documentation?
• [x] Functionality: Have the functional claims of the software been confirmed?
• [x] Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

• [x] A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
• [x] Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
• [x] Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
• [x] Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
• [x] Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
• [x] Community guidelines: Are there clear guidelines for third parties wishing to 1) Contribute to the software 2) Report issues or problems with the software 3) Seek support

Software paper

• [x] Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
• [x] A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
• [x] State of the field: Do the authors describe how this software compares to other commonly-used packages?
• [x] Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
• [x] References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

[ebezzam]

Checking in - @ebezzam, will you have a chance to look at this soon?

yes sorry just got back from holiday and should be able to finish by next week!

[ebezzam]

@JorenSix, @bmcfee

From my initial review, there are a couple blockers:

• Large overlap with Panako, especially in the paper content (LInes 1-14, 26-37), and there being a previous publication of Olaf. I don't know how this fits with JOSS ethics on self-plagiarism. Maybe @bmcfee can clarify?
• Given performance claims (Line 43), some quantificiation of the low memory usage.
• Given large overlap with Panako, performance comparison (I see there was one in the other Olaf paper, footnote 2). Chromaprint seems like another library worth mentioning, and interesting to compare to.

Minor:

• I think it would be more appropriate to have Footnotes 1-5 as references. Just as Olaf, these are open source libraries that should be referenced.

[JorenSix]

@ebezzam thanks for taking the time for the review!

• Indeed there is some overlap with the Panako JOSS paper, as I disclosed at the submission stage. The two projects have similarities but each have their own niche (time/frequency robustness in Panako, embedded use in Olaf). The similarities explain the overlap in the more general intro and similar references. The referenced Olaf ISMIR LBD abstract is not peer-reviewed and serves a different purpose than this paper: the LBD abstract is more technical and aimed at domain experts, while this paper introduces Olaf to a more general audience and - crucially - is peer reviewed ;).
• Measuring memory use is indeed a good idea. I have already started using valgrind to get insights and identified a memory leak in the use of the hash table! The memory use depends on the size of the database but is below 3MB in the case of using the test dataset. More on this later..
• A performance comparison seems like a good idea. I can run the same benchmark for both system and report them.
• Chromaprint serves another case: it creates a fingerprint of a full track and only matches full duplicates of tracks. It is not designed for granular queries. I can add this to text to clarify this.
• The minor comment is noted. I am not sure about the best way to reference software in latex but I will look into it.

[bmcfee]

Thanks @ebezzam !

• I don't know how this fits with JOSS ethics on self-plagiarism. Maybe @bmcfee can clarify?

I think this is fine and in accordance with the ethics guidelines. As @JorenSix notes, ISMIR Late-breaking demos (LBDs) are not peer-reviewed and are not published in the proceedings. It's quite common for preliminary work to first appear as an LBD and later be published properly (at ISMIR or elsewhere).

The minor comment is noted. I am not sure about the best way to reference software in latex but I will look into it.

If the software being cited does not have a DOI, the JOSS submission guidelines give an example of how to do this: https://joss.readthedocs.io/en/latest/submitting.html#example-paper-and-bibliography - excerpt:

@misc{fidgit,
  author = {A. M. Smith and K. Thaney and M. Hahnel},
  title = {Fidgit: An ungodly union of GitHub and Figshare},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  url = {https://github.com/arfon/fidgit}
}

[ebezzam]

Thanks @JorenSix and @bmcfee for the quick responses and clarifications. I've opened a few issues on minor things / quick fixes.

On the unchecked points of my review:

• [x] Performance: (1) low memory and (2) comparison with Panako would be great.
• [x] Installation instruction: comments in https://github.com/JorenSix/Olaf/issues/31. Moreover, a package manager option would be great.
• [x] Example usage: I think a dedicated "Examples" section could be nice in the README, where you provide a dummy folder of audio files, as you do for the tests.
• [x] State of field: While Chromaprint serves another case, as you mentioned in your response, I think it's interesting to make that distinction. Other relevant software that a quick Google search reveals are dejavu and SpectroMap.
• [x] References: having the footnotes as references.

Feature request, which I think could really enhance usability and usage, is a Python wrapper. Like this for SoX. I'll let @bmcfee weigh in if he thinks this is necessary for acceptance.

Hope that helps!

[bmcfee]

Thanks @ebezzam !

I don't think a python wrapper is necessary here, though it would of course be a nice feature at some point.

[JorenSix]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[JorenSix]

Thanks for the feedback! I have been busy and Olaf has improved quite a bit! The changes:

• I have measured memory use and reported memory use on a separate page. The readme is already long so I think the separate page makes sense. I have fixed a memory leak in the hashtable use which came to my attention during the memory use measurement. It also simplified the matcher implementation
• I have compared the performance of Panako and Olaf directly. Surprisingly the Java implementation (Panako) is slightly faster on my machine but this makes sense considering the fact that Panako is multi-threaded and has a much higher memory use. Also Java is not that slow in pure number crunching.
• The same page (eval readme) also contains info on the evaluation script which has been updated as well.
• I have improved the readme fixing several typos and included a copy-pastable example use. Also the installation instructions are more clear now and include a complete set of instructions.
• The paper has been improved by additional references to audio fingerprinting systems and a note on the memory use. The reference to software are done properly now.
• The functional tests do not show a confusing error any more.

I agree that a python wrapper and packaging (for apt, yum, pacman, ...) are two nice-to-haves which would increase adoption of Olaf but I see this as separate from the core Olaf software (as is the case for the pysox).

Thanks again for taking the time to review this paper/software and please let me know if there are additional points to tackle!

[ebezzam]

@JorenSix thanks for the updates to the repo and paper! I've opened a few "issues" on some minor points.

[JorenSix]

Thanks @ebezzam for the additional pointers. Now these small issues have been fixed and the paper and docs have improved!

[JorenSix]

@editorialbot generate pdf

[JorenSix]

Just an FYI, I have improved the speed of Olaf significantly with an algorithmic optimisation in the max-filter step. After a profiling session It became clear that this was the main speed bottleneck. Now it implements a (partial) Van Herk max filter as described in Van Herk, M. (1992). A fast algorithm for local minimum and maximum filters on rectangular and octagonal kernels. Pattern Recognition Letters, 13(7), 517-521.

The Panako vs Olaf results have changed and this is now reflected in the graphs/texts. @ebezzam , @bmcfee let me know if there are additional points or the we can move on ;).

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[bmcfee]

Thanks for the updates! I'm OOO for a few days, but will be able to check back in on this next week.

[ebezzam]

@JorenSix thanks for quickly addressing the points! Paper is looking much more complete, and great that you could resolve this bottleneck! Just one more formatting :see_no_evil: https://github.com/JorenSix/Olaf/issues/35

[JorenSix]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[JorenSix]

The formatting of the author names is fixed, the use of authors={{auhor list}} made latex quote the long author list literally. The use of authors={author list} fixed it.

Thanks again @ebezzam for spotting the issue!

[ebezzam]

thanks @JorenSix, everything checks off for me!

[JorenSix]

I have just pushed another performance improvement (SIMD and a conditional which avoids a performance bottleneck) together with new Panako vs Olaf results. Olaf is about ten times faster than before the performance improvements, while all the rest remained the same. See here for more background on the performance optimizations.

@bmcfee It would be great to be able to round things off :)

[bmcfee]

@editorialbot generate pdf

[bmcfee]

@editorialbot check references

[editorialbot]

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

OK DOIs

- 10.1109/MMSP.2002.1203273 is OK
- 10.5281/zenodo.1416340 is OK
- 10.5281/zenodo.1417593 is OK
- 10.1109/TASL.2012.2234114 is OK
- 10.1007/978-3-319-73165-0_10 is OK
- 10.1007/s11265-005-4151-3 is OK
- 10.5281/zenodo.1414728 is OK
- 10.1121/1.400476 is OK
- 10.1121/1.404385 is OK
- 10.1109/CDMA47397.2020.00010 is OK
- 10.1145/3144749.3144755 is OK
- 10.1109/ICASSP39728.2021.9414337 is OK
- 10.5281/zenodo.1417008 is OK
- 10.5281/zenodo.1416190 is OK
- 10.1007/s12193-015-0196-1 is OK
- 10.48550/arXiv.2302.12258 is OK
- 10.5281/zenodo.1422385 is OK
- 10.5281/zenodo.1422385 is OK
- 10.5281/zenodo.1417973 is OK
- 10.5281/zenodo.4245544 is OK
- 10.21105/joss.04554 is OK
- 10.1007/s12193-015-0196-1 is OK
- 10.5281/zenodo.7343030 is OK
- 10.1007/978-3-031-18444-4_16 is OK
- 10.1007/978-3-031-07015-0_16 is OK

MISSING DOIs

- None

INVALID DOIs

- None

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[bmcfee]

Post-Review Checklist for Editor and Authors

Additional Author Tasks After Review is Complete

• [x] Double check authors and affiliations (including ORCIDs)
• [x] Make a release of the software with the latest changes from the review and post the version number here. This is the version that will be used in the JOSS paper.
• [x] Archive the release on Zenodo/figshare/etc and post the DOI here.
• [x] Make sure that the title and author list (including ORCIDs) in the archive match those in the JOSS paper.
• [x] Make sure that the license listed for the archive is the same as the software license.

Editor Tasks Prior to Acceptance

• [x] Read the text of the paper and offer comments/corrections (as either a list or a PR)
• [x] Check the references in the paper for corrections (e.g. capitalization)
• [x] Check that the archive title, author list, version tag, and the license are correct
• [x] Set archive DOI with @editorialbot set <DOI here> as archive
• [x] Set version with @editorialbot set <version here> as version
• [x] Double check rendering of paper with @editorialbot generate pdf
• [x] Specifically check the references with @editorialbot check references and ask author(s) to update as needed
• [x] Recommend acceptance with @editorialbot recommend-accept

[bmcfee]

Thanks again @ebezzam and @liscio for your reviews!

@JorenSix - the bot has created a post-review checklist for us to complete before the paper is accepted. (The steps should be familiar from your previous submission, but now we have a way to track them!)

Let me know as you complete the "author tasks" steps on your side, and I'll get to work on the editor tasks.

[JorenSix]

See below:

• [x] Double check authors and affiliations (including ORCIDs)
• [x] Make a release of the software with the latest changes from the review and post the version number here. See the Olaf v1.1.0 release on GitHub
• [x] Archive the release on Zenodo/figshare/etc and post the DOI here:
• [x] Make sure that the title and author list (including ORCIDs) in the archive match those in the JOSS paper.
• [x] Make sure that the license listed for the archive is the same as the software license.

Also from me a thanks to @ebezzam and @liscio for the constructive feedback. It certainly improved Olaf!

[bmcfee]

I read through the paper, and have a few comments:

• Line 38: "MIR" used without definition
• Line 43: "similarly specced" could be more precisely worded
• Line 72: "file handling and transcoding is" → "... are"

General comment: in the bibliography entries, many parts that should be capitalized are not, e.g., "Lmdb". You can force proper capitalization by embedding the appropriate parts in curlies, eg, "{LMDB}: A general-purpose ..." .

Citation for Lòpez-García et al., seems to have an awkwardly formatted DOI (ie it's URL-encoded). Please check this one.

[bmcfee]

@editorialbot set v1.1.0 as version

[editorialbot]

Done! version is now v1.1.0

[bmcfee]

@editorialbot set 10.5281/zenodo.8093527 as archive

[editorialbot]

Done! archive is now 10.5281/zenodo.8093527

[JorenSix]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[JorenSix]

@editorialbot generate pdf

[editorialbot]

:point_right::page_facing_up: Download article proof :page_facing_up: View article proof on GitHub :page_facing_up: :point_left:

[JorenSix]

@bmcfee Thanks for the pointers.

I fixed the capitalization issues in the bibliography (PFFFT, LMDB, inconsistent title capitalization), the DOI, introduced MIR and expanded on the microcontroller specs.

[bmcfee]

Thanks @JorenSix - i just noticed one more bib entry with a url-encoded DOI (Cortès et al). Everything else looks good to me.