philippgadow
commented
12 months ago Dear @jpata,
thank you very much for your careful review of the software, its documentation and the associated paper draft. We apologise for our long delay in answering the comments.
The new version of Umami has much improved documentation which hopefully should address the issues you raised. We are grateful for the issues, in hindsight we should have provided a tutorial / worked out example how to operate on non-ATLAS data from the start as this will make the software more useful for users outside of the ATLAS collaboration.
I have answered the related issues you raised and closed those where I had the impression that the issue and response were clear to not require discussion. If this assumption was not justified, please re-open the issues and I'd be happy to re-iterate.
Cheers, Philipp for the Umami developers
jpata
It's really great to see that the development of mostly collaboration-internal tools can be done in the open, with public artifacts such as the JOSS paper, tests, codebase etc. Congratulations on the significant amount of work it has taken to bring things to this level!
The paper is well written and there are no major issues. Here I will leave the few items that came up on my side in the JOSS review.
The paper contains some plots, and it's mentioned it's easy to reproduce them using open JetClass data, but it's not entirely clear if there's a ready-made configuration file for this. It would be great to provide some example config that is relevant for and usable by a non-ATLAS person. In the documentation, it might be useful to be more explicit what parts require access to internal data.I was now able to reproduce the plots with the provided tutorial file and docker image (2024-01-24).related to the point above, can be done once some documentation or clear example is available on how to use the JetClass dataset (or some other open dataset)The functional claims can be validated with the provided tutorial (2024-01-24).as aboveNot unusually for a big HEP software package, it's not entirely clear from the documentation what constitutes the external user-facing API of the project. Is it just the configuration files, or also some of the python functions? For example, I see that some of the plotting functionality is currently being spun off to an external package. It might be useful to clarify what (if any) is the user-facing API beyond the configurations.A comprehensive API reference is now linked from the documentation.some clear guidelines should be added to tick this item. Some items to consider: are contributions from outside the collaboration relevant / considered? Should issues and PRs be created on github or gitlab? Is github simply a mirror, or is development expected to happen here as well?Contribution guidelines have been added to the documentation.Once these are addressed, I will be happy to recommend this for publication in JOSS, and also as a milestone / reference point for future software tool development in HEP collaborations.