Closed matt-graham closed 2 months ago
Thank you for the feedbacks, we are tackling them.
Add state of the field comparison to other comparable packages For example chaospy allows computing sensitivity indices as does SALib, and it looks like the package sensitivity might also be relevant.
We will try to rephrase some things to make this clearer. This package's main argument is visualisation. This is totally complementary to the packages that you mention and not at all to be compared with.
SALib, which I am a maintainer of, is the most complete package in Python for computing indices. Here we have one extra method to compute indices, but it's theoretically, and asymptotically, Sobol' indices. i.e. the same as what is in SALib and SciPy (recent version, I am a maintainer and the one who added it to SciPy with help from even Saltelli, Kucherenko and Owen). I am not talking about the others are not in the same league IMHO. OpenTURNS if you want to cite something else would be better.
The value proposition here is the visualisation method. It's not just a tool to show things, but a tool providing new insights and analysis.
@matt-graham we addressed your points, thanks again. What remains is providing a script to reproduce the figure itself. Moving this to the other issues.
Thanks @tupui, changes looks good. I've added a couple of small suggested changes to the updated text on #23.
Sorry missed this part of your response previously:
SALib, which I am a maintainer of, is the most complete package in Python for computing indices. Here we have one extra method to compute indices, but it's theoretically, and asymptotically, Sobol' indices. i.e. the same as what is in SALib and SciPy (recent version, I am a maintainer and the one who added it to SciPy with help from even Saltelli, Kucherenko and Owen). I am not talking about the others are not in the same league IMHO. OpenTURNS if you want to cite something else would be better.
Being 'in the same league' is very subjective and I would say not of relevance here. There is an explicit requirement as part of the review checklist to check that the paper 'describe[s] how this software compares to other commonly-used packages' which I still think the current paper does not satisfy completely. I would also say only citing other packages for which you are a developer is not ideal. The examples I gave were not meant to be prescriptive and agree that OpenTURNS looks like it would be another relevant package to compare to, though I also think chaospy
falls in this category. Note that what you said in
The value proposition here is the visualisation method. It's not just a tool to show things, but a tool providing new insights and analysis.
would itself be a useful comparison point to put in paper to make clear what SimDec brings over existing available packages.
Being 'in the same league' is very subjective and I would say not of relevance here
Not relevant for this paper yes. Still, SALib and SciPy are the references in Python. Like sensobol would be a reference in R. This is not an opinion but a fact. See https://www.sciencedirect.com/science/article/pii/S1364815224000380
I would also say only citing other packages for which you are a developer is not ideal.
Well, I contributed to almost all of them in the Python space. Either by creating issues, talking privately to maintainers, or at workshops. This is a small space and we all know each other. Both on the dev side and the research side.
OpenTURNS, I made my PhD with it and interacted with the devs and collaborated on a few projects with them (see my Batman paper in JOSS for a quick glance.)
chaospy is only providing Sobol' indices as an output from the Polynomial Chaos decomposition. i.e. not at all the focus of the project. It's similar to scikit-learn which proposes to get feature importance with a Random Forest.
would itself be a useful comparison point to put in paper to make clear what SimDec brings over existing available packages.
The text now reads:
Traditional methods to analyse the uncertainties focus on quantitative methods to compare the importance of factors, there is a large body of literature and the field is known as: Sensitivity Analysis [...] Simulation Decomposition or SimDec moves the field of SA forward by supplementing the computation of sensitivity indices with the visualization of the type of interactions involved, [...] In short, SimDec is a hybrid uncertainty-sensitivity analysis approach that reveals the critical behavior of a computational model or an empirical dataset.
Without rewriting an article about SimDec, I don't see what is missing here to say that the visualisation is the new thing. No other package has it because none of them is focused on visualisation. I don't see what it would bring to the discussion to add the list of packages about SA and say they don't do it. It's out of their scope, nothing more.
Raising as part of JOSS review openjournals/joss-reviews/issues/6713
Some assorted changes I would recommend making to the JOSS paper
https://simdec.io
pointing to web dashboard a hyperlinkchaospy
allows computing sensitivity indices as doesSALib
, and it looks like the packagesensitivity
might also be relevant.