Pre-submission enquiry - MAPAS

[npetrov937]

Hi,

My colleagues and I believe that there is a gap in the statistical software world. Specifically, to our awareness, there is no User Interface-driven, free, quick and easy-to-use software that allows to simultaneously apply a range of multiple test correction procedures (10 in our case) and then format the results into APA-styled tables. The problems with that are manifold.

First, there is a worrying lack of application of multiple test correction procedures:

• Neuroimaging - in papers published in 2008, between 15% and 40% of studies failed to include adjustments for multiple testing
• Pathology - out of 800 papers published in 2003, 37 papers performed multiple comparisons and 56% failed to account for that.
• Optometry - out of 142 papers between 2003 and 2013, 33% failed to correct for multiple comparisons.
• Psychology - out of all papers published between 1985 and 2013 in 8 major journals, merely 0.3% contained the string "Bonferroni"

Second, we think the reasons for this lack of application can be laid at the feet of the demands on time and effort for scientists:

• Developing research software takes 30% of scientists' time on any given research project. This is exacerbated by the fact that 56% of researchers are developing their own software in their research, and, alarmingly, 21% had never had formal training.
• 26% of scientists rely on SPSS or Excel for analyses, both of which do not allow easy - if at all - application of multiple test correction procedures
• The current ways to apply a single multiple test correction procedure in Excel are time-consuming, inconvenient and non-replicable.

To address these problems, we are developing open-source, Python-based software that accepts multiple types of input (raw data, summary statistics, SPSS tables or a list of p values) and allows the user to apply multiple test corrections and saves the output in APA-styled tables.

You can see the interface of the software for one option on the image below or you can download a beta version using the links below. Please note that given that it's still a beta version, there are known bugs, which will be addressed by the time of submission.

See here for sample input/output files.

We would be grateful to know your opinion of whether an article describing this software is within the remit of the Journal of Open Source Software.

[arfon]

Hi @npetrov937 - thanks for this enquiry. This may be in scope for us but I'm not able to see the software anywhere here (there seems to be a rar file linked but no source code available somewhere browsable online).

One minor thing:

Please note that given that it's still a beta version, there are known bugs, which will be addressed by the time of submission.

Please don't submit to JOSS in the hope that reviewers will flag all of the known bugs. If you know of bugs, please fix them before submitting. The review process may uncover further bugs as a natural part of the review process but I don't believe it is appropriate to ask reviewers to identify bugs you already know about.

We would be grateful to know your opinion of whether an article describing this software is within the remit of the Journal of Open Source Software.

Possibly, but I suspect this is a bit of a stretch for us. We describe research software as follows:

JOSS publishes articles about research software. This definition includes software that: solves complex modeling problems in a scientific context (physics, mathematics, biology, medicine, social science, neuroscience, engineering); supports the functioning of research instruments or the execution of research experiments; extracts knowledge from large data sets; offers a mathematical library, or similar.

Software that helps researchers apply a statistical test could fit within here (possibly under supports the functioning of research instruments or the execution of research experiments or extracts knowledge from large data sets).

[npetrov937]

Hi @arfon ,

Thank you for your reply; it is greatly appreciated.

In retrospect, I have done a terrible job highlighting the fact that the software was in beta version at the time of writing the above enquiry. I would like to assure you that even as of now, the software is substantially expanded and improved. It also has undergone extensive unit testing (will be made available along with the source code by the time of submission) and the results of any statistical operations are calibrated against SPSS Statistics as well as R.

Your clarification on what constitutes research software is also greatly appreciated. I believe that our software will fit under this definition.

Many thanks, Nikolay

[arfon]

OK thanks @npetrov937. I think you should go ahead and submit this.