nice_fit() index cutoffs

[TDJorgensen]

For review: openjournals/joss-reviews#5701

When nice_fit(..., nice_table = TRUE), there is a bottom row of "ideal values", with a table footnote indicating they were "proposed by Schreiber (2017)". This is inaccurate in 2 ways:

1. Schreiber (2017) did not propose any cutoffs, but rather cited the source of fit-index cutoffs as Hu & Bentler (1999), who actually suggested RMSEA < .06, whereas your table suggests both < .05 and a range extending to .08, which originate from different sources:
   • Browne & Cudeck (1992) proposed RMSEA < .05 indicates close fit, RMSEA < .08 indicates reasonable fit, and RMSEA > .10 indices poor fit
   • MacCallum, Browne, & Sugawara (1996) proposed RMSEA between .08–.10 indicates mediocre fit
2. Schreiber (2017, p. 640) explicitly did not advocate the cutoff values in the table:

Many researchers grew up with the combination suggestion of SRMR less than or equal to .08 and CFI greater than or equal to .95 (Hu & Bentler, 1999). This has not worked out in simulations (Sivo et al., 2006).

If anything, Schreiber's very next paragraph "proposed" using SRMR < .10 as a cutoff.

I do not recommend including cutoffs in the table, as doing so would perpetuate their misuse. Fit indices are not test statistics, and their suggested cutoffs are not critical values associated with known Type I error rates. Numerous simulation studies have shown how poorly cutoffs perform in model selection, including one of my own. Instead of test statistics, fit indices were designed to be measures of effect size (practical significance), which complement the chi-squared test of statistical significance. The range of RMSEA interpretations above is more reminiscent of the range of small/medium/large effect sizes proposed by Cohen for use in power analyses, which are as arbitrary as alpha levels, but at least they better respect the idea that (mis)fit is a matter of magnitude, not nearly so simple as "perfect or imperfect."

If you insist on including cutoffs in the table, I recommend:

• including multiple suggestions (e.g., in multiple rows), with multiple subscripts that accurately link each suggestion to its original proposal. For example, Bentler & Bonett (1980) proposed incremental fit indices > .90 were acceptable.
• calling them "suggested cutoffs" instead of "ideal values".

[rempsyc]

This is very useful, thank you. It probably is the most critical for publishing the paper given that it includes tables with this footnote, and I would not want to crystalize bad practice in a paper such as this one.

First of all, may I have your permission to include the following quote from you (with your name, if you agree) in the documentation of nice_fit so people know what they are in for? And also including the Jorgensen et al. (2018) reference in the corresponding section.

I do not recommend including cutoffs in the table, as doing so would perpetuate their misuse. Fit indices are not test statistics, and their suggested cutoffs are not critical values associated with known Type I error rates. Numerous simulation studies have shown how poorly cutoffs perform in model selection (e.g., , Jorgensen et al. (2018). Instead of test statistics, fit indices were designed to be measures of effect size (practical significance), which complement the chi-squared test of statistical significance. The range of RMSEA interpretations above is more reminiscent of the range of small/medium/large effect sizes proposed by Cohen for use in power analyses, which are as arbitrary as alpha levels, but at least they better respect the idea that (mis)fit is a matter of magnitude, not nearly so simple as "perfect or imperfect."

Second, I must admit that the cutoff guidance is a highly requested and appreciated feature, but at the same time, I am sensitive to the idea that such cutoff values should not be taken too seriously as demonstrated by simulation studies and so on (but does it have any value for people who have not reference at all?). I like your idea of using "suggested cutoffs" instead of ideal values.

What about "Suggested soft cutoffs" or simply "Commonly used reference values" to convey this idea even more strongly? Do you have a preference for any of the three? Or another suggestion still?

Besides, I will correct the references and cutoffs and think about a way to include multiple sources.

[rempsyc]

For reference, I relied on Table 3, with a column (rather boldly) called "General rule for acceptable fit if data is continuous":

This is where I got the < .05 [.00, .08] RMSE reference value. So I feel like it should be ok to cite this source as Table 3, and people can check the other references from that source if needed? Interestingly, Schreiber does not cite the references directly from the table either. I also like the idea of having a single reference source that people can look up for more information and references in the original literature.

[rempsyc]

OK, so for now I have:

1. Changed "Ideal value" for "Suggested soft cutoffs"
2. Specified that it is based on Schreiber (2017), Table 3, and not "proposed by".
3. Added an argument cutoffs so people can decide whether to include the reference row or not, based on their preference.

That said, I do feel like the word "cutoff" is more binary than ideal value, which in the way I was seeing it, was more expressive of the continuous nature of the fit indices while stile pointing the way toward more preferable values.

[TDJorgensen]

I also like the idea of having a single reference source that people can look up for more information and references in the original literature.

The problem is that Schreiber was doing the same thing you are doing (providing a quick summary of many different original sources), but only for a particular field. And he didn't seem to summarize accurately or with adequate references to original sources. So I wouldn't rely on this as your citation.

I feel like it should be ok to cite this source as Table 3, and people can check the other references from that source if needed?

I don't even think it is necessary to provide a citation in the table, because writers should be discussing their own decision rules in their actual text, where citations can be more thorough than in a table note. My recommendation for you is to include more detail in your nice_fit() documentation, where there is room for more thorough citation of original sources. You could have a bullet-point list in the Details section, where you list and cite the full range of interpretations for RMSEA, the .90 and .95 guidelines for incremental fit indices, etc.

I do feel like the word "cutoff" is more binary than ideal value, which ... was more expressive of the continuous nature of the fit indices

I agree, but the "ideal" value is 1 for goodness-of-fit indices (CFI and TLI) and 0 for badness-of-fit indices (RMSEA and SRMR). How about "Common guidelines" for the bottom row? And it can include alternatives (e.g., "> .90 or .95" for CFI), but let the readers find the details in your documentation. If you want to be sure to draw their attention to it, then the nice_fit() function could print a message() when both cutoffs=TRUE and nice_table=TRUE, e.g.,

if (cutoffs && nice_table) {
  message("See the ?nice_fit help-page Details for more information about guidelines")
}

[rempsyc]

Ok thanks, reopening this issue so I can come back to it and fix in the future (since it will take a bit of time to scan the literature, etc.).