Upcoming Making Data Count change to two-sigma rule logic used in tools

[MatthewCallus]

Hi all.

Just a heads-up that we will be changing the two-sigma rule logic soon. Currently only points between two- and three-sigma are counted, so if you had one point in this range and then the next point beyond three-sigma (i.e. a process limit), this would not be sufficient to trigger the rule (although the second point would trigger the astronomical point rule obviously). The change will mean that all points beyond two-sigma are counted, so both points above would trigger the rule. As before, all three points still need to be on the same side of the mean, and the third point would not trigger this rule if it is not beyond two-sigma itself (i.e. only between the mean and the two-sigma in question).

I am currently producing a document with multiple datasets to demonstrate all the logic used in our single chart tool with edge cases for others to test their tools produce the same results. You may find this useful. It should be ready soon – I think this month sometime, once we've released the updated tool.

[ThomUK]

Thanks @MatthewCallus! Just linking this with issue #153, which was opened to discuss this type of edge-case.

The example data would be very useful in due course - we'll use it in the unit tests to confirm the new behaviour is as expected (and stays that way over time). We'll attach the code changes to this issue (rather than re-open the closed one).

[alexmcqueen]

@MatthewCallus do you know when this will happen? We have been looking at the new Custom SPC done by West Australian Health Department’s Healthcare Quality Intelligence Unit and in tesitng noticed we are off on this rule. I am concious that changing this rule will mean us and a lot of other providers will need to go change their own code bases to ensure everything is the same. Is there some comms on Futures about it already?

[MatthewCallus]

Hi @alexmcqueen. I imagine we'll update through our monthly Analyst Network meetings. Nothing's been shared yet (besides on here and Slack) as we're still making changes. We'll also share the above-mentioned document, which will explain the rules with examples.

The single and multi Excel tools have been updated, and I'll be updating our SQL script soon. There's been a delay in this and the document as we are in the process of changing our colour palette and ensuring the trend rule is applied consistently across our tools.

There's always been a level of variation between tools, even our own. Some users will continue to use older versions; some will use other tools such as the R package, which removes moving range outliers by default; some will create their own tools with slightly different logic; some will apply other Nelson rules. The main thing is that the right signals are shown and investigated.

[alexmcqueen]

Thanks for the info @MatthewCallus. We look forward to the updates. In terms of using the tools at a provider and system level; we really need ours tools to output the same results, otherwise customers etc will query why they don't. That means we will need to align everything to whatever the output of the Custom SPC Visual in Power BI finally does as that is the only thing we cannot replicate. Ideally that will be what you guys at making Data Count decide to do as we want to aligin with you. May I ask why you're changing the two-sigma rule?

[ThomUK]

@alexmcqueen issue #153 is relevant here, and covers a couple of cases where 2-3 sigma is used (as current) instead of >2 sigma (as proposed). This R package aims to follow the NHSE guidance as closely as possible, but we can't compel maintainers of other projects (eg. The PBI project) to do the same. It's worth mentioning that @MatthewCallus is in the MDC team, so is giving advance notice of how this edge-case will be handled by their tools in future, partly to help us stay in sync.

[alexmcqueen]

@ThomUK thanks for your comments. Think we are just trying to ascertain who the PBI project are getting their information from in terms of rule implementation. An issue I raised over there has been passed over to here but we have been informed it was all done by MDC. This helps us understand what could be a bug versus what is by design; i.e. 2 out of 3 rule implementation.

[MatthewCallus]

That makes sense, @alexmcqueen. I have checked the Power BI custom visual for differences uses the data in the document and this is what I found (which I will also include in the document):

• The Single Chart tool shows process limits and summary icons (except for one fix we need to make) at 13 points and above, but the custom visual does show from 2+ points.

• The two- to three-sigma rule includes the third point regardless of where it is, whereas the Single Chart tool requires all three are on the same side of the mean, and only highlights the points beyond two-sigma. This means that if the second and third point is beyond two sigma in the custom visual, the first and fourth will always be highlighted too. There is an option to not highlight all points in the pattern but this does not work correctly.

• When special cause is triggered for both improvement and concern, this conflict is handled differently. In the Single Chart tool, improvement is shown. I couldn't quite figure out what logic is used in the custom visual.

• I could not see how to set a baseline in the custom visual.

I know others in the MDC team have been in discussion with the custom visual team, but I'm not sure about plans for total alignment.

Although the document is not yet finished (we need to update our tools first), I can share it if interested. Can also share the Power BI file with the custom visual replicating the charts in the document, which shows the differences.

I'll be moving on from the MDC team at the end of the year, but will get this document as finished as possible. The team inbox (england.improvementanalyticsteam@nhs.net) will be able to assist further from the new year.

[MatthewCallus]

And as for changing the two-sigma rule, the change makes sense to me, but I'm not sure where it's come from. Someone in the team probably has a reference they can share, if you want to contact us on the email above.

[alexmcqueen]

Thanks @MatthewCallus. Would be good to get those docs if possible; my NHS e-mail is alex.mcqueen@nhs.net. Good luck with your new endeavours in the new year.