search

rcpch / rcpch-audit-engine

Epilepsy12 Audit Platform
https://e12.rcpch.ac.uk/
GNU Affero General Public License v3.0
5 stars 5 forks source link

E12 team review of KPI validation/testing logic #502

Closed pacharanero closed 11 months ago

pacharanero commented 1 year ago

The logic we are testing in each of the test files is written in (fairly) plain English in the commented sections at the top of each of the testing files. To keep this documentation close to the code, and preserve one 'source of truth' then we would like the E12 team @nikyraja @AmaniKrayemRCPCH @coldunk @FridahC to review these sections and comment on any that don't look right to them.

If in practice this is harder than it seems then we could alternatively have a meeting session in which we go through the detail of each KPI test logic, one-by-one, all together, reading the text out loud, and inviting comments/thoughts.

The links to the relevant files to check are:

The only bit that the E12 team need to read is the section at the top of each file within the triple quoted comments

"""
eg Measure 1 - paediatrician_with_expertise_in_epilepsies
"""

If you notice a problem with any of the lines, click on the line number and use the small ... menu which pops up to the left to create a new Issue which will reference the actual line and link directly to it. We can then discuss and work through the Issues as needed.

AmaniKrayemRCPCH commented 1 year ago

Thanks! Can I confirm that for measures 1, 2, 4, 5, 6, 7, 8, 9A, 9B, and 10, patients are ineligible if they are not diagnosed with epilepsy?

https://www.rcpch.ac.uk/sites/default/files/2022-09/epilepsy12_round_4_methodology_overview.pdf

I have a couple more comments that I'll share once I confirm them with the team.

eatyourpeas commented 1 year ago

There should no longer be any children in the audit that do not have epilepsy. They cannot complete the multiaxial diagnosis form either unless at least one episode is epileptic

I think there is a check at the beginning to make sure they are registered and have agreed to the criteria before being included in the KPIs, but it should be academic as they should never get beyond the registration step

From: Amani Krayem @.> Date: Monday, 5 June 2023 at 16:34 To: rcpch/rcpch-audit-engine @.> Cc: Subscribed @.***> Subject: Re: [rcpch/rcpch-audit-engine] E12 team review of KPI validation/testing logic (Issue #502)

Thanks! Can I confirm that for measures 1, 2, 4, 5, 6, 7, 8, 9A, 9B, and 10, patients are ineligible if they are not diagnosed with epilepsy?

https://www.rcpch.ac.uk/sites/default/files/2022-09/epilepsy12_round_4_methodology_overview.pdf

I have a couple more comments that I'll share once I confirm them with the team.

— Reply to this email directly, view it on GitHubhttps://github.com/rcpch/rcpch-audit-engine/issues/502#issuecomment-1577029408, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAICOV4EIPSODREPVXQDQA3XJX4A5ANCNFSM6AAAAAAYYOCILE. You are receiving this because you are subscribed to this thread.Message ID: @.***>

nikyraja commented 1 year ago

Thanks Simon, you are correct! The eligibility criteria means that only CYP with epilepsy will be registered onto the platform, and clinicians need to confirm a diagnosis is in place before they can proceed to the rest of the form. So no need to factor this into the KPI calculations.

AmaniKrayemRCPCH commented 1 year ago

Hello! Looks good. We have a few comments.

Measure 2 - You said: PASS IF input from/referral to ESN by first year Should be: PASS IF input from an ESN by first year

Measure 3 - You said : PASS IF criteria met AND referred/seen by neurologist OR referred/seen by epilepsy surgery OR both Should be: PASS IF criteria met AND (seen by neurologist OR referred to epilepsy surgery OR both)

Measure 8 - You said: PASS IF female AND >12yrs AND on valproate AND annual risk acknowledgement form is completed AND pregnancy prevention programme is in place Should be: PASS IF female AND >12yrs AND on valproate AND (annual risk acknowledgement form is completed OR pregnancy prevention programme is in place OR both)

Measure 9A - You said: PASS IF individualised care plan in place AND care plan updated when necessary Should be: PASS IF individualised care plan in place AND evidence of agreement AND updated when necessary

Measure 9Ai - You said: PASS IF care plan is in place AND evidence of agreement AND updated when necessary Should be: PASS IF care plan is in place

Hopefully what I've written makes sense. Let me know if you have any questions.

anchit-chandran commented 1 year ago

Re measure 2 -> I'm assuming a patient has to have had a Referral, in order to get input from an ESN. So the only passing case we test is:

Is this fine? @eatyourpeas @AmaniKrayemRCPCH

eatyourpeas commented 1 year ago

I think this issue is now handled - there is a full suite of tests which look at KPI calculations. So long as @AmaniKrayemRCPCH is happy with the methodology, these are rules that we are using and the tests are set against. If @AmaniKrayemRCPCH is able to confirm the rules as we have them are correct, I will close this.

AmaniKrayemRCPCH commented 1 year ago

Heya,

Noticed a couple things. https://github.com/rcpch/rcpch-audit-engine/blob/live/epilepsy12/tests/common_view_functions_tests/calculate_kpi_tests/test_measure_3.py - to pass, they need to be seen within the first year of care (by either epilepsy surgery or neurology). Timeliness is not part of the calculation for 3b

https://github.com/rcpch/rcpch-audit-engine/blob/live/epilepsy12/tests/common_view_functions_tests/calculate_kpi_tests/test_measure_8.py - They pass if they have either a annual risk acknowledgement form OR a pregnancy prevention programme (or both). They don't need both to pass. (this might be worded incorrectly in our performance indicator table)

anchit-chandran commented 1 year ago

Measure 3 - You said : PASS IF criteria met AND referred/seen by neurologist OR referred/seen by epilepsy surgery OR both Should be: PASS IF criteria met AND (seen by neurologist OR referred to epilepsy surgery OR both)

hey amani, going through these now.

Wanted to confirm the change is 100% correct? decreasing the bar for pass to at least one of: 1) an input date for neurologist is entered (with current UI set up, this means input date must be entered, otherwise audit incomplete) 2) a referral date for neurologist is entered (as above, means input date must also be entered)

This seems different to what I thought the original spec was:

Number of children ...WHO HAD CRITERIA... AND had [evidence of referral or involvement of a paediatric neurologist] OR [evidence of referral or involvement of CESS]

FYI updated all the rest, they made sense!

AmaniKrayemRCPCH commented 1 year ago

Hi Anchit,

So they need to be either

  1. receive input from a neurologist within 1 year from first paediatric assessment (not referral)
  2. be referred to epilepsy surgery (regardless of if they've had input, as waiting times for surgery can be years)

Hopefully that makes sense. The original spec was an early draft and the PIs were updated since then - sorry about the confusion.

Are you saying that the date of neurology input is necessary for the form to be completed? This should be an optional field, as not all patients will get input by the audit deadline.

eatyourpeas commented 1 year ago

Thanks @AmaniKrayemRCPCH All dates have been mandatory from the start. We did make changes to the EEG dates on the basis that some children refuse EEG following the review that Frida made of the dates. Nothing was said about the other dates.

This is actually quite a big change to implement, since it touches the logic in the form scoring, the KPI calculation and the tests that check all these things as well. Are we now saying that the second date in all date pairs (gen paeds referal, neurology referral, surgery referral, nurse specialist referral) are optional and not mandatory?

@pacharanero listed out all the KPIs above as a check list: could you please tick off the ones you are happy with, so that we know which ones are not accurate?

anchit-chandran commented 1 year ago

Hey Amani, as @eatyourpeas said, this is a significant change

One big danger is not having a Single Source of Truth now, with multiple sources all saying different things. I propose the only source of information re KPI calculations being these files below. Please could you confirm these are all correct (again just looking at the """docstrings"""), instead of Marcus' list above? From now on, we will only refer to these.

Image

AmaniKrayemRCPCH commented 1 year ago

Yep you're right. Niky and I had this conversation yesterday - these calculations are outlined in too many places with too many inconsistencies.

I've updated our methodology overview to be our internal source of truth, which should be correct, although please let me know if you have any questions - https://www.rcpch.ac.uk/sites/default/files/2023-08/Epilepsy12%20Round%204%20Methodology%20Overview_20230803.pdf

All of the KPIs linked look correct! Thanks again for checking

anchit-chandran commented 1 year ago

Make md table on docs site with code testing

pacharanero commented 1 year ago

All of the KPIs linked look correct! Thanks again for checking

closing

nikyraja commented 11 months ago

Hello,

Following our discussion on Monday, Hannah has been updating the KPI methodology to improve the robustness. Subsequently, the logic for KPI 9b here needs updating. From line 123 here: https://github.com/rcpch/rcpch-audit-engine/blob/development/epilepsy12/common_view_functions/calculate_kpi_functions/score_kpi_9.py

I've attached the STATA code here for info (2 = Yes/True in all variables except the defined_child_epi variable, where 1 = epilepsy within the audit).

9b STATA code

We had a go at updating the PASS criteria, but this may need reviewing as we are not experts in python!

pass_criteria1 = [
(rescue_medication_been_prescribed if False), (management.individualised_care_plan_include_first_aid is True), (management.individualised_care_plan_addresses_water_safety is True), (management.individualised_care_plan_includes_service_contact_details is True), (management.individualised_care_plan_includes_general_participation_risk is True), (management.individualised_care_plan_addresses_sudep is True)]

pass_criteria2 = [
(rescue_medication_been_prescribed if True), (management.individualised_care_plan_parental_prolonged_seizure_care is True), (management.individualised_care_plan_include_first_aid is True), (management.individualised_care_plan_addresses_water_safety is True), (management.individualised_care_plan_includes_service_contact_details is True), (management.individualised_care_plan_includes_general_participation_risk is True), (management.individualised_care_plan_addresses_sudep is True)]

Please let us know if you have any questions or need any more info. Thanks!

anchit-chandran commented 11 months ago

@eatyourpeas

eatyourpeas commented 11 months ago

Thanks @nikyraja and Hannah. I have made these changes and written a test to match:

# score kpi
    base_pass_criteria = [
        (management.individualised_care_plan_include_first_aid is True),
        (management.individualised_care_plan_addresses_water_safety is True),
        (management.individualised_care_plan_includes_service_contact_details is True),
        (
            management.individualised_care_plan_includes_general_participation_risk
            is True
        ),
        (management.individualised_care_plan_addresses_sudep is True),
    ]

    pass_1_criteria = [
        (management.has_rescue_medication_been_prescribed is False),
    ] + base_pass_criteria

    pass_2_criteria = [(management.has_rescue_medication_been_prescribed is True),
        (management.individualised_care_plan_parental_prolonged_seizure_care is True)] + base_pass_criteria

    if all(pass_1_criteria) or all(pass_2_criteria):
        return KPI_SCORE["PASS"]
    else:
        return KPI_SCORE["FAIL"]

If you are happy with this I will merge and close

nikyraja commented 11 months ago

Thanks @eatyourpeas ! This looks correct, so happy for you to merge this into live and close the issue