Validating SCMD + DDD data quality with profiling tools

[robinyjpark]

I started exploring the merged SCMD and DDD data to track the completeness of each field, flag products with missing DDD and counted how many hospitals appeared in the data by month. The notebook can be viewed here.

I used ProfileReport from pandas_profiling to automatically generate summaries of each column to check for completeness. I chose this tool as it seems to allow automatic generation of expectations that can be fed into Great Expectations to test whether the field values are sensible (documentation; more investigation to be done).

The notebook also contains a distinct list of products with missing DDD (6,756 products), and a table and interactive plot displaying the number of hospitals per month.

[Jongmassey]

Nice work.

A deeper dive into some of these missing DDDs has shown some curious examples...

Only 2of 14 injectable Vancomycin VMPs (identified by ATC==J01XA01 - whocc page for this code ) has a DDD.

The ones with a DDD figure are

• 324337001 : Vancomycin 500mg powder for solution for infusion vials
• 324333002 : Vancomycin 1g powder for solution for infusion vials

And those without are

• 38345011000001109: Vancomycin 1.5g/530ml infusion bags
  • 38345111000001105: Vancomycin 2g/540ml infusion bags
  • 35055311000001104: Vancomycin 1.5g/500ml infusion bags
  • 35055911000001103: Vancomycin 2g/500ml infusion bags
  • 35051211000001109: Vancomycin 750mg/250ml infusion bags
  • 21276711000001101: Vancomycin 500mg/100ml infusion bags
• 35048511000001100: Vancomycin 10mg/2ml solution for injection pre-filled syringes
• 35049111000001102: Vancomycin 50mg/10ml solution for injection pre-filled syringes
• 21277911000001101: Vancomycin 4mg/2ml solution for injection pre-filled syringes
• 21272811000001108: Vancomycin 30mg/3ml solution for injection pre-filled syringes
• 21273411000001102: Vancomycin 20mg/4ml solution for injection pre-filled syringes
• 35050611000001101: Vancomycin 1.25g/250ml infusion bags

In the DDD table, only the former two have a BNF code (5010700). Not sure if this is related.

The omission of a DDD for the latter set feels like an omission from TRUD. Thoughts @brianmackenna @richiecroker @orlamac ?

[Jongmassey]

Further to my previous comment, I've done a slightly deeper dive into this issue along with a nice UpSet plot of the column population rates in the DDD dataset: https://github.com/ebmdatalab/open-nhs-hospital-use-data/blob/ddd_data_quality/notebooks/antibiotics/ddd_quality.ipynb