ICEES PCD datasets

[karafecho]

This issue is to report bugs that were identified during testing of the ICEES PCD datasets (v2, Hong's first run).

• [ ] The 'index' variable is missing. This is used to link patients across years. It is generated by FHIR PIT using the PCORnet identifiers.
• [ ] All exposure estimates were captured properly (this is an advancement from the first run), but they were not binned.
• [ ] All patient demographics were captured properly (this is also an advancement from the first run), but they were not binned.
• [ ] TotalEDVisits and TotalInpatientVisits are missing. I believe this is my fault, as FHIR_mappings are missing for these variables. TotalEDInpatientVisits is calculated by FHIR PIT.
• [ ] No meds (Rx variables) were captured.
• [ ] Diagnoses (Dx variables) were largely coded as "0", apart from a handful of variables, which contain codings of "1", etc in some but not all years (FailureToThriveDx, ProstateCancerDx, SitusInversusDx, HypertensionDx, DepressionDx, AlcoholDependenceDx, DepressionDx, DrugDependenceDx, CoughDx). Diagnoses were not binned.
• [ ] The order of variables differs from file to file, which is manageable to deal with, but I wouldn't think this would happen.
• [ ] All non-flag labs are coded as "0".
• [ ] Lab flags (X_flag_first, X_flag_last) were not captured.
• [ ] CLARK/SME crosswalk files were not captured, but can be joined at the pre-deidentification step of the FHIR PIT pipeline.
• [ ] Bin boundaries are captured in a separate csv file and are generated by FHIR PIT pipeline pre-binning. This is important for variables that are binned using pandas.cut or pandas.qcut.

[karafecho]

Update from new run, 08.17.2022, 2016 and 2020 datasets (v3, Hong's second run):

This issue is to report bugs that were identified during testing of the ICEES PCD datasets.

• [ ] The 'index' variable is missing. This is used to link patients across years. It is generated by FHIR PIT using the PCORnet identifiers.
• [ ] All exposure estimates were captured properly (this is an advancement from the first run) and binned properly (this is an improvement).
• [ ] All patient demographics were captured properly (this is also an advancement from the first run) and binned properly (this is an improvement).
• [ ] TotalEDInpatientVisits were captured for 2020, not 2016. This seems impossible.
• [ ] Meds (Rx variables) were captured (this is an improvement) but coded as "0".
• [ ] There's a new variable named MedicationRequestDependenceDx that I did not create. This seems weird and may have been introduced with the domain:Drug -> domain:MedicationRequest update.
• [ ] Diagnoses (Dx variables) were largely coded as "0", apart from a handful of variables, which contain codings of "1", etc in at least one of the years (AlopeciaDx, PneumoniaDx, ChronicNasalCongestionDx, UterineCancerDx, FailureToThriveDx, ProstateCancerDx, SitusInversusDx, HypertensionDx, DepressionDx, AlcoholDependenceDx, DepressionDx, DrugDependenceDx, CoughDx). Diagnoses were binned properly (this is an improvement).
• [ ] All non-flag labs are coded as "0".
• [ ] Lab flags (X_flag_first, X_flag_last) were not captured.
• [ ] CLARK/SME-confirmed diagnoses crosswalk files were not captured, but can be joined at the pre-deidentification step of the FHIR PIT pipeline.

[karafecho]

Update from new run, 08.4.2022, 2016 and 2022 datasets (v4, Hong's third run):

• [ ] We appear to have captured all of the Dx variables, with the exception of perhaps one or two, although those may be captured in years other than the ones I checked. I have not yet confirmed that we were able to replicate the published findings using the existing ICEES PCD instance, but it's looking promising.
• [ ] We did not capture a single Rx variable. The variable names were captured, but they are all coded as "0", which is the default coding that FHIR PIT (incorrectly) inserts. As the Rx FHIR mappings are the same ones we used for asthma (plus additional PCD-related Rx), then we should be capturing them. I checked the first three mappings for AlbuterolRx using the Athena interface and confirmed that they are correct. The PCD-related Rx mappings were pulled by direct export from the Athena interface after manually confirming that the mappings were correct, so we should be capturing them. I think this issue may be related to the domain mapping, but I'm not certain.
• [ ] We captured TotalEDVisits, TotalInpatientVisits, and TotalEDInpatientVisits for year 2020, but not year 2016. Again, the variable names were captured, but the values are all "0" for 2016, which is highly unlikely. I suspect this issue relates to the need to specify the year from which to pull these data from the Visit tables. Update 08.25/2022: I confirmed that the variables in question were captured for year 2020 only.
• [X] We captured all demographic and exposures data.

[karafecho]

JAN2023 tests of v6 datasets:

Note that I ran a variety of summary statistics on the datasets for years 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2018, 2020, and 2021. I looked at key demographics, exposures, visits, diagnoses, and meds within each dataset. My tests were semi-systematic.

1. Years 2010, 2011, 2012, and 2013 are missing data on Rx variables and TotalEDVisits, TotalInpatientVisits, and TotalEDInpatientVisits. This is to be expected, I think, because James' updates were intended for years 2014-2020 (meds are legacy data prior to 2014, not sure what the issues are for visit). However, it's unfortunate because we have pretty rich airborne pollutant exposures data for years 2010-2013. That said, I'm inclined to perhaps expose the data but conceal (delete) the Rx and visit variables from the csv files for years 2010-2013, so that it's not misleading to users.
2. We now have Rx variable mappings for years 2014-2021! Some of the new Rx variables (e.g., OrkambiRx) appear to be missing, although it's possible that they were simply never prescribed/administered. I am, however, seeing at least a few of the new Rx variables (e.g., IvacaftorRx), so that's reassuring.
3. We are now capturing TotalEDVisits, TotalInpatientVisits, and TotalEDInpatientVisits for years 2014-2021!
4. We are missing airborne pollutant exposure estimates for years 2017-2021. This is a data gap issue that I'm working to fix. As such, we should probably conceal (delete) those variables from the csv files for years 2017-2021.
5. For consistency, and to aid analysis, we may want to replace the empty cells for the variable Confirmed_Dx with either a string entry of "Not_CF_Brx_PCD" (or something similar) or a zero, as these are patients who had their charts reviewed and were deemed to not have a diagnosis of CF, Brx, or PCD. FWIW, I'm inclined to replace the empty cells with a string entry.

In sum, @hyi and @maximusunc, I think we're set to move forward with new ICEES+ and ICEES KG PCD deployments, but only after a decision is made re (1) and (4). If the "empty" variables show up as "null" in the APIs, then I think we should be fine, but I'd prefer to get your input before making a decision.

[karafecho]

I split Confirmed_Dx into three variables, Confirmed_CF_Dx, Confirmed_IdiopathicBronchiectasisDx, and Confirmed_PCD_Dx, corresponding to those in that all_features YAML. I then copied the files to hop.renci.org at /projects/ebcr/pcd/data/patient/v6_rev_csv_files.

[karafecho]

@maximusunc : The new v6 pcd datasets are located on hop.renci.org at /projects/ebcr/pcd/data/patient/v6_rev_csv_files.

@hyi : ICEES+ PCD is now ready for redeployment with the new datasets.

[karafecho]

Closing as this is complete ...