Write functions for variable qc checks

Analyticsphere / bq2

SQL query development for Connect for Cancer Preventions' BQ2 database.

https://analyticsphere.github.io/bq2/

0 stars 1 forks source link

Write functions for variable qc checks #15

Open jacobmpeters opened 10 months ago

jacobmpeters commented 10 months ago

df <- identify_tail_type_1(module_schema, data_dictionary) etcetera...

jacobmpeters commented 9 months ago

From @jeannewu on MS Teams:

[10:32 AM] Wu, Jing (NIH/NCI) [C] I have completed the functions we discussed last Friday.

https://github.com/jeannewu/Jeanconnect/blob/b477fe668bdc7727d04ad9bfc984a79bc22f1e67/M1_variables_duplicates_checks_12142023.Rmd#L743-L827

jacobmpeters commented 9 months ago

Thanks, @jeannewu. I'll review your functions before the next BQ2 meeting and get back to you with questions/suggestions. Then we can put them in separate files in the bq2/custom_r_functions subdirectory when we are ready to use them in the query generator scripts.

jacobmpeters commented 9 months ago

Useful to have separate functions for each check
Useful to store information in schema (Jing's code)
Using regex can simplify transformations and avoid complex mutate functions @jeannewu plans to review her code and @jacobmpeters code and consolidate our approaches to align with the key points above that we agreed to in our BQ2 meeting.

jeannewu commented 9 months ago

@jacobmpeters , here are seven pairs of duplicated variables in module 1 version 2 which might be removed with your help on the flattening procedure:. Hope you can see and grab them here to double check via GCP.

jeannewu commented 9 months ago

Sorry, the ones above are not correct. The correct duplicates left in the module 1 version2 are: SELECT D_115616118_D_403180970, D_115616118_D_115616118_D_403180970,D_173240848_D_195093589, D_173240848_D_173240848_D_195093589,D_431628922_D_963867920,D_431628922_D_431628922_D_963867920,D_517307064_D_942347130,D_517307064_D_517307064_D_942347130, D_525535977_D_396618548,D_525535977_D_525535977_D_396618548,D_630675760_D_660541817,D_630675760_D_630675760_D_660541817,D_814664694_D_787193493, D_814664694_D_814664694_D_787193493 FROM nih-nci-dceg-connect-prod-6d04.FlatConnect.module1_v2_JP

jacobmpeters commented 9 months ago

Thank you, @jeannewu . I will review these when I look at the others.

jeannewu commented 9 months ago

@all, I have completed this function and made a demo in the BQ meeting on Jan. 26, 2024 via