6/28 Automation Update

[KELSEYDOWLING7]

• [x] Biospecimen Report Update- several fixes [success]
• [x] Operations Report- update with BSW and BSW Race variable [delivered]
• [x] Weekly Operations CSV files- NEW Refusal and withdrawal report for BSW in folder number 271949968140
• [x] Automated Biospecimen QC- test STRECK tube rules added to automated QC [rules updated, tested/delivered to box]
• [x] Weekly Notifications QC- find out why it failed and fix if needed. Didn't fail locally [bigrquery error last week, but succeeded when tested]
• [x] Monthly Cancer History- update with BSW
• [x] Monthly Duplicates Report- update with BSW
• [x] Weekly PROMIS Completion vs Notifications- update with BSW
• [x] Weekly CCC Module Metrics- add BSW
• [x] Monthly Data Destruction Report- update with BSW
• [x] Biospe_Custom_QC_Accession_Dup- update with BSW

If we have time:

• [x] Monthly update to refresh all tables with new variables from raw tables [done]

• [ ] Biospecimen QC- add and test Home MW QC. If this goes well, then I'll do a PR to remove these rules from the Custom Biospecimen QC report

• [ ] BUM/BU/MW/COVID/Menstrual Cycle Survey Rules from IMS- test and add to automated QC

Need to be tested:

• [x] Biospecimen Report
• [x] Operations report
• [x] Operations CSVs [successful]
• [x] Biospecimen QC for STRECK tubes (and for Home MW if we get to it) [STRECK rules were tested]
• [x] Weekly notifications QC [successful]

Don't need to be tested:

• Monthly Cancer History
• Monthly Duplicates
• Weekly PROMIS report
• Monthly Data Destruction Report
• Weekly Operations CSV files
• Weekly CCC Module Metrics
• Biospecimen Custom QC

[KELSEYDOWLING7]

@jacobmpeters Rather long list this week... don't know if it's possible to handle the ones that don't need to be tested on Friday and the ones that do need to be tested on Thursday or vice versa? I'm trying to test the operations report locally because it was a smaller change but the token keeps timing out. I'll try again tomorrow

[jacobmpeters]

@KELSEYDOWLING7

Regarding Weekly Notifications QC failure from last week. There is a bigrquery:bq_table_download() error. You might need to reduce the page_size parameter..

[KELSEYDOWLING7]

The Weekly Operations CSV file will need to be tested, I'm sorry for the confusion.

It's in the code but the BSWH Weekly Log will go here https://nih.app.box.com/folder/271950400797 And the BSWH refusal and withdrawal file will go here https://nih.app.box.com/folder/271949968140

[jacobmpeters]

The Operations Report failed with an R error:

[KELSEYDOWLING7]

Ok I'll work on that right now then. The PR for Biospecimens is done. Another error popped up so I'll be adding to that PR right now before it goes to main

[KELSEYDOWLING7]

Operations CSV Files up and merged to main

[jacobmpeters]

Operations CSV Files up and merged to main

Great! I'll let that build and test it by noon

[KELSEYDOWLING7]

Fixed the bug in the operations report and did new merge to main for that. ready for testing

[jacobmpeters]

The biospecimen report failed with an R error:

[jacobmpeters]

The Weekly Operations Report also failed with an R error:

[KELSEYDOWLING7]

Ok working on those now. Worst case scenario we can back track to the old Operations report and I can reuse that code from scratch and I can re-add BSW. Worst Worst case scenario, BSW isn't on next week's report just yet but is included in the csv files at least

[KELSEYDOWLING7]

Found the issue on the biospecimen report, I think I just messed up the merge in the PR. Resent the PR but I can't tell if there are merge conflicts. The page won't load or its stuck on reviewing for merge conflicts so please let me know if you notice anything on your end

[jacobmpeters]

There is another R error in the Biospecimen Report.

[KELSEYDOWLING7]

@jacobmpeters Oh no! Idk why these errors aren't happening when I knit the report. Let me test that chunk and the entire report again. Sorry about this

[KELSEYDOWLING7]

Also is it way too much work to try to run the older version of the operations report (before BSW) to see if that also fails with similar error messages or if it runs? That would make it much easier for me to edit since I'm not able to test run that report locally

[jacobmpeters]

Also is it way too much work to try to run the older version of the operations report (before BSW) to see if that also fails with similar error messages or if it runs? That would make it much easier for me to edit since I'm not able to test run that report locally

I will try to run that version on my local machine.. I'll let you know how it goes.

[KELSEYDOWLING7]

Thank you!!

[KELSEYDOWLING7]

The Biospecimen report merge conflict resolutions just got screwed up. I'm thinking during my computer freezes it didn't save properly. Trying again. If this doesn't work I'll pull what's in the main branch and handle the conflicts locally

[jacobmpeters]

The Biospecimen report merge conflict resolutions just got screwed up. I'm thinking during my computer freezes it didn't save properly. Trying again. If this doesn't work I'll pull what's in the main branch and handle the conflicts locally

Thanks for the update. Let me know if you need any support with that. I'm on a call until 3:30 PM.

A pre-BSWH version of the Operations Report is running on my machine right now.. I'll let you know if I encounter an error.

[KELSEYDOWLING7]

Great. I pulled that version of the code and re-added BSW. Both Biospecimen and Operations are ready to be retested

[jacobmpeters]

Looks like a table is missing in the Biospecimen report still..

[jacobmpeters]

The operations report failed with an R error as well:

[jacobmpeters]

The operations report failed with an R error. I can help run this on Monday locally if needed:

[jacobmpeters]

@KELSEYDOWLING7 Looks like the Biospecimen report succeeded on Friday!

[KELSEYDOWLING7]

@jacobmpeters Yay! Just pushed the Operations report updates if you want to copy that code and try it on your local machine

[jacobmpeters]

@KELSEYDOWLING7 The R code completed successfully, but as it was rendering the PDF, we received this error: <simpleError: LaTeX failed to compile Connect_WeeklyOperationsMetrics_2024-07-01_boxfolder_183922736204.tex

I will try to figure out what is causing this tomorrow morning.

[KELSEYDOWLING7]

@jacobmpeters Agreed, I'd consider it progress at least!

Just a heads up these reports failed to run. Not overly concerned considering they're mostly the monthly reports, but while they failed in GCP they ran locally no problem.

• Monthly BU and BUM Monthly reports
• Weekly Biospecimen Custom QC (can easily run this locally on Monday, no rush to get solved)
• Monthly Duplicates report

[jacobmpeters]

@jacobmpeters Agreed, I'd consider it progress at least!

Just a heads up these reports failed to run. Not overly concerned considering they're mostly the monthly reports, but while they failed in GCP they ran locally no problem.

• Monthly BU and BUM Monthly reports
• Weekly Biospecimen Custom QC (can easily run this locally on Monday, no rush to get solved)
• Monthly Duplicates report

Ok. I will investigate those today and post what I learn here.

[jacobmpeters]

With the las commit that I pushed to main, I changed "Baylor Scott &" to "Baylor Scott and" in all of the string labels. This avoids the situation below where the "&" leads to an extra column and breaks the latex tables.

The Weekly Operations Report was run and delivered on 7/2.

[KELSEYDOWLING7]

A new push has been made for all repos to change the & in BSW to 'and'. They don't need to be tested, just merged to the main branch.

• [x] Operations Repo:

• Operations report

• Operations CSVs

• Monthly Cancer History

• Monthly Duplicates

• [x] Module Repo:

• Weekly PROMIS report

• Monthly Data Destruction Report

• Weekly CCC Module Metrics

• [x] Biospecimen repo:

• Biospecimen Report

• Biospecimen Custom QC

[jacobmpeters]

Merged :)