Stage data destruction stub record, remaining issues

[robertsamm]

Reviewed stub record in stage for connectID 1475895409 and found a few remaining issues. After the stub record was pushed, it did the same thing in the SMDB where I can't view the Participant Summary page for this participant anymore, I'm not sure if that is intentional or not but I don't see it in the SOP. So looking at this record in the SMDB and the CIDs in the array Jessica sent me here's the few remaining issues I found

1. Preferred email should not be retained, needs to be removed (I see it in the SMDB and the list of CIDs in the array you sent)
2. Preferred name should also not be retained, needs to be removed (I see this listed in the array)
3. Participant demographic variables should not be retained (site reported age, site reported race/ethnicity, site reported sex). I see this in the SMDB but I don't see the CIDs listed in the array you sent. These are all vars sent by the site.
4. Participant verification table variables for site match vars and campaign type should not be retained (first name match, last name match, DOB match, PIN match, token match, zip code match, age match, cancer status match). These are also all vars sent by the site and I see them filled in still on the SMDB. Should only need to retain verification status and time of verification.
5. Participant Summary page unclickable after stub record created
6. I'm not able to check to confirm that the biospecimen variables were not retained without the pt summary page. Can someone doublecheck this data on the backend?

[jeannewu]

HI, everyone, I have double checked the data destructions in dev for this Connect_ID: all the data is deleted correctly as expected

[jhflorey]

Hi all - I checked SMDB and there was one issue for connect_id: 2361618927. I could not download the original HIPAA and consent agreement forms. Instead I received an error message that said "An error has occured generating the pdf please contact support". Otherwise, everything else looked good - I was able to download the data destruction and HIPAA revocation forms and the signatures were correct on SMDB, I was able to access all 4 forms in MyConnect, the notifications are gone from both the MyConnect and the SMDB, and the correct variables were the only ones remaining in SMDB.

@kmazzilli this issue will be fixed by this PR https://github.com/episphere/dashboard/pull/568

[kmazzilli]

@jhflorey thanks! I was able to download the forms on SMDB without the error message just now.

[brotzmanmj]

@jhflorey We just noticed someone entered a data destruction request for a HFH participant in stage. We don't know who originated this one... what this one you initiated?

[robertsamm]

Just pushed a data destruction in stage for Connect ID: 2055639272 and signed the data destruction and revocation forms. @jhflorey we can check tomorrow to see if everything looks good for the stub record.

[jeannewu]

Hi, @robertsamm, may you please let me know where I can find the stub record for Connect ID:2055639272. It will be helpful for me to double check on my end for the data destruction. Thanks, Amelia

[robertsamm]

Addtl data destructions that were pushed to stage today, neither have study activity data in them. Connect IDs: 8553496854, 3800067442

[jeannewu]

@robertsamm, the data destructions for these two Connect IDs will be processed as same as the previous one for all the data in Stage, right?

[jhflorey]

@jhflorey We just noticed someone entered a data destruction request for a HFH participant in stage. We don't know who originated this one... what this one you initiated?

@brotzmanmj my test participant in stage

• Connect_ID: 3800067442
• Email: testddonstage@mailinator.com

[robertsamm]

@jeannewu Yes, you'll want to check all 3 records. I confirmed w/ Michelle - the stub record is not a separate record. It is just a list of the only variables that should remain after everything else is destroyed. They will reside in the data structure the same place they always do.

We'll first need to have @jhflorey confirm the destruction was run overnight and that the 3 Connect IDs (CIDs: 2055639272, 8553496854, 3800067442) data was deleted and that they look correct in Firestore first. Then Jing you can confirm the same in BQ after that. If you have any questions about the stub record, please ask Jessica who would know best.

[jeannewu]

@robertsamm I am checking in the all the data in Dev, I don't see any data on these three Connect_IDs even in the participants table, right?

[robertsamm]

@jeannewu these are stage participants

[jeannewu]

@robertsamm Thanks. I am checking them now.

[jeannewu]

@robertsamm @jhflorey I have just checked the BQ data: right now the 2055639272 data are still shown in the stage, (seen the table)

data in stage	undeleted
bioSurvey_v1_JP	2.06E+09
biospecimen_JP	2.06E+09
clinicalBioSurvey_v1_JP	No data on Connect_ID: 8553496854, 3800067442, 2055639272
covid19Survey_v1_JP	2.06E+09
module1_v1_JP	2.06E+09
module1_v2_JP	No data on Connect_ID: 8553496854, 3800067442, 2055639272
module2_v1_JP	2.06E+09
module2_v2_JP	No data on Connect_ID: 8553496854, 3800067442, 2055639272
module3_v1_JP	2.06E+09
module4_v1_JP	2.06E+09
participants_JP	8553496854, 3800067442, 2055639272
participants_noPII_JP	8553496854, 3800067442, 2055639272

[jhflorey]

@jeannewu @robertsamm The job was still working last night but it ran into a problem about the index key since we have not generated it in stage (firebase) yet. I asked for help from @anthonypetersen and will inform to you soon. Sorry for this inconvenience.

[jeannewu]

@jhflorey Thank you very much. Please let me know if anything is needed from me.

[jhflorey]

@jeannewu it's already worked, i have also checked my test participant (3800067442). It worked as my expectation. I suggest that you can check it on PWA and SMDB first then check data in BQ later.

[robertsamm]

Hi @jhflorey can you check the other two participants as well and then we'll check the SMDB and PWA?

[jeannewu]

@jhflorey I have just checked the stg-5519.Connect: all the data of these three participants are deleted as expected

data in stg-5519.Connect	data deletion
bioSurvey_v1	No data on Connect_ID: 8553496854, 3800067442, 2055639272
biospecimen	No data on Connect_ID: 8553496854, 3800067442, 2055639272
clinicalBioSurvey_v1	No data on Connect_ID: 8553496854, 3800067442, 2055639272
covid19Survey_v1	No data on Connect_ID: 8553496854, 3800067442, 2055639272
menstrualSurvey_v1	No data on Connect_ID: 8553496854, 3800067442, 2055639272
module1_v1	No data on Connect_ID: 8553496854, 3800067442, 2055639272
module1_v2	No data on Connect_ID: 8553496854, 3800067442, 2055639272
module2_v1	No data on Connect_ID: 8553496854, 3800067442, 2055639272
module2_v2	No data on Connect_ID: 8553496854, 3800067442, 2055639272
module3_v1	No data on Connect_ID: 8553496854, 3800067442, 2055639272
module4_v1	No data on Connect_ID: 8553496854, 3800067442, 2055639272
participants	2055639272|3800067442|8553496854 dim= 3 181

[jeannewu]

Maybe I will check their deletion in the BQ data next Monday when the BQ data is refreshed.

[jhflorey]

@jeannewu yup, thank you so much.

[jhflorey]

Maybe I will check their deletion in the BQ data next Monday when the BQ data is refreshed.

You can check on PWA and SMDB first.

[robertsamm]

Thanks all. Kaitlyn and I will check PWA and SMDB. @jeannewu Monday is a federal holiday, so check on Tuesday.

[jeannewu]

@robertsamm Yup, I will check the data of these three CID next Tuesday.

[robertsamm]

Kaitlyn and I checked the SMDB and PWA for all 3 participants and they looked good in stage. Destruction and stub record variables are as expected.

[jhflorey]

@robertsamm Yup, I will check the data of these three CID next Tuesday.

@jeannewu Have you checked it?

[jeannewu]

@jhflorey @robertsamm I've just checked the BQ data: all the data of these three CID are deleted as expected. Thanks for double checking with me.

[jhflorey]

@jeannewu thanks for your confirmation