search

episphere / connect

Connect API for DCEG's Cohort Study
10 stars 5 forks source link

Preferred Language Unexpectedly Retained for Participations that have Data Destroyed #1138

Closed mnataraj92 closed 4 hours ago

mnataraj92 commented 1 month ago

Michelle and Amelia pointed out that the Preferred Language variable (d_255077064) is currently being retained for participants that have data destroyed. This variable is not listed as a stub record variable that should be retained, and Jessica confirmed that this variable is not listed as a stub record variable in her code as well. Jessica mentioned that this may be because of some added logic in the PWA or SMDB, and she will look into it. It would be good to know where it's coming from and if it's necessary to some process so we can decide whether to keep it or not; this is more of a mechanistic decision.

This is for the November release.

mnataraj92 commented 1 month ago

Quick note- during testing for the any refusal or withdrawal & participation status derivations, Jessica also checked the preferred language variable for a test pt in dev that had data destroyed, and it was not retained for this participant, so she's unsure as to why it's being retained in prod for pts with data destroyed. She will investigate this further in November.

mnataraj92 commented 3 weeks ago

Will close this issue for now as this was tested in dev and stage and d_255077064 was removed. It is likely that this variable was set for a handful of those with data destroyed after the fact. We will monitor to see if any pts w/data recently destroyed have d_255077064 retained and will reopen this issue if needed.

@brotzmanmj should we enter in a prod change log item to nullify this variable for those with data destroyed?

brotzmanmj commented 3 weeks ago

@mnataraj92 How many people in prod with data destroyed have this flag populated?

mnataraj92 commented 3 weeks ago

@brotzmanmj there are 87 people where data has been destroyed but d_255077064 is retained

brotzmanmj commented 3 weeks ago

Hi @mnataraj92 I do think we should destroy the preferred language flag for these cases but it is not urgent to do so. Can you coordinate with the team as to what's the best way to do this and when?

mnataraj92 commented 3 weeks ago

@brotzmanmj sounds good! i'll update here once I find out more info on the best way to do this

mnataraj92 commented 3 weeks ago

Spoke with @sonyekere and she mentioned she will take this back to devops and see where deleting the preferred language flag (d_255077064) for these 87 individuals will fit.

jhflorey commented 2 weeks ago

Just removed the preferred language flag of 25 pts in dev

{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 1286106230
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 8010150172
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 7956451897
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 2537423491
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 5773487914
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 2758935713
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 8961882555
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 7009372190
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 5810010461
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 1463424173
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 1855423655
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 6451999189
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 4151371961
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 7940377048
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 4723748117
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 4048677829
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 2412004348
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 7846399594
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 7876237212
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 7204238865
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 2038757656
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 9138035066
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 8736582209
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 3086354774
}
{
  '255077064': 163149180,
  '861639549': 353358909,
  Connect_ID: 4487686178
}
Updated 25 participants successfully
jhflorey commented 2 weeks ago

Updated 35 pts in stage

image
jhflorey commented 2 weeks ago

@mnataraj92 @sonyekere Updated 87 pts in prod.

{"255077064":163149180,"861639549":353358909,"Connect_ID":8437377292}
{"255077064":163149180,"861639549":353358909,"Connect_ID":3665757145}
{"255077064":163149180,"861639549":353358909,"Connect_ID":6808048849}
{"255077064":163149180,"861639549":353358909,"Connect_ID":6575848257}
{"255077064":163149180,"861639549":353358909,"Connect_ID":5810762484}
{"255077064":163149180,"861639549":353358909,"Connect_ID":7611644934}
{"255077064":163149180,"861639549":353358909,"Connect_ID":7329688967}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8484005367}
{"255077064":163149180,"861639549":353358909,"Connect_ID":7815518337}
{"255077064":163149180,"861639549":353358909,"Connect_ID":6880853459}
{"255077064":163149180,"861639549":353358909,"Connect_ID":3899176500}
{"255077064":163149180,"861639549":353358909,"Connect_ID":6792098176}
{"255077064":163149180,"861639549":353358909,"Connect_ID":1974642387}
{"255077064":163149180,"861639549":353358909,"Connect_ID":7058269278}
{"255077064":163149180,"861639549":353358909,"Connect_ID":9365463918}
{"255077064":163149180,"861639549":353358909,"Connect_ID":4276016386}
{"255077064":163149180,"861639549":353358909,"Connect_ID":6851074592}
{"255077064":163149180,"861639549":353358909,"Connect_ID":7570850444}
{"255077064":163149180,"861639549":353358909,"Connect_ID":5516837502}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8841443120}
{"255077064":163149180,"861639549":353358909,"Connect_ID":3425358443}
{"255077064":163149180,"861639549":353358909,"Connect_ID":9873667215}
{"255077064":163149180,"861639549":353358909,"Connect_ID":2468652076}
{"255077064":163149180,"861639549":353358909,"Connect_ID":3952268104}
{"255077064":163149180,"861639549":353358909,"Connect_ID":2759755942}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8796662583}
{"255077064":163149180,"861639549":353358909,"Connect_ID":4259019665}
{"255077064":163149180,"861639549":353358909,"Connect_ID":1720924551}
{"255077064":163149180,"861639549":353358909,"Connect_ID":7753333627}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8950310971}
{"255077064":163149180,"861639549":353358909,"Connect_ID":3290460921}
{"255077064":163149180,"861639549":353358909,"Connect_ID":3829528105}
{"255077064":163149180,"861639549":353358909,"Connect_ID":5547802251}
{"255077064":163149180,"861639549":353358909,"Connect_ID":9427163882}
{"255077064":163149180,"861639549":353358909,"Connect_ID":1905833856}
{"255077064":163149180,"861639549":353358909,"Connect_ID":3116697697}
{"255077064":163149180,"861639549":353358909,"Connect_ID":7146217731}
{"255077064":163149180,"861639549":353358909,"Connect_ID":9595521302}
{"255077064":163149180,"861639549":353358909,"Connect_ID":3511226123}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8850194605}
{"255077064":163149180,"861639549":353358909,"Connect_ID":6674684215}
{"255077064":163149180,"861639549":353358909,"Connect_ID":7495838452}
{"255077064":163149180,"861639549":353358909,"Connect_ID":9335037273}
{"255077064":163149180,"861639549":353358909,"Connect_ID":2735697521}
{"255077064":163149180,"861639549":353358909,"Connect_ID":5691170745}
{"255077064":163149180,"861639549":353358909,"Connect_ID":1308062179}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8220473345}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8615148046}
{"255077064":163149180,"861639549":353358909,"Connect_ID":7549049278}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8003874306}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8098140233}
{"255077064":163149180,"861639549":353358909,"Connect_ID":2925129751}
{"255077064":163149180,"861639549":353358909,"Connect_ID":7648348294}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8790139556}
{"255077064":163149180,"861639549":353358909,"Connect_ID":1108023704}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8868913771}
{"255077064":163149180,"861639549":353358909,"Connect_ID":1637720696}
{"255077064":163149180,"861639549":353358909,"Connect_ID":9286933970}
{"255077064":163149180,"861639549":353358909,"Connect_ID":5069500066}
{"255077064":163149180,"861639549":353358909,"Connect_ID":6548751891}
{"255077064":163149180,"861639549":353358909,"Connect_ID":6363772898}
{"255077064":163149180,"861639549":353358909,"Connect_ID":1848404718}
{"255077064":163149180,"861639549":353358909,"Connect_ID":6625754395}
{"255077064":163149180,"861639549":353358909,"Connect_ID":3013518213}
{"255077064":163149180,"861639549":353358909,"Connect_ID":6522969630}
{"255077064":163149180,"861639549":353358909,"Connect_ID":4086680624}
{"255077064":163149180,"861639549":353358909,"Connect_ID":4451434373}
{"255077064":163149180,"861639549":353358909,"Connect_ID":4471979336}
{"255077064":163149180,"861639549":353358909,"Connect_ID":4411304157}
{"255077064":163149180,"861639549":353358909,"Connect_ID":4417795004}
{"255077064":163149180,"861639549":353358909,"Connect_ID":9047328610}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8409194117}
{"255077064":163149180,"861639549":353358909,"Connect_ID":6540099079}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8339529319}
{"255077064":163149180,"861639549":353358909,"Connect_ID":3858439647}
{"255077064":163149180,"861639549":353358909,"Connect_ID":1045584164}
{"255077064":163149180,"861639549":353358909,"Connect_ID":9590323728}
{"255077064":163149180,"861639549":353358909,"Connect_ID":9806051949}
{"255077064":163149180,"861639549":353358909,"Connect_ID":3388199139}
{"255077064":163149180,"861639549":353358909,"Connect_ID":4617994050}
{"255077064":163149180,"861639549":353358909,"Connect_ID":7872538688}
{"255077064":163149180,"861639549":353358909,"Connect_ID":3516658074}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8919280570}
{"255077064":163149180,"861639549":353358909,"Connect_ID":9843524210}
{"255077064":163149180,"861639549":353358909,"Connect_ID":8777216716}
{"255077064":163149180,"861639549":353358909,"Connect_ID":4914540989}
{"255077064":163149180,"861639549":353358909,"Connect_ID":5494771979}
Updated 87 participants successfully
brotzmanmj commented 2 weeks ago

@mnataraj92 Does this need to be checked before it gets done in prod? Can you coordinate review? Thanks

brotzmanmj commented 2 weeks ago

Actually please ignore comments, I see this change was made in prod and the expected number of record (87) was updated so no need to review dev and stage. Thank you.

mnataraj92 commented 2 weeks ago

Sounds good. Once BQ refreshes I can coordinate confirming just as a double check

mnataraj92 commented 2 weeks ago

Confirmed this looks good in prod

sonyekere commented 4 hours ago

Closing, confirmed correct in Prod