search

HMIS / LSASampleCode

Longitudinal System Analysis (LSA) Sample Code and Documentation
22 stars 10 forks source link

Email about ~125K flags #1208

Closed nataliesus closed 9 months ago

nataliesus commented 9 months ago

Hi @JulesABrown , @MollyMcEvilley ,

We received the following email:

I wanted to reach out because we have been running into some issues with a few CoCs that are getting the “red banner” error at upload. This banner usually just means the file got stuck in the processing and we need to kick it off again. However, there are a handful of CoCs where we try that, but even then it can’t run through all the flags in the allotted 15 minute window. Today, we were finally able to dig into why that is happening and our example happened to be .... When we pulled their raw flag data down, their file turned out to be tripping over 128K flags, nearly all of which were associated with the LSAPerson file. I am guessing you probably are already aware of these issues, but I wanted to reach out and see if there is anything I or Molly can do to support you.

In the attached file were a lot of flags i.e. Flag 117:

Data Error,"There is a mismatch in your data between the HHTypeEST field and the HoHEST field. The HHTypeEST field identifies the household types for all clients served in ES/SH/TH. The HoHEST field identifies the household types for heads of households served in ES/SH/TH. In row 2 in your LSAPerson file, we see zero clients for Adult Only households in ES/SH/TH. But there are also people classified as heads of Adult Only households in ES/SH/TH. If a person was not served in an Adult Only household, they can’t be a head of household for an Adult Only household. This is not an issue that can be fixed by looking at your LSAPerson file because client-level information (like a personal ID) is not stored there. You may need vendor assistance to resolve this error.",Observation Number (not counting the header row) in your LSAPerson file,2,hhtypeest,0,hohest,1,,,,,,,,,,,,,

I wonder if there is an issue in the flag validation rules because:

then 0z are replaced with "": image

and then records are inserted into lsa_Person table: image

I've generated the report again and confirm there is no record in lsa_Person where HHTypeEST = 0 and HoHEST = 1. https://hdxsandbox.abtsites.com/module/lsa/uploads/530

Could you please advise?

Thank you! Natalie

JulesABrown commented 9 months ago

Just a note that I see this and I will need to dig into it more soon. I wanted to investigate your flag report more this week, but things got hairy with flags disappearing and my time got wiped out.

My work day may look different than your work day. Please do not feel obligated to respond out of your normal working hours.

Jules Brown | HMIS SME | he/him | Abt Associates O: 301.347.5352 | Cell: 773.750.7886

From: Natalie Sus @.> Sent: Friday, December 8, 2023 4:41 PM To: HMIS/LSASampleCode @.> Cc: Jules Brown @.>; Mention @.> Subject: [HMIS/LSASampleCode] Email about ~125K flags (Issue #1208)

Warning from Abt: External email. Be careful opening links and attachments.

Hi @JulesABrownhttps://github.com/JulesABrown , @MollyMcEvilleyhttps://github.com/MollyMcEvilley ,

We received the following email:

I wanted to reach out because we have been running into some issues with a few CoCs that are getting the “red banner” error at upload. This banner usually just means the file got stuck in the processing and we need to kick it off again. However, there are a handful of CoCs where we try that, but even then it can’t run through all the flags in the allotted 15 minute window. Today, we were finally able to dig into why that is happening and our example happened to be .... When we pulled their raw flag data down, their file turned out to be tripping over 128K flags, nearly all of which were associated with the LSAPerson file. I am guessing you probably are already aware of these issues, but I wanted to reach out and see if there is anything I or Molly can do to support you.

In the attached file were a lot of flags i.e. Flag 117:

Data Error,"There is a mismatch in your data between the HHTypeEST field and the HoHEST field. The HHTypeEST field identifies the household types for all clients served in ES/SH/TH. The HoHEST field identifies the household types for heads of households served in ES/SH/TH. In row 2 in your LSAPerson file, we see zero clients for Adult Only households in ES/SH/TH. But there are also people classified as heads of Adult Only households in ES/SH/TH. If a person was not served in an Adult Only household, they can’t be a head of household for an Adult Only household. This is not an issue that can be fixed by looking at your LSAPerson file because client-level information (like a personal ID) is not stored there. You may need vendor assistance to resolve this error.",Observation Number (not counting the header row) in your LSAPerson file,2,hhtypeest,0,hohest,1,,,,,,,,,,,,,

I wonder if there is an issue in the flag validation rules because:

and are used as they are when records are inserted into lsa_Person table: image.png (view on web)https://github.com/HMIS/LSASampleCode/assets/28144855/63194cbd-76d8-4e41-b045-847bb4fa83c6

I've generated the report again and confirm there is no record in lsa_Person where HHTypeEST = 0 and HoHEST = 1. https://hdxsandbox.abtsites.com/module/lsa/uploads/530

Could you please advise?

Thank you! Natalie

— Reply to this email directly, view it on GitHubhttps://github.com/HMIS/LSASampleCode/issues/1208, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKU33GP4SRJT37WAJCHLUIDYIOJOLAVCNFSM6AAAAABANI7UWWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGAZTGMZZGQ3TOMI. You are receiving this because you were mentioned.Message ID: @.**@.>>

This message may contain privileged and confidential information intended solely for the addressee. Please do not read, disseminate or copy it unless you are the intended recipient. If this message has been received in error, we kindly ask that you notify the sender immediately by return email and delete all copies of the message from your system.

JulesABrown commented 9 months ago

Definitely our fault – something weird happened with that flag when run locally. Still digging into implications re: red banners, but in the meantime, you can scratch the set of Flag ID 117 off your list. Here is an updated version of the WA-500 and WA-501 locally-generated flag files with that issue corrected. Not sure which I sent before – I think it was mislabeled.

Jules

My work day may look different than your work day. Please do not feel obligated to respond out of your normal working hours.

Jules Brown | HMIS SME | he/him | Abt Associates O: 301.347.5352 | Cell: 773.750.7886

From: Jules Brown @.> Sent: Friday, December 8, 2023 4:48 PM To: HMIS/LSASampleCode @.>; HMIS/LSASampleCode @.> Cc: Mention @.> Subject: RE: [HMIS/LSASampleCode] Email about ~125K flags (Issue #1208)

Just a note that I see this and I will need to dig into it more soon. I wanted to investigate your flag report more this week, but things got hairy with flags disappearing and my time got wiped out.

My work day may look different than your work day. Please do not feel obligated to respond out of your normal working hours.

Jules Brown | HMIS SME | he/him | Abt Associates O: 301.347.5352 | Cell: 773.750.7886

From: Natalie Sus @.**@.>> Sent: Friday, December 8, 2023 4:41 PM To: HMIS/LSASampleCode @.**@.>> Cc: Jules Brown @.**@.>>; Mention @.**@.>> Subject: [HMIS/LSASampleCode] Email about ~125K flags (Issue #1208)

Warning from Abt: External email. Be careful opening links and attachments.

Hi @JulesABrownhttps://github.com/JulesABrown , @MollyMcEvilleyhttps://github.com/MollyMcEvilley ,

We received the following email:

I wanted to reach out because we have been running into some issues with a few CoCs that are getting the “red banner” error at upload. This banner usually just means the file got stuck in the processing and we need to kick it off again. However, there are a handful of CoCs where we try that, but even then it can’t run through all the flags in the allotted 15 minute window. Today, we were finally able to dig into why that is happening and our example happened to be .... When we pulled their raw flag data down, their file turned out to be tripping over 128K flags, nearly all of which were associated with the LSAPerson file. I am guessing you probably are already aware of these issues, but I wanted to reach out and see if there is anything I or Molly can do to support you.

In the attached file were a lot of flags i.e. Flag 117:

Data Error,"There is a mismatch in your data between the HHTypeEST field and the HoHEST field. The HHTypeEST field identifies the household types for all clients served in ES/SH/TH. The HoHEST field identifies the household types for heads of households served in ES/SH/TH. In row 2 in your LSAPerson file, we see zero clients for Adult Only households in ES/SH/TH. But there are also people classified as heads of Adult Only households in ES/SH/TH. If a person was not served in an Adult Only household, they can’t be a head of household for an Adult Only household. This is not an issue that can be fixed by looking at your LSAPerson file because client-level information (like a personal ID) is not stored there. You may need vendor assistance to resolve this error.",Observation Number (not counting the header row) in your LSAPerson file,2,hhtypeest,0,hohest,1,,,,,,,,,,,,,

I wonder if there is an issue in the flag validation rules because:

and are used as they are when records are inserted into lsa_Person table: image.png (view on web)https://github.com/HMIS/LSASampleCode/assets/28144855/63194cbd-76d8-4e41-b045-847bb4fa83c6

I've generated the report again and confirm there is no record in lsa_Person where HHTypeEST = 0 and HoHEST = 1. https://hdxsandbox.abtsites.com/module/lsa/uploads/530

Could you please advise?

Thank you! Natalie

— Reply to this email directly, view it on GitHubhttps://github.com/HMIS/LSASampleCode/issues/1208, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKU33GP4SRJT37WAJCHLUIDYIOJOLAVCNFSM6AAAAABANI7UWWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGAZTGMZZGQ3TOMI. You are receiving this because you were mentioned.Message ID: @.**@.>>

This message may contain privileged and confidential information intended solely for the addressee. Please do not read, disseminate or copy it unless you are the intended recipient. If this message has been received in error, we kindly ask that you notify the sender immediately by return email and delete all copies of the message from your system.

LaurenBianchi commented 9 months ago

Hi @JulesABrown ! We did not receive the updated flag files for WA-500/WA-501. Just wanted to note that the issue Natalie identified with Flag 117 also appears to apply to the other 'mismatch' flags (i.e., 261, 278, 279, 283, 284, 288, 289, etc.) that make up the ~128k flags in the initial flag file sent on 12/5. Hope you have a wonderful tuesday! LB