Mod4 Stage testing- nonmatching responses/skip logic error and one missing variable

[KELSEYDOWLING7]

(Ran backend testing without using the flattened table; reviewed straight from the participants table)

DATA DICTIONARY ISSUE: -DAYSWORK is listed twice, D_774087638 in Mod 4 and D_842657297 in Mod1. D_842657297 works, so it should be changed to Mod4 in the dictionary and D_774087638 needs to be removed

General issues within test scenarios: TEST SCENARIO 1 ISSUES: (email= taratdrp@gmail.com , Connect_ID=6537895747) -PREWORKRES (D_980007418==104430631): response populates as 353358909

TEST SCENARIO 3 ISSUES (email= isabelleintersex@gmail.com , Connect_ID=9857825959) -CHILDCITY (D_632533534_128827522) == NR: response populates as Franklin square, yet CHILDCITY_FOLLOWUP (D_264797252_890792569) question was still populated and correct answer of El Carmen populates. That skip logic doesn't make sense -CHILDST (D_632533534_439447560) == NR: response populates as New York, yet CHILDST_FOLLOWUP (D_264797252_451394598) question was still populated and correct answer of Valencia populates. That skip logic doesn't make sense

TEST SCENARIO 4 ISSUES (email= irinainstitute@gmail.com , Connect_ID=4103954097) -CURWORKT10A (D_786253125_659799855)==NA; populates response of 428999623 instead -CURWORKT10B (D_786253125_552410034)==NA; populates response of 428999623 instead -CURWORKT10C (D_786253125_772382520)==NA; populates response of 428999623 instead -CURWORKT10D (D_786253125_146307133)==NA; populates response of 428999623 instead -CURWORKT10E (D_786253125_100591237)==NA; populates response of 428999623 instead -CURWORKT10H (D_786253125_419151253)==NA; populates response of 428999623 instead

[FrogGirl1123]

@boyd-mj @danielruss @naiyume Could you please review as soon as possible to evaluate what might be markup issues vs quest issues. None of these are flattening issues as Kelsey is running her code on the raw data.

[danielruss]

What are the Scenarios?

[boyd-mj]

@danielruss the DAYS WORK issue seems unlikely to be a markup issue so that might be a good place for you to start

Scenarios 1&3 may be harmonization issues. We'll take a look at those and test the skip patterns and report back.

[FrogGirl1123]

@KELSEYDOWLING7 could you please post a link to the scenario document for Daniel to see.

[FrogGirl1123]

@danielruss this is where the data entered during testing is entered. There are 6 scenarios designed to help test where we expect to see skip logic issues.

[FrogGirl1123]

@boyd-mj do you have access to the scenarios? Would it be helpful for you to if you don't?

[KELSEYDOWLING7]

Yes, @danielruss this is where all the answers are stored https://nih.app.box.com/file/965975750266 . The Connect ID and email are listed on the top right, as well as on the issue description above

[guinevere83]

For the skip pattern problems, I might need some clarification. What questions were displayed or not displayed in these cases? Or were your responses for some questions being stored as responses to other questions? I was having trouble telling from the post.

[KELSEYDOWLING7]

For Test Scenarios 3 and 5, SEASADDOTH asks "In the past 20 years, have you spent a total of 24 months or more in any other seasonal or vacation homes in the U.S.?" To which they replied NO. By the skip logic that causes SEASADDSUM1 to populate, which asks "You told us that you have not spent a total of 24 months or more at any seasonal or vacation homes in the past 20 years. Is this correct?" Instead, somehow the skip pattern seems to be going from SEASADDOTH (correctly answered no) to SEASADDSUM2, which asks "Please review the list below and tell us if it includes all of the seasonal or vacation home addresses where you have spent a total of 24 months or more in the past 20 years."

[boyd-mj]

@FrogGirl1123 I don't have access to that document. It must be a different Box folder that we don't have access to. Could someone share it on our Box? Or alternatively if you want to grant us access to the existing location, could you please give access to me, Gwen, Deepti, and Pete? This may help Gwen to proceed with testing of these skip pattern issues as well. Thanks!

@boyd-mj do you have access to the scenarios? Would it be helpful for you to if you don't?

[KELSEYDOWLING7]

For Test Scenario 3 as well, if they answer CHILDCITY or CHILDST there should be no follow up questions of CHILDCITY_FOLLOWUP and CHILDST_FOLLOWUP, as those specifically state that city and state were skipped and asks the participant to enter an answer. However here, City and State are answered (when they should not have been, testing says no responses were given) , and yet the follow-up questions for each are still displayed and answered. Checking Firestore shortly as well to see if these incorrect answers are showing up on both ends

[danielruss]

@KELSEYDOWLING7 @naiyume @boyd-mj The Commuting section (where DAYSWORK) is located is a very simple section. The DAYSWORK question actually does not depend on any other response.

Was this tested with after concept-ids were add (in other words, Kelsey could you have the wrong concept id during testing?)

[DAYSWORK?] As you worked on that job, about how many days per week did you usually work for pay? If you cannot remember, please make your best guess.
(0) 1 day per week
(1) 2 days per week
(2) 3 days per week
(3) 4 days per week
(4) 5 days per week
(5) 6 days per week
(6) 7 days per week

[FrogGirl1123]

@boyd-mj I'll see if I can invite you and the others to the document. I don't want to try to maintain it in more than one place.

@cusackjm @cunnaneaq @KELSEYDOWLING7 do you remember which browser you were using for these scenarios?

[cunnaneaq]

@FrogGirl1123 I was using a regular Chrome window

[KELSEYDOWLING7]

@danielruss @FrogGirl1123 DAYSWORK was previously in Mod1 as D_842657297. It was removed from the markup but its apparently still in the Data Dictionary twice as D_842657297 in Mod1 and D_774087638 in Mod4. The concept ID in Mod4 populates nothing, but if I use the concept ID for DAYSWORK from Mod1 I now see the right answers populate

[KELSEYDOWLING7]

SEASADDSUM1 problems seems to be a confusion with the skip logic and wrong responses marked on the Box response file. I'll remove those issues

[naiyume]

The doesNotEqual here seems to always be returning true if the question D_203943120 is not answered/seen at all. Just wanted to double check if this is the intended behavior.

[naiyume]

Checked with @danielruss and this seems to be intended behavior from the quest side. If you want these to not show up in the grid then the markup needs to be changed. @KELSEYDOWLING7

[KELSEYDOWLING7]

The logic for CURWORKT10 is: IF CURWORKT7=NO, then display rows ONLY for responses marked in CURWORKT9. IF CURWORKT8=NO, then also display responses marked in CURWORKT2.

She answered no to bothCURWORKT7 and for CURWORKT8 was supposed to answer no but it didn't show. Is that why all CURWORKT2 chosen transportation methods are asked in CURWORKT10 as opposed to CURWORKT9 transportation methods?

[danielruss]

The doesNotEqual here seems to always be returning true if the question D_203943120 is not answered/seen at all. Just wanted to double check if this is the intended behavior.

If the question is not seen/answered use isDefined. My preference is for you NOT to use the old functions. They have proven to be quite buggy and difficult to use. I would help you modify the logic, but because the old functions were so poorly designed (my bad, sorry) I have a hard time understanding what they are doing.

[huip-ims]

@KELSEYDOWLING7 Could you please summarize what issues you are having with CURWORKT10?

[KELSEYDOWLING7]

@huip-ims Sure, I believe CURWORKT10 should only be asking about the time it takes to get home using the transportation methods listed in CURWORKT9 (aka CURWORKT10F and CURWORKT10G for biking and walking) . Instead, it is also asking about the time it takes to get home for all transportation methods listed in CURWORKT2 (aka CURWORKT10A-E and CURWORKT10H)

[huip-ims]

@KELSEYDOWLING7 Thanks for the summary. I don't have permission to access your testing responses for CURWORKT1-CURWORKT9. Could you please provide those here, so I can follow where the skip logic failed and led to the summary you provided?

[FrogGirl1123]

@huip-ims you should have access to the file, please check your email for an invite from Box

[huip-ims]

@FrogGirl1123 @KELSEYDOWLING7 OK, thanks! Just found them in the Stage M2 Questionnaire Testing excel file. Will take a look and get back to you.

[cunnaneaq]

RESPONSE: Tested again with @naiyume and @KELSEYDOWLING7. We think this is an error cause by entering city/state in CHILDCITY and CHILDST and then going back and deleting this info, which triggered the skip logic to display CHILDCITY_FOLLOWUP and CHILDST_FOLLOWUP. The deleted entry for CHILDCITY/ST was not deleted on the back-end, so Kelsey saw answers for both questions (which is incorrect). The auto-fill function could increase the likelihood of participants making this error.

ISSUE NOTED IN KELSEY'S COMMENT ABOVE: TEST SCENARIO 3 ISSUES (email= isabelleintersex@gmail.com , Connect_ID=9857825959) -CHILDCITY (D_632533534_128827522) == NR: response populates as Franklin square, yet CHILDCITY_FOLLOWUP (D_264797252_890792569) question was still populated and correct answer of El Carmen populates. That skip logic doesn't make sense -CHILDST (D_632533534_439447560) == NR: response populates as New York, yet CHILDST_FOLLOWUP (D_264797252_451394598) question was still populated and correct answer of Valencia populates. That skip logic doesn't make sense

[huip-ims]

@KELSEYDOWLING7 It looks like the Questionnaire Word doc and M4 markup will need to be updated. The skip logic is failing when handling NR for CURWORKT7 and CURWORKT8.

CURWORKT9 is asked when CURWORKT7=0 (NO) or NR. However, the conditional display for grid CURWORKT10 states 'IF CURWORKT7=NO, then display rows ONLY for responses marked in CURWORKT9.' Should this be 'IF CURWORKT7=NO or NR, then display rows ONLY for responses marked in CURWORKT9'? If so, please update the Word doc.

CURWORKT8 is asked if CURWORKT7=1(YES). Otherwise it is skipped. The conditional display for grid CURWORKT10 states 'IF CURWORKT8=NO, then also display responses marked in CURWORKT2.' If participant is asked CURWORKT8 and provided NR, should the responses marked in CURWORKT2 be displayed in grid CURWORKT10? If so, please update the Word doc.

Currently, the markup is coded to display grid CURWORKT10 for the following:

1. If CURWORKT7=0 (NO) or NR, then responses marked in CURWORKT9 are displayed, Or,
2. If CURWORKT8=0 (NO) or NR, then responses marked in CURWORKT2 are displayed. The problem with this code is, if CURWORKT8 is asked and participant did not respond or CURWORKT8 is skipped (i.e., CURWORKT7 is not 1), both situations end up with CURWORKT8=NR and responses marked in CURWORKT2 are then displayed. This is what you're seeing when you're using your test responses to CURWORKT1-CURWORKT9. We need to know how to handle CURWORKT8=NR for when CURWORKT8 is asked and participant did not respond and CURWORKT8=NR due to CURWORKT8 being skipped.

[cusackjm]

@huip-ims I've added your suggestions to the qx document: https://nih.box.com/s/o816d6804w154fqf5y161ii7ngvcezqg

Let me know if this is correct, or if further edits need to be made

[huip-ims]

@cusackjm Hi Julie, if CURWORKT8 is skipped (which will also result in CURWORKT8=NR), we should NOT display responses marked in CURWORKT2, is this correct?

[cusackjm]

@huip-ims no, we should display responses marked in CURWORKT2

[FrogGirl1123]

@boyd-mj let me know if we need to make updates in the data dictionary relative to DAYSWORK

[huip-ims]

@cusackjm I've made the markup updates to address display issues for CURWORKT10. It's currently going through internal testing.

[boyd-mj]

The CURWORKT logic has been updated and will be included in the next delivery, along with the skip pattern fix for the CHILD address section. Note that we are now seeing that backend issue that @naiyume is looking into for CHILDCITY & CHILDST.

Per our conversation today, @cusackjm will update the DD to remove the duplicate DAYSWORK entry.

That leaves the PREWORKRES issue. We aren't able to identify any issues in the markup or recreate an issue during testing. Could NCI either re-test or provide more details on the testing scenario and what is occurring?

Thanks!

[KELSEYDOWLING7]

Great, thank you! @boyd-mj it is likely that PREWORKRES was simply a mistype when recording the answers. They said they answered no when instead they answered yes

[cusackjm]

@boyd-mj the old DAYSWORK was removed from the DD and is in github

[boyd-mj]

Great, then we should be all set pending re-testing!

[cunnaneaq]

CURWORKT issue looked good in testing of scenario 4 in the PWA

[FrogGirl1123]

Good news!

[boyd-mj]

RESPONSE: Tested again with @naiyume and @KELSEYDOWLING7. We think this is an error cause by entering city/state in CHILDCITY and CHILDST and then going back and deleting this info, which triggered the skip logic to display CHILDCITY_FOLLOWUP and CHILDST_FOLLOWUP. The deleted entry for CHILDCITY/ST was not deleted on the back-end, so Kelsey saw answers for both questions (which is incorrect). The auto-fill function could increase the likelihood of participants making this error.

ISSUE NOTED IN KELSEY'S COMMENT ABOVE: TEST SCENARIO 3 ISSUES (email= isabelleintersex@gmail.com , Connect_ID=9857825959) -CHILDCITY (D_632533534_128827522) == NR: response populates as Franklin square, yet CHILDCITY_FOLLOWUP (D_264797252_890792569) question was still populated and correct answer of El Carmen populates. That skip logic doesn't make sense -CHILDST (D_632533534_439447560) == NR: response populates as New York, yet CHILDST_FOLLOWUP (D_264797252_451394598) question was still populated and correct answer of Valencia populates. That skip logic doesn't make sense

@naiyume @danielruss has this issue been fixed or is it ongoing? Just want to be aware one way or the other as we test this section for the other fix. Thanks.

[danielruss]

@boyd-mj going back and forth can cause problems because we grab variables from the html elements. We don't clear the HTML elements because we want to allow the users to go back and forth while keeping there old responses. @naiyume, I think we need to clean the DOM on stepping BACK (or only use LF to get the responses, this would be a major, not my preferred option). What do you think?

[naiyume]

@danielruss I agree with this. The problem right now s just that the dom is not being properly cleared when going back. Once we add that, this issue should be resolved. There shouldn't be a reason not to clear the dom on back anyways.

[naiyume]

@cunnaneaq or @cusackjm issue should be fixed, could you retest when you have the chance?

[cunnaneaq]

6/14 testing - I entered all address fields for CHILDADD and then went back and deleted the city and state. Received CHILDADD2 and entered f/u city and state here. Piped address in following questions looked good. @KELSEYDOWLING7 I made notes in the testing spreadsheet, please let me know if you need clarification about what I did. Thanks!

[KELSEYDOWLING7]

I'm still seeing the deleted city and state in CHILDADD1 as well as the city and state entered in the follow up CHILDADD2 on the back end

[cunnaneaq]

Seems fixed in dev, meeting with @naiyume and @KELSEYDOWLING7 6/16 AM to check in stage/Firestore/BQ