3/16: Resolve all Post Tests w/EES and Sessions 11 and 12

[staceypark]

Create MTL Live post-tests for sessions 11 & 12 and hand off to Elizabeth for MTL Video pilot review and for go_live

[dlkibbe]

@staceypark @ritahitching @branscombj can you tell me where the video links are now? I went to the mtl.how/document to find them but the links aren't working? I need to see Session 11 and 12 videos to ensure the post-tests are comprehensive; using the say guides but need the videos to help me think about other knowledge questions. I also searched for video links in the Issues list.

[staceypark]

@dlkibbe they are the ones at the very bottom of #206

[dlkibbe]

Thank you @staceypark :-)

[staceypark]

@dlkibbe how are these coming?

[dlkibbe]

@lzim @staceypark @branscombj @ritahitching @Bowlied I had an email exchange with Elizabeth for clarification on post-test requirements. She indicates we could have 9 and 10 be true/false or MC and could keep the open-ended questions as post-test items 11 and 12. However, she cautioned on "adding too many fields for users to fill in. They have to complete and pass the post test, then complete the evaluation. One of the most common complaints we receive in the open ended question is how many fields a user has to complete." Suggestion: Session 12 Post-test has 11 questions: question 9 would become knowledge Q, question 10 would be a multiple choice Q that has "check all that apply" as an option and would relate to the most helpful or useful components of MTL. Q11 would be open ended regarding improvements. Thoughts on this? Should we just stick with 10 knowledge questions and figure out how to get open-ended feedback in other ways?

[branscombj]

@lzim @staceypark @branscombj @ritahitching @Bowlied I had an email exchange with Elizabeth for clarification on post-test requirements. She indicates we could have 9 and 10 be true/false or MC and could keep the open-ended questions as post-test items 11 and 12. However, she cautioned on "adding too many fields for users to fill in. They have to complete and pass the post test, then complete the evaluation. One of the most common complaints we receive in the open ended question is how many fields a user has to complete." Suggestion: Session 12 Post-test has 11 questions: question 9 would become knowledge Q, question 10 would be a multiple choice Q that has "check all that apply" as an option and would relate to the most helpful or useful components of MTL. Q11 would be open ended regarding improvements. Thoughts on this? Should we just stick with 10 knowledge questions and figure out how to get open-ended feedback in other ways?

Hi @dlkibbe , I think we built in as much feedback as we felt we could to the evaluation, so I would stick with the minimum required post-test questions for accreditation. I don't think "check all that apply" about value of MTL qualifies as a measurable post-test question since there's not a right and wrong answer. T/F questions are easier to come up with so I'd suggest using that kind if you still need 1 or 2 more for any of the post-tests. Thanks!

[staceypark]

Thanks @dlkibbe and @branscombj I agree with Jane. Let's stick with 10 knowledge Q's for the post-test that are measurable.

I think the evaluations and baseline/6 month surveys we are doing per the grant, will be enough space to give general feedback.

@lzim @staceypark @branscombj @ritahitching @Bowlied I had an email exchange with Elizabeth for clarification on post-test requirements. She indicates we could have 9 and 10 be true/false or MC and could keep the open-ended questions as post-test items 11 and 12. However, she cautioned on "adding too many fields for users to fill in. They have to complete and pass the post test, then complete the evaluation. One of the most common complaints we receive in the open ended question is how many fields a user has to complete." Suggestion: Session 12 Post-test has 11 questions: question 9 would become knowledge Q, question 10 would be a multiple choice Q that has "check all that apply" as an option and would relate to the most helpful or useful components of MTL. Q11 would be open ended regarding improvements. Thoughts on this? Should we just stick with 10 knowledge questions and figure out how to get open-ended feedback in other ways?

Hi @dlkibbe , I think we built in as much feedback as we felt we could to the evaluation, so I would stick with the minimum required post-test questions for accreditation. I don't think "check all that apply" about value of MTL qualifies as a measurable post-test question since there's not a right and wrong answer. T/F questions are easier to come up with so I'd suggest using that kind if you still need 1 or 2 more for any of the post-tests. Thanks!

[lzim]

@dlkibbe @lijenn @branscombj

Since go_live ends today, I'm moving this to teams_data. But, since EES is asking to resolve post-tests questions this week, and we still need them to post videos to TMS ASAP. I'm hoping we can still do this one this week? I've edited the due date above. Thanks!

[branscombj]

These are the edits needed for the Post-Tests. @dlk and I will check off as completed.

Overall feedback:

• [x] Please make the questions and response choices shorter and less complicated. Particular sessions noted:
• Session 2: concerns about retention of information at the level of detail in some of the questions.
• Session 5 - Concerned that learners may have difficulty retaining the information needed for the post-test.
• Session 7 - The questions do not seem to be spread out to cover the session, but seem to be focused on information that is lumped together.
• Session 8 - There were a lot of steps covered in this video with a lot for learners new to MTL to recall for the post-test.
• Session 9 - Too complex. Most of the answers will require a side by side monitor to translate the verbal into the point and clicks that take a person from one screen to another.
• Session 10 - Too complex. Most of the answers will require a side by side monitor to translate the verbal into the point and clicks that take a person from one screen to another.
• [x] Make sure Q/A are provided explicitly in the session and don’t require inference. (See for example S03-Q10.)
• [x] Remove all “D-All the Above” responses. (I assume that we need to remove other multiple-correct-answer responses too – see for example S01-Q3.)
• [x] Remove all Fill in the blank responses.

Specific questions: Session 1:

• [x] Question 2: This was not covered in Session 1.
• [x] Question 3: Reviewer thought the correct answer should be (d).
• [x] Question 4: Reviewer thought (d) should be the correct answer here also.

Session 2:

• [x] Questions: 1 and 2 – Unsure this content was covered in Session 2. Was this previously covered in Session 1 and if so, should it be on the Session 1 post test instead?
• [x] Question 5 – Not completely clear from the video that access is restricted only to Clinics within your VISN.

Session 3:

• [x] Question 10: Reviewer thought (d) should be the correct answer.

[lzim]

Thanks @dlkibbe @branscombj 🤩

FYI @lijenn

[branscombj]

Things noticed while editing Post Tests:

• A See guide (S07?) with the YAML header not coded to be hidden - see bug report
• S08 and S09 have identical learning objectives - did we mean that? I guess so; it's probably fine.

[branscombj]

@MattMoore2 @gyiwamasa @hmarcia @theresaschmitz @JayLCohen @clairecollie Thank you so much for your feedback on the post-tests. It is compiled in my comment above for reference, and @dlkibbe and I have finished editing them accordingly.

I would be SUPER appreciative if 1 or 2 QIICs and 1 or 2 others from TeamPSD (@staceypark @lzim @lijenn @jessfroe ) could review them (today!) for clarity before we give @JoeBloomquist and @Bowlied the green light to post them in TMS.

They are compiled in a single Word document, attached, for ease of adding comments or edits in Track Changes: posttests1thru12.0312.docx

THANK YOU, in advance, if you can do this!

[gyiwamasa]

I took a quick look. Please find my suggested edits attached. Thanks for changing several of the items such that the response choices are not lines and lines long.

A few comments: --It concerns me that ALL of the correct responses are choice “a.” --There is heavy reliance on “blah blah blah except” phrasing of items. --Likewise, there are NO items that include “all of the above” as a choice.

I would suggest changing some of the “all except” items into “all of the above” choices. Hope this feedback is helpful.

[branscombj]

@gyiwamasa THANK YOU that is Super helpful! I will make the changes you suggested in the Word document. Some background on the points below:

It concerns me that ALL of the correct responses are choice “a.”

We put all correct answers as "a" as a signifier to EES for their coding. They will randomize the response order.

There is heavy reliance on “blah blah blah except” phrasing of items.

This is sort of a "work-around" for the below. I will revise some of these to rely on them less often, or see if there's another way to do those questions.

Likewise, there are NO items that include “all of the above” as a choice.

EES asked us to eliminate this type of answer. I believe the reason for it is that it means they can't randomize the answer choice order.

We figured out that the places where yall ran into incorrect-looking answers in the post-tests were where the correct answer was not "a" (as it said at the top), but "d. All of the above" or "d. a and c only", etc.

[branscombj]

Revised with @gyiwamasa 's suggested edits. Some stats on this version:

• Flesch-Kincaid grade level = 7.8
• 7/120 questions wrap past one line (no answer choices wrap)
• 16/120 questions are T/F; no more than 2 per test
• 13/120 questions are "all of the following except"; no more than 2 per test

posttests1thru12.0313.docx

[branscombj]

@staceypark Got a minute? I'd like to send to @JoeBloomquist before I leave. I'm especially hoping for a look-through for these 2 things:

1. Anything jump out as not what someone inside the VA would say?
2. Anything questionable or wrong from a MTL point of view?

[lijenn]

@branscombj @dlkibbe Hi Jane and Debbie, HQ will review the post-tests with those 2 items you mentioned in mind by COB of Monday 3/16.

Folks here at TeamPSD are remote working for the first time and it has been a learning process, thank you all for your patience as we are getting our bearings!

FYI: @JoeBloomquist @staceypark

[gyiwamasa]

Thanks Debbie. Now I know why some post test questions and answers for TMS are really crummy! Not necessarily the presenters’ fault!

[gyiwamasa]

Thanks Jane!

[JoeBloomquist]

Good afternoon Jane and Deb,

It is my understanding all of the post test questions are being revised, based on Gayle’s comments below.

Can you please confirm which posts test are considered final?

Gayle's Comments: --It concerns me that ALL of the correct responses are choice “a.” --There is heavy reliance on “blah blah blah except” phrasing of items. --Likewise, there are NO items that include “all of the above” as a choice. I would suggest changing some of the “all except” items into “all of the above” choices. Hope this feedback is helpful.

[branscombj]

Good morning, @JoeBloomquist . The final proposed post-tests are in the Word document attached to my March 13 (2:31pm) comment above, incorporating Gayle's feedback of the same day. They need to be reviewed by @staceypark , @lijenn , and/or @lzim before we post them in TMS; but I believe the start dates for MTL are being delayed, so you should still have the time you need for doing that.

[staceypark]

@lzim and @branscombj I made some edits and comments throughout. I also tried to italicize all the instances of Modeling to Learn or MTL as I came across them. posttests1thru12_sp_edits.docx

[branscombj]

I made some edits and comments throughout.

@staceypark Those are fabulous - exactly what I was hoping you would notice and improve! Accepted/incorporated.

I also tried to italicize all the instances of Modeling to Learn or MTL as I came across them.

Great! I was going to do a search-and-replace for this. I ran through and changed the stragglers.

@JoeBloomquist The final set of post-test questions form MTL Blue and MTL Video is in the document below. Notes about formatting:

1. The correct answer is shown as option (a) for every question. Please scramble the order of the answer choices.
2. Our MTL style guide has "Modeling to Learn" and "MTL" italicized except when shown as part of a web address or in quotation marks. If you can, we'd appreciate your keeping that formatting.
3. There are a few places where a word is underlined for emphasis - e.g., the "not" in a question stem. It will be really helpful to test-takers for these to have formatting for emphasis - thought it could be bold instead of underline if needed.

Please let us know if you have any questions, and thank you for your patience and assistance!

posttests1thru12_final.docx

[staceypark]

Thanks @branscombj @JoeBloomquist 🎉 🎆

[JoeBloomquist]

Good morning all! I now have the post test and my team and I will get this started on our end. Thank you all for your efforts and stay well!

[JoeBloomquist]

@branscombj, @staceypark. Please remember, we are only missing the Pilot review for Session 4. The posttests are all in, just need that last session.

[JoeBloomquist]

@branscombj, @staceypark, Good morning! Marcia Hunt just submitted Session 4, yeah! She recommends revising Post Test question #9 for session 4. Will you be doing any further revisions to the posttest at this point?

[branscombj]

Hi @JoeBloomquist ! Can you tell me what the S04 question 9 was that she was given - was it from the new post-tests or the old ones - and what her comment was if she said anything specific. I'm not sure if we may have already addressed her concern?

[JoeBloomquist]

@branscombj, I don't know the version she was given, her email states Lindsey asked her to review session 4, as others could not. Her statement is: "Please look at question #9 in the post-test. I found it confusing as there does not seem to be a correct answer without data about the team’s actual vision and priority. Excellent video training."

[branscombj]

Thanks for that @JoeBloomquist - That comment makes sense in re: the old question 9:

9. Based on your team’s vision and the highest priority need selected by your team, which MTL simulation module do you think your team should use?

That question has already been removed and replaced in the new posttests, for the reason that Marcia states. The new one is:

9. By the end of Session 4, Modeling to Learn teams have completed all of the following except

[JoeBloomquist]

@branscombj, thank you, very much for checking on this. We will get this submitted.

[branscombj]

@lzim @lijenn @dlkibbe @staceypark Here is where we thought we'd wrapped up the Blue/Video post-tests. I've noticed that the version called final (attached above on 3/19) has at least one question that's now outdated, about browser limitations. I am replacing that question and checking for any other issues in all the sessions. Then I will revisit MTL Red post-tests, since I modeled them after the old version of Blue. Should have clean post-tests for both by this evening.