Participatory Learning, Design and Research

[jamesmrollins]

@lzim wants a method to maintain the focus on learning with the MTL program. To that end, we need a method to continue to evolve the MTL program by gathering inspiration, criticism, ideas and other "eureka" moments when the user is so inspired. Some ideas discussed where:

1. A omni-present button that the user can click that brings up a form to gather data.
2. The form would have a few quick questions and a free-text field to gather information.
3. It will offer the user the opportunity for Team PSD to follow up.
4. It will offer the user the opportunity to join an advisory capacity for MTL.
5. The data would be stored, sorted and analysis to inform further development efforts.

[jamesmrollins]

Here is a button concept:

I intend to contact DEV to discuss technical feasibility and potential for execution before 5 May.

[lzim]

Thanks @jamesrollins

Note: I know there is a lot of work underway but we are looking at 5/2 not 5/5 as our key date to have this in place (if possible)

Our short list of draft prompts

Participatory Design

1. Feature request... MTL feature/function most...
   • ...loved
   • ...dreaded
   • ...wanted
2. Alert us something has changed...
3. Alert us something needs review...
4. User-experince
   • Satisfied with...
   • Frustrated with...
   • Confidence in...

Participatory Research

1. Was this helpful?
2. Top uses...
3. Top questions...
4. Most important factors...
5. Data you’d like to see...
6. More ways to partner...

Open ended ideas:

.

[jamesmrollins]

Seems that green, no matter what green, is just difficult to make compliant with WCAG - 508.

But, VA Evergreen on 25% VA Bright Green shows promise:

[lzim]

@jamesmrollins totally fine with the VA Evergreen on 25% VA Bright Green

• Let me workshop the questions this afternoon and give you an update to make sure we're better capturing what we need, particularly under the participatory research header.

I should be able to get you a final draft this afternoon!

[jamesmrollins]

Hi @lzim. We need to get this into DEV this week if possible in order to meet 3 May deadline. Do you have an estimate to completion of the form? Do you want me to work on it a little more, especially with respect to gathering quantitative data? Thanks, James

[jamesmrollins]

While using my Outlook to schedule a meeting with a client, I discovered this robust architecture to gather ideas and facilitate participation from users. It is a third party data collection service provided by UserVoice, https://www.uservoice.com/. I am going to investigate this a little more by talking with their representative.

How the buttons are presented:

General Information and dashboard:

The Ideas form and vote management.

The Bugs form and vote management.

[jamesmrollins]

@lzim - sorry, moved this discussion to the right thread!! I also spoke with Owen Barkley, from UserVoice to get information on pricing and such. He gave a link to a demonstration of the full range of the product - https://www.uservoice.com/product/demo/. He indicated to me the product costs between $270 and $1000 per month, depending on the depth of the product you use.

My impression is the product supports the decision making that Eric Ries describes in his book "The Lean Startup." It seems to cover all the avenues including integrating with other customer management systems.

[lzim]

@jamesmrollins @staceypark @TomRust

Thanks! This is somewhat close to what I was picturing, but at first glance, may deviate from what I'm hoping for in a couple of ways:

1) It's proprietary - UserVoice is too high a % of our budget for field testing/improvement - preferred, would be to see if we can develop anything more sustainable in house or even work with open source options. 2) The service product is designed to support R&D of a business (akin to Lean Start Up = Good), but we also need to translate into a form that advances science) - all the more reason need to find out if we can develop something more customizable to specific research questions that would generate generalizable knowledge.

We want to develop ways to test our "underlying leap of faith hypotheses" about our three participatory dimensions: a) learning, b) design, c) research.

Note Stacey Park and Tom Rust, this is yet another proprietary solution for the "knowledge base" akin to the one Microsoft proposed for the CRM.

James, how does this work? I'm worried about platform fatigue for our Team and our learners, so I'm very interested in knowing more about integration.

[jamesmrollins]

@lzim I think I am beginning to understand what you want in terms of user research and improvements- but I'm not totally there yet. I provided this suggestion, not necessarily to recommend purchase, but to see if I am in the ballpark. I do plan to see if there are other options available that may be free or open source - it would be good not to repeat efforts already available. I hope I am able to attend Menlo Park in a few weeks. We really, really need to spend time fleshing out your vision for this.

With regard to platform fatigue - I'm not clear what your definition of this is . . . meaning, are you concerned that people are going to stop using MTL resources because they are tired/overwhelmed/distracted? It seems to me, that usage is directly proportional to perceived value. And perceived value dependent upon the effectiveness of the platform in achieving care delivery improvements. A tall order indeed!

[lzim]

@jamesmrollins @tomrust @staceypark

Yes, we do need more time on it!

Re: “platform fatigue” I just mean learners already have to (at a minimum) navigate to and use to:

• mtl.how
• mtl.how/live
• mtl.how/data
• mtl.how/sim

By platform this is GitHub, Adobe, Microsoft and Forio.

• That means I’d like to “bake” our methods for testing and inviting input into these platforms, rather than add more.

• By baked in, I mean maximizing methods for passive data collection to inform tests, such as A/B version testing and more tracking of user activities than we do now.

[staceypark]

Cross-referencing Issue #410 where we also discuss the Participate Button. @lzim wanted to keep #339 open over #410 as this thread discusses the participatory research and language for the participate button.

[jamesmrollins]

Hi @lzim .

Situation In discussion of the design for the data collection algorithm and screens, DEV pointed out that we will already know the user's email, by virtue of them logging in either to the sim, or by requesting a demo login. We need the email in order to notify them that we have received their issue. If they want to join, we can also follow up.

The only exception to the above, is if we want the button on the MTL.HOW/DEMO landing page. In this case, we will not have their email, but at the same time I am not sure why we would necessarily need it.

Decision Needed: I recommend that we automatically collect the email. It is one less field for the user to interact with, and we already have the information. We would only need to drop the email field and amend the privacy policy, by dropping a sentence (see below).

As is: "In the following screens, we will not collect any personal information. We will ask you only for your email, which is voluntary. We will use your email to confirm receipt of your message and to contact you for any necessary followup. We will not share your email with anyone outside VA Team Participatory System Dynamics (Team PSD)."

Amended as: "In the following screens, we will not collect any personal information. We will use your email to confirm receipt of your message and to contact you for any necessary followup. We will not share your email with anyone outside VA Team Participatory System Dynamics (Team PSD)."

If they don't click the "Continue" button on the privacy screen, then they would be dropped from the algorithm. However, this would be the case no matter if we keep the email field or not, or change the privacy statement, or not.

[lzim]

@jamesmrollins

• Great point that learners have already provided their email.

But, given that is the case, can’t we simply say:

Recommended edit: “We will use your email to confirm receipt of your message and to contact you for any necessary followup. We will not share your email with anyone outside VA Team Participatory System Dynamics (Team PSD)."

• We don’t need the button on the demo landing page.

Thanks!

[staceypark]

@jamesmrollins @lzim This issue went to dev last week, but they are on holiday for the rest of the week and will likely turn to this on Sunday. We've update the due date to 9/20 to reflect that we will revisit this issue on that day if there haven't been any further updates.

[lzim]

@jamesmrollins

I know we identified radio buttons to collect structured data, not just open text data from our participate button.

However, there are two functions that we have yet to address that are top of mind for our consideration

1. The ability to upvote feature requests submitted by others.

2. Development of a shared community of practice space - a message board beyond email - that enables both learners and Team PSD members to exchange questions and answers.

The first function came up from the OMHSP ICs requesting a name change of a stock in PSY that 1) is not consistent with feedback we got at the frontline, and 2) could put us significantly behind to try to change now, rather than gather more data and consider it for a future release of MTL 2.0 it would be quite helpful to put it in a feature list and see whether or not it gets upvoted or not over time as compared to other requests that may come in.

The second function we have considered from the beginning as desirable, so that 1) we have a larger pool of potential answerers, and 2) answers have the widest possible reach.

FYI: @staceypark & @ritahitching

[jamesmrollins]

@lzim I will get on this. I think there are several apps that we can model. Something of a wiki that allows voting and discussion threads. I have been also thinking that we need an admin screen similar to what we have for the TeamPSD/MTL Demo site, were we can download database records, and perhaps put in custom flag-messages that flash on the screen during login and encourages people to navigate to the wiki and comment or vote on pending issues.

[lzim]

@jamesmrollins

I’m not sure whether the GitHub Wiki function can do any of this? But, as always, I am interested in fitting things in to our existing platforms (GitHub or Forio) if we can

[jamesmrollins]

@lzim @TomRust @staceypark @saveth @dlkibbe @branscombj @dlounsbu @ritahitching

Here is the PLD back-end design. As you know, "It happened on the way to the forum." So that's what inspired the name 😁. Please review design concepts below and comment. I am currently costing out development hours.

Issue Forum

Issue Manager

Bouncing Message Arrow Feature

Supporting Screens - Admin-user Main Menu

The Normal-user Main Menu

Import GitHub Issue Screen

Input Issue from SimUI

[lzim]

@lzim @TomRust @staceypark @saveth @dlkibbe @branscombj @dlounsbu @ritahitching

Please review design concepts below and comment. I am currently costing out development hours.

I am wondering if we could briefly talk about this during a call, after folks have had time to respond.

Perhaps after one of our MTL Facilitate trainings?

Thanks @jamesmrollins for your work on this!

[staceypark]

@lzim @TomRust @staceypark @saveth @dlkibbe @branscombj @dlounsbu @ritahitching Why don't we try to meet after our Friday 10/4 co-facilitation meeting at 12Noon Pacific / 3pm Eastern (gives everyone a break between the co-facilitation and this meeting)? This will give folks a week to review and comment. Please "like" if that works for folks.

I am wondering if we could briefly talk about this during a call, after folks have had time to respond.

Perhaps after one of our MTL Facilitate trainings?

[jamesmrollins]

@lzim @TomRust @staceypark @saveth @dlkibbe @branscombj @dlounsbu @ritahitching - I am good with Friday.

[staceypark]

@branscombj and @dlounsbu can do 11:30p-12Noon Pacific for half an hour on Friday.

[jamesmrollins]

@lzim we have a 45 hour estimate to put in the back end. Also, I think we will need a "Membership Manager" as well to manage all the people who "Join." What is our algorithm for handling new people? Currently, we send them an email saying that someone will be in touch. We will also need a procedure for handling "I have a question?" queries. I will put together some algorithms to facilitate discussion. ~ James

[branscombj]

@jamesmrollins @lzim @dlounsbu @dlkibbe @ritahitching @staceypark I entered some questions on the design screenshots. A couple of my Qs got answered as I kept going! Might make minor tweaks to the page text, if you want to provide it in a format that I can edit in track changes. jbquestions.pptx

[staceypark]

@lzim are we good to go on a 30 min this Friday?

[jamesmrollins]

@hirenp-waferwire, THIS IS GREAT!! Please see minor adjustments below. We also need to discuss transitions between menu selections. I would like to slow them down and use a "sliding" transition.

)

[staceypark]

@lzim @TomRust @staceypark @saveth @dlkibbe @branscombj @dlounsbu @ritahitching Hi all I just sent an hour meeting invitation for this Friday at 11:30am Pacific / 2:30pm Eastern. I know some folks can only join for half an hour due to conflicting schedules.

Thanks @branscombj for posting feedback. Others, please post feedback before the meeting so we can be efficient with our time

[saveth]

I don't have a strong preference on the aesthetic, but I am curious about how the data is being collected, stored, and analyzed.

JMR>> The data is being stored in an Epicenter date API and also exported to GitHub. I am very interested in your ideas about what exports or analytics we should build in.

[dlkibbe]

@jamesmrollins this is very cool. Did you consider making the "time left" a clock face so the user can easily see the difference between the 2 gauges? Question for @staceypark what is the easiest way to identify an issue number in GitHub - is it the keyword search or some other method? I'm thinking about @jamesmrollins image for the "Import GitHub Issue Screen" and how many people would easily be able to find the issue number (outside of you, @lzim and @jessfroe who have them memorized :-) )

JMR>> I think we could add a clock face with ease. My design assumption was that the individual entering an issue, by referencing a GitHub number, would know. If they don't know, they have an option to directly enter the issue into GitHub via MTL. I think having a keyword search function would be a good addition - I will make a note.

[staceypark]

1. If we don't want people's email information on GitHub, we probably don't want it in this forum either since it's public (is it public). Maybe we can do: J.Rollins or James R. format instead? James I see your comment about alias system since it is going back and forth on GitHub to Sim UI, but I'm thinking even for the forum page we may not want emails.

JMR>> I agree. We will be better off, just giving folks an alias.

2. I think we shouldn't call it "Issues" because that tends to signal things that are broken. What about "Ideas" or "Discussion Forum" or even "Issues & Ideas" Forum and we can separate the two by iconography.

JMR>> I think that we are too sensitive to the term "issue." I mean, in our public lexicon, we discuss issues, we resolve issues and so forth. I can get behind "Discussion Forum" as an alternative.

3. I see that you can import an issue in from GitHub, but what happens when Issues are created directly in the Forum? This will auto create a GitHub issue as well?

JMR>> Yes.

4. I think as the issues pile up, an expand/contract function would be useful. Instead of having two columns showing the picture and the comments, I'm imagining when you initially get to the page there are bars going across the screen with the issue title/description left aligned & a dropdown arrow right aligned. When you click on the arrow, it would expand/contract the issue so you could see the image and the associated comments.

JMR>> True. I will make a note. Hopefully, we will not be overrun with issues 😉

5. Sorting function by most commented, who submitted, most recently updated, shortest amount of voting time left, etc.

JMR>> Probably a good idea. I will make a note.

[jamesmrollins]

Hello @branscombj @dlkibbe @lzim @staceypark @jessfroe @ritahitching @saveth @holbrooa @TomRust . Below are some of Jane's comments and my responses regarding the participate button. Jane's comments are in the yellow boxes in the images.

1. An issue is entered into the forum via the issue manager panel. The issue can either be originated directly into the forum from MTL -or- it can imported by number from GitHub. In either case, GitHub will be the repository for issues. When an issue is created directly in MTL, it will be simultaneously coordinated with GitHub and assigned a number. The reason is isn't recommended, is so TeamPSD can review the issue prior to putting up on the forum.
2. The date on the issue title is the deadline. This may not be necessary for an issue in the forum, since the voting period will set the deadline for responses.
3. Yes, we will be able to organize it in many ways, but the default order will be by close date.
4. Keyword search is a good idea (see @dlkibbe comment in thread above).

1. Icons are wide open - what do you think makes sense?

2. Yes, the issue manager is only for those with a facilitator login. A user can create a new issue, but it goes to GitHub.

3. I agree, sending everything to GitHub would be prudent, because it will force a review by TeamPSD prior to posting.

4. A couple of things to clarify here - the message is entirely customizable, It can say whatever you put in. There is a maintenance window to accomplish this. The arrow bounces to get attention and is presented at login, because there is an issue in the forum we want them to look at.

5. A user can put an issue in at any time by pressing the Participate button. Go to TEST if you have a moment and try it out - it will clarify many things.

@branscombj I think you meant to put something here . . .

1. I am not sure what you mean on the first comment - please clarify.
2. Ok, I guess so 🤨😕😲😐😂 (just let me know which one).
3. No, not all options lead to the forum. The other options are for users to enter an idea, issue, like or question. The "Go to the Forum" puts the users into the Issue Forum. If you go to TEST, you can take a look at the other options in a live environment. That will give you a better idea how it works.

1. This screen is only available to a facilitator login.
2. I suppose, the bouncing arrow will have to focus on the latest issue. We may be better off just have a single source of a bouncing arrow message in the main Issue Manager panel. The bouncing arrow would appear for all users with a team member or team leader login.
3. The bouncing arrow would appear at login, with each new login. It would not carry over into the "Play" page. The user will also be able to dismiss it.

[staceypark]

Next Steps:

1. Focus on algorithm that flows from each of the 4 categories:

   • issues/bugs & ideas/features go to MTL.how/teampsd
   • questions & answers go to MTL.how
   • likes come to TeamPSD for tracking purposes

2. Core Principles:

   • Use as much open source, online existing technology as we can
   • Building community capacity to answer each others questions in mutual learning paradigm

[jamesmrollins]

Goal of the meeting to elicit feedback from people present, regarding the participate button.

Gather recommendations.

Recommendations:

1. Searchable question forum. I imagine like social media. Anyone who knows what they are doing can answer the question.
2. Do this entirely in GitHub
3. Knowledge Base function that catalogs Newsletters, Tips and Hacks, Troubleshooting and others.
4. Need to leverage the GitHub functionality as much as possible (such as WikI functionality).
5. Focus on the Algorithm that flow from each of our categories. 
6. Team PSD Repo needs to be a landing place.  
7. GitHub has the like/dislike function, how can we use this for everything from question and answers to ideas and new features.
8. Really need to work on the algorithm, use cases and validate it.
9. Put the original slide deck in the issue thread.

[lzim]

USE THE RESPECTIVE GitHub WIKIS to Provide Tutorials/Orientation

Framework for Interacting with Team PSD Repository - Interacting w/Team PSD

1. Bug/Issues
2. Ideas/Features - upvote/downvote

Framework for Interacting with Modeling to Learn Repository - Interacting w/MTL Community of Practice

1. Questions/Answers - upvote/downvote

Try to move the algorithm for how this would work from within the Sim UI out to PUBLIC open source (science) resources (GitHub).

• Include use cases for the algorithm.

[jamesmrollins]

@lzim @TomRust @staceypark @saveth @dlkibbe @branscombj @dlounsbu @ritahitching

Thanks everyone for helping me get this straightened out. I will get a algorithm put together hopefully by Wed next week.

[jamesmrollins]

@branscombj @dlkibbe @lzim @staceypark @jessfroe @ritahitching @saveth @holbrooa @TomRust

Distiguished Team PSD:

The link below contains a updated PLDR algorithm. Please study carefully and provide feedback. One note, I didn't think it a good idea to combine user inputs directly into the Team PSD repository. I honestly believe that it would be very disruptive to the workgroup lead's workflow, and users may be overwhelmed by what is all in there.

Thank you, James

participatory_design_algorithm_2.pdf

[branscombj]

@jamesmrollins This looks GREAT to me! My only question is, Does the mtl.how repository wiki page offer a place to enter a question that they still have after reviewing the help info? Re: migrating the help repository from the sim to GH, I'm not strongly on board with that because when you're in an app you should have access to info about it without leaving the app. But I do think the info in the sim needs to be pretty stable, static information where there might also be a more dynamic FAQ/wiki in GH?

[jamesmrollins]

Hi @branscombj , Thanks for the input Janie. I don't know how a user would get help if they are not able to find what they need in the wiki. I brought this up with @lzim during our discussion. We might need to send an email concurrently that instructs them on how to contact us in another way.

Regarding leaving the app for the wiki, that is a good point. I think it might be a good idea to start thinking about an admin panel to manage sim ui inputs. That way Admins could use it to dynamically update the SIM UI help screens (although we would need to reformat it), and maintain the "i" information in the data API, instead of on a spreadsheet.

[lzim]

@jamesmrollins and @branscombj Thanks for this!

@staceypark Should we have you run point on this on behalf of HQ?

• Thinking through what more frequent updates in the SIM UI would need to look like?
• Investigating the "MTL Community" Q&A on the mtl.how Wiki?

Lindsey

[staceypark]

@lzim I'm not sure how you are imagining the Wiki to look but it's not an interactive page. You wouldn't be able to write answers to questions using Wiki. It could however function more as a page FAQ place, where we post the Questions & Answers for people to search and read about.

I think if we want folks to interact in any manner, it would need to be through the Issues function like we have been using on the TeamPSD repo.

[lzim]

@staceypark

• Is there an example out there on GH that we want to model?

[staceypark]

@dlkibbe @dlounsbu @lzim @ritahitching @saveth Can you take a look at @jamesmrollins updated algorithm as soon as you can, so he can start on the design documents. thanks @branscombj for already giving feedback! 😄

Distiguished Team PSD:

The link below contains a updated PLDR algorithm. Please study carefully and provide feedback. One note, I didn't think it a good idea to combine user inputs directly into the Team PSD repository. I honestly believe that it would be very disruptive to the workgroup lead's workflow, and users may be overwhelmed by what is all in there.

Thank you, James

participatory_design_algorithm_2.pdf

[staceypark]

@lzim I will investigate this. We can transfer issues between repositories as long as they are under the same owner. @jamesmrollins and I are thinking that keeping the discussion type function under Issues in the mtl repository (instead of the teampsd repository) may be a good idea where we are sending our learners anyways.

@staceypark

• Is there an example out there on GH that we want to model?

[dlkibbe]

@jamesmrollins Two items: I like number 4 in your assumptions list! At the top of the algorithm where it says, "User has a need for information, has an issue or related support requirement." and Community of Practice Forum (CPF). How would a user know that they need to go to the Community of Practice Forum (CPF) or that it would be valuable for them to do so? I'm just curious how we drive users to engage if they don't know what they name means or the service it provides?

[jamesmrollins]

@dlkibbe you bring up good points. I don't know off hand, but I will keep this in mind as I develop the user tutorial that will likely be a part of this. When the user comes into the sim for the first time, a tutorial will present that explains what all the buttons are for. Perhaps we can focus their attention on the uses of the PLDR button. Thank you! James

[saveth]

@jamesmrollins I like the document. It's quite comprehensive. Something to consider when it's implemented is who will be monitoring all the issues coming through or the conversation on the forum. Will there be a tag or something to track who's posting and whom to respond to?

[jamesmrollins]

Thanks @saveth , you ask a good question. I'm not sure exactly how we will monitor. I think the vision is that we all kind-of passively monitor and that anyone in the community can respond. And by "we all" I mean anyone who participates in the forum.

[staceypark]

Note: @jamesmrollins points that we need to promote #339 (this issue), #620 and #775 at the same time as they are all in TEST. Currently, #339 is holding up promotion of the other issues.

[lzim]

Note: @jamesmrollins points that we need to promote #339 (this issue), #620 and #775 at the same time as they are all in TEST. Currently, #339 is holding up promotion of the other issues.

Okay @staceypark - Can we try to get James back on my schedule to go over this?

[staceypark]

@lzim @jamesmrollins Yes, we are scheduling you in for 1pm next wed

Note: @jamesmrollins points that we need to promote #339 (this issue), #620 and #775 at the same time as they are all in TEST. Currently, #339 is holding up promotion of the other issues.

Okay @staceypark - Can we try to get James back on my schedule to go over this?