2024_10: ANDON: Sim UI PROD DDD initializations takes 10 seconds & drops with simultaneous "Go" clicks

matthewtomo commented 3 months ago

Discovered 7/29/2024 by Debbie, Laura, Jenn and Matthew in PROD instance.

Debbie, Laura, Jenn and Matthew all tried to run simulations at the same time.
This resulted in the Sim UI entering an unresponsive loading state that does not finish. In this first attempt, everyone was trying to run for the same team's team world.

We also tried to run simulations in different team worlds at the same time.
It resulted in the same outcome of the Sim UI entering an unresponsive loading state that does not finish.

Debbie informed us that this is a major issue, since facilitators are often having clinicians run simulations on their own computers at the same time.

tejaspesquare commented 3 months ago

Hi @matthewtomo,

As per screenshot, this timeout issue has been known for quite some time, but unfortunately, it cannot be resolved through the SIM UI. The resolution of this matter must be addressed by Forio.

matthewtomo commented 3 months ago

@tejaspesquare Do you know if this is related to our use of the V3 Forio API?

lzim commented 3 months ago

@matthewtomo

Will you please add a current clarification for our communication with learners using the simulation in the field?

[ ] FYI: @ljmoody & @lijenn - May need clarification messages in our MTL 3.7 Consult Manual related to this.

It sounds like @tejaspesquare is saying that current state is such that if any single user runs the simulation at the same time from any world Sim UI PROD fails. Is that correct?

If yes, recommend that this is escalated as an ANDON to communicate to draf clarying messages for field users.

FYI: @jamesmrollins

matthewtomo commented 3 months ago

@lzim, It looks like this is the case. I will be emailing David from Forio shortly to set up a meeting and uncover the cause of this issue.

@dlounsbu @dlkibbe This was the issue we encountered on the Monday support, systems_sme, & sim_ui workflows sync meeting. The Sim UI should still work as expected when only one person is using it. Avoid starting multiple simulation sessions at the same time.

fyi: @jamesmrollins @tejaspesquare @lijenn @ljmoody

Alert message included in the Sim UI:

The Sim UI is currently encountering an issue where simulation sessions will fail to run if more than one user attempts to start a session at the same time.

If after starting a session your page enters a loading screen for more than 30 seconds, the best course of action is to refresh the page and try again in a few minutes.

We are actively working to resolve this issue.

tejaspesquare commented 2 months ago

Hi Matthew,

@tejaspesquare Do you know if this is related to our use of the V3 Forio API?

This is unrelated to V2 or V3 API and not sure if this is linked with multiple user sessions. We have observed this error even when only a single user session is active.

ljmoody commented 2 months ago

@matthewtomo, when more information is known about this via David at Forio, please let me know if a clarification message is needed in the MTL 3.7 Consult Manual and what the limits are for users, etc. so the wording is specific and clear. Thank you!

matthewtomo commented 3 weeks ago

Discussed Dev Standup wk_1 2024_10 @matthewtomo @tejaspesquare

This issue is being worked on from two ends.
Dev is working on splitting up the calls to the PSY model, to avoid hitting the timeout error
Forio is working with us to develop a fix in the backend.
@tejaspesquare will be able to provide an estimate on this tasking by next Monday.
@matthewtomo will continue working with Forio to develop a fix for the backend.

lzim commented 3 weeks ago

@matthewtomo - Keep in really close communication with #systems_sme_workflow about the impact of this on their PREP.

FYI: @dlounsbu @dlkibbe

matthewtomo commented 3 weeks ago

Discussed Monday Dev Standup wk_2 2024_10 @matthewtomo @tejaspesquare

Dev has split up the calls for the PSY model, but they are still encountering the timeout issue on one machine when trying to start a session at the same time on two machines.
We are moving forward with a POC to get only the minimal amount of variables required at any given time to spread out the load of requests sent to the backend

lzim commented 3 weeks ago

@matthewtomo

Please estimate scope and timeline for impacted users and add to agenda for #9am_workflow leads

#consult_workflow MTL 3.7 Blue Consult PREP #systems_sme and workflows: #consult_workflow

FYI: @dlounsbu @dlkibbe Thanks!

lzim commented 3 weeks ago

wk2 2024_10 - #sim_ui_workflow @lzim @matthewtomo @ljmoody

ideas for helping with asynchronous communication and catching impacted users.

_1. Why/how_ -> rationale/intent -> why you're going about it the way you are.

2. ANDON -> impact on quality of relationship with partners, user experience.
3.Meetings: Slow velocity (story points/week) Decision or Discussion to unstuck.
4. Possible requirements and scope in story points and when delivered:

#workflow_users - Laura, Matthew, Lindsey #user_workflows - #consult_workflow #support_workflow

5. Rule out approaches by users, timeline & scope
[ ] 1.#sim_ui - Next Steps: Expand_ timeout threshold to 30 seconds -> Matthew is testing wk2 2024_10 Forio support thru Feb 2025 -> meets the VHA size scale of client
[ ] 2. #sim_ui - Message in the DDD in red font 2 pts -> Pop-up: "We detect a slow network. Please refresh your browser and click Go again."
[ ] 3. #systems_sme_workflow - @dlkibbe @dlounsbu need training in how to handle the delay live with sites add to wk4 2024_10 #9am_workflow leads.

Later possible after the #consult_workflow

[ ] #support_workflow - @ljmoody add this to https://mtl.how/manual - WAIT 3 pts

Ruled Out

[x] #sim_ui_dev - Ruled out - Message that alerts all users when someone is running a simulation because not possible to know -> message to prevent multiple calls that frame this has a high-use exciting innovation during product launch.
Current State Forio Epicenter 3.0 API cannot monitor multiple run IDs; there is not a way up way track all runs in all worlds.
[x] #ilead_workflow - Should we recalibrate our back-to-backs 2 hours.
Too late to readjust all the personnel on accreditation timelines.

matthewtomo commented 2 weeks ago

Discussed Friday Dev Standup wk2 2024_10 @matthewtomo @tejaspesquare

@matthewtomo @tejaspesquare will await response from David at Forio in order to ensure that our timeout limit increase is being handled correctly

lzim commented 2 weeks ago

Discussed Friday Dev Standup wk2 2024_10 @matthewtomo @tejaspesquare

@matthewtomo @tejaspesquare will await response from David at Forio in order to ensure that our timeout limit increase is being handled correctly

Thanks @matthewtomo - Let's keep a close eye on this for the consults next week!

matthewtomo commented 1 week ago

Discussed Friday Dev Standup wk3 2024_10 @matthewtomo @tejaspesquare

[x] @matthewtomo has reached out to Forio support on getting the timeout to handle properly
[x] @matthewtomo will CC @tejaspesquare on that email
Timeout parameter is not working when issued as a query parameter (it should be part of the request body for the POST request)
David from Forio mentioned that Forio Support might be able to better help with this issue
Timeout is currently set for 1 minute

fyi: @lzim

lzim commented 1 week ago

Thanks for the update on this @matthewtomo

I did not see this before we met.

From #9am_workflow leads today: @lzim @matthewtomo @ljmoody

We DECIDED that we do need to continue to search for solutions to the first behavior of the Sim UI at DDD log-in & initialization.
Review options, including more coaching, training, workflow and documentation solutions (not just DEV) at #sim_ui_workflow on Wed. Thx!

matthewtomo commented 1 week ago

Discussed #support_workflow wk4 2024_10 @matthewtomo @lzim

Current state of the timeout issue:

When multiple users are interacting with the Sim UI from different worlds, they are no longer encountering the timeout error.
- Example: @matthewtomo can click go in clinic_a_team world and @lzim can click go in other_clinic_b_team world at the same time and they should not run into issues where either of their sessions "hang" and become unresponsive.
When multiple users are interacting with the Sim UI from the same world, they can encounter timeout errors if they click go at the same time.
- Across four user sessions used in testing, no timeout errors were encountered if each user waited their turn to click go.
- In each of these tests, it took approximately 15 seconds for each user's models to initialize (and stocks to fill).

Decision

For consults, have each participant wait their turn to "click go" and wait for stocks to fill.

Action Items:

[ ] Make sure this decision/information is available in documentation (mtl.how/community)

fyi: @ljmoody @dlkibbe @dlounsbu

lzim / teampsd