search

lzim / teampsd

Team PSD is using GitHub, R and RMarkdown as part of our free and open science workflow.
GNU General Public License v3.0
9 stars 23 forks source link

Sim UI - EDIT .xls EXPORT (CURRENT ISSUE) - OLD issue of the default values (RESOLVED) #247

Closed branscombj closed 5 years ago

branscombj commented 6 years ago

I'm working in PROD tonight, wondering if it's operating properly; maybe in flux with development and if so should I hold tight while UI changes are being made?

Browser, time, module, dataset, screen shots are on the attachment. Basically, I'm wondering if the widget in the model diagram shouldn't be reflecting the value in the team data table for Starting Rate when the switch is set to Use Team Data for Starting Rate. And the two base cases (with and without that switch set to On) yield Starting Rate charts that don't seem to jibe. (Didn't include those in screen shots.)

cc questions 102818.pptx

lzim commented 6 years ago

@branscombj and @jamesmrollins and @TomRust FYI @staceypark and @ericasimon

I'm not sure if I understand the issue. @branscombj can you take a screenshot of your inspector or developer information to help with any debugging?

branscombj commented 6 years ago

inspector or developer information? I don't know what that means. Just wondering if the current PROD version of the sim UI should be running properly right now because I'm seeing things like I attached above. If I should be using a different version, please let me know. I'm trying to build out session 5-10 scripts and need to know that what I'm seeing is correct. In the case above, I expected to see an oscillation in some outputs in the Base Case, but did not. And noticed that the widget, Team Data table, and output values didn't align. screenshots of outputs and home screen (widgets and team data table) for BCs using default (off) and On settings for Use Team Data for Start Rate. @TomRust does this look right? cc_bc_defaultstartratesetting cc_bc_defaultstartrate_homescreen cc_bc_teamdataforstartrate cc_bc_teamdataforstartrate_homescreen

jamesmrollins commented 6 years ago

Hi @branscombj !

I will dig into this and see what I can find. I will start with running the Vensim model to see what values it is returning versus what the Sim UI is reporting. This is going to take me a while to track this down, and I have doctor appointments this morning and early afternoon. I will try to resolve/explain today.

Thank you, James

jamesmrollins commented 6 years ago

@branscombj Hi Jane - here is an update. I have verified that the Sim UI and the vensim model (CC_PROD.vmf) are returning different results for the Starting Rate. Small difference between the variable and the Team Data could be explained at time=0, since the Starting Rate variable is calculated one simply passed through from the ModelParameters file (@TomRust please confirm). But, their should be no difference between the model run on my client version Vensim and the server version Vensim (Sim UI). I am not sure what the sudden change is, but we will investigate further. The are a few hypothesis we will follow:

  1. The server version of Vensim may have a bug. We have had this issue with the PSY model before. Forio did a system update last night.
  2. We are somehow not pointing the code at the right variables. This is highly unlikely since we haven't changed the code for quite a few months now, and normally the model would crash if not referring correctly to variables.

I will let you know what we find out.

Thanks for the report, James

lzim commented 6 years ago

Thanks @jamesrollins and @branscomj!

@staceypark are you able to put something together that explains how to capture any errors recorded by your browser, for example it’s called inspector under the develop Safari menu item and it’s under Tools > develop in Chrome.

It’s helpful to know how to do this for debugging some things.

Thanks!

branscombj commented 6 years ago

I'm still finding the default view and Base Case Run values in widgets and outcome charts do not line up with values in Team Data Table; and I see no oscillating-before-stabilizing characteristics in output charts in the Base Case results. Help! @TomRust @lzim @jamesmrollins @dlkibbe @dlounsbu PROD version; Chrome; Ind facilitation pilot world; MM with 640_a0_blue_2018_08_30.xlsx; CC with 640ha_stc_2018_07_23.xlsx image

bc_useteamdata_on

lzim commented 6 years ago

@branscombj @jamesmrollins @saveth @staceypark @ericasimon @joycepyang @dlounsbu @TomRust @dlkibbe @holbrooa

Thanks @branscombj for bringing this to the Team's attention! @jamesmrollins and I were on the line first thing this AM to discuss the issue, which Forio is investigating today. I have asked @jamesmrollins to provide updates to EVERYONE here, as you may find similar issues to those @branscombj identified until this is resolved.

We will be working to resolve ASAP! :rescue_worker_helmet:

jamesmrollins commented 6 years ago

I just spoke with Epicenter and they are tossing the dead chicken over the fence to me. According to their results, the model is running correctly in the Epicenter platform. Somehow, according to them, our code is not interpreting it correctly. Although I am unsure why, all of a sudden, this is the case as the model has been running for over 6 months now. Nevertheless, we will dig into their suggestion and see what we can resolve by Monday. Thank you for your patience . . .

branscombj commented 6 years ago

@jamesmrollins @lzim @TomRust @dlkibbe @dlounsbu A couple of folks have told me the sim's running properly again, but I'm still getting the results in the image above If those results for BC in CC with Use Team Data ON are correct, I would really appreciate if someone could give me a few minutes of time to talk through them. Debbie and I are having a hard time finalizing scripts for sessions 5-10 without being able to line up sim results with system stories as we understand them. Thanks!

staceypark commented 6 years ago

@jamesmrollins the sim isn't opening at all for me right now? I'm stuck on the "Please wait while redirecting to user's decision details" message.

@lzim @TomRust @dlkibbe @dlounsbu @branscombj anyone else experiencing the same problem?

jamesmrollins commented 6 years ago

I will take a look at it when I get home

dlkibbe commented 6 years ago

Yes, I've been having problems getting on but I thought it might be my dad's router... I have been having internet problems since Sunday due to rural connectivity issues.

branscombj commented 6 years ago

@staceypark @jamesmrollins I just logged in with no problem to 2 worlds, logging off in between.

jamesmrollins commented 6 years ago

@staceypark @branscombj @dlkibbe I just logged in as well. Sometimes, when the internet is slow, there are issues loading the Sim UI. But they usually don't last long. I was having some issues this morning, because I was also using Skype. Soon as I closed Skype, everything worked well.

jamesmrollins commented 6 years ago

@branscombj @staceypark @TomRust @dlounsbu @lzim @dlkibbe Pursuant to decisions made during a July 11 Lucid Standardization Committee Meeting, Takouba will ensure the following Team Data use settings are "on" for the following models and variables:

CC Model | "Use Team Data for Starting Rate" | "On" MM Model | "Use Team Data for Starting Rate" | "On" AGG Model | Select Mode | "Service Proportions from Team Data"

The decision was supported in part, to reduce the potential for confusion for the learner. Most teams would select to use their team data for a starting point. Therefore, it was thought that we should set these as the defaults.

If you have any issues with these decisions, please register them here before COB (PST), 9 NOV 2018.

Thank you, James

jamesmrollins commented 6 years ago

@branscombj @staceypark @TomRust @dlounsbu @lzim @dlkibbe Regarding issues surrounding the values portrayed in the widgets of the SIM UI, compared to values in the Team Data table, the following are findings of our investigations with a proposed solution:

Background: @branscombj noticed that the Starting Rate widget values did not match team data values in the CC model. I investigated this and determined that the default setting for "Use Team Data for Starting Rate" is in the "off" position, thus the model presented calculated data. Further, the model presents pre-equilibrium data valuables, which exacerbates the problem.

The simulation was originally designed to do "time stepping," which is a method of moving the sim to a particular point on the timeline between zero and week 104 (2 years) and then stopping. The user would then be able to look at outputs and determine new experimental settings for the remaining time and run the simulation to the next desired time stop. When the sim was tested, we found that the time-step function did not work as designed, because every time the user tried to iterate the simulation, it would back up to its pre-equilibrium time position (usually -500 or -1000 weeks). After some investigation @TomRust and I determined that to accomplish the time stepping function, we must use gaming variables for all the rates. This would have meant significant re-engineering of the models that would have affected schedule. The ability to incrementally iterate the simulation within the same session is not possible at this time, but is planned for future development.

Problem: The values portrayed in the widgets are pre-equilibrium values and the defaults of the "Use Team Data for Starting Rate" switches are in the incorrect default setting. Therefore, these values can be way off expected returns and create confusion for the learner.

Proposed Solution: Change the default settings to "on" for the "Use Team Data for Starting Rate" switches. When the user selects a session from the home screen, the simulation will present values at time zero. The user can then select experimental values, which will be entered into the model and run from the start (negative time position), through to week 104.

Please let me know if you have any concerns - Thanks! James

lzim commented 6 years ago

Thanks again, in particular to @branscombj @TomRust and @jamesmrollins for work to identify and address this issue. FYI: @staceypark @dlounsbu @dlkibbe @ericasimon as this is related to finalizing MTL 1.7 scripts for the MTL Video shoot in a couple weeks.

Despite, SD conventions about starting models in equilibrium, I concur with @jamesmrollins proposed solution to have the team data on as default, which is consistent with our standardization committee meeting decision over the summer (see Decisions in Lucid and refer to 7/11/2018 meeting for discussion and context) to:

  1. Always run the base case with all feedback loops on
  2. Always run the base case with team data to enable "apples to apples" comparisons of bc, exp 1, exp 2, and exp 3 system behaviors, in which the "what if" of "base case of no new decisions" accurately reflecting many teams out-of-equilibrium.

We did a lot of piloting, particularly with the AGG module, to identify this is that best approach for our stakeholders' needs.

Proposed Solution: Change the default settings to "on" for the "Use Team Data for Starting Rate" switches. When the user selects a session from the home screen, the simulation will present values at time zero. The user can then select experimental values, which will be entered into the model and run from the start (negative time position), through to week 104.

lzim commented 6 years ago

@jamesmrollins @saveth @staceypark @ericasimon - related to issue #245

@jamesmrollins question we are working on the export function with @saveth And we are trying to find out where it is recorded what the base case values are that the model reads, for example -0.01 or -1. Is that in the master crosswalk?

What we want @saveth to be able to list what the experimental values are from the export and filter out those values that were not changed (i.e., base case values), since those are already reflected in the team data table.

jamesmrollins commented 6 years ago

Usually, the base case signal value is the first negative increment of the scale. So if the increments are 0.01, then the value that signals the model to use base case is -0.01. The default value, scale and base case signal value is not uniformly indicated in the crosswalk, since we didn't have those values listed as a part of the format until agg. That's part of the reason why I have checked it out, because I need to correct all of that.

lzim commented 6 years ago

yeah @jamesmrollins that what I thought and why I put it in this issue, not just issue #245 We want to include the tt reports in the MTL Video shoot in St. Louis in a couple weeks, can you be sure to ping @saveth when you post the Master Crosswalk update?

She is going to be working on this over the weekend.

@jamesmrollins is working on this now, @saveth fyi @staceypark and @ericasimon

Usually, the base case signal value is the first negative increment of the scale. So if the increments are 0.01, then the value that signals the model to use base case is -0.01. The default value, scale and base case signal value is not uniformly indicated in the crosswalk, since we didn't have those values listed as a part of the format until agg. That's part of the reason why I have checked it out, because I need to correct all of that.

jamesmrollins commented 6 years ago

@saveth @lzim I have updated the master crosswalk table to include desired base case value information for all models.

saveth commented 5 years ago

@jamesmrollins thanks for the updating the master crosswalk. I used it to create the table of parameter changes the team made for each experiment in #271 .

jamesmrollins commented 5 years ago

Hi @saveth, If you have access to 1.7, try downloading an export file. It should be exporting an .xls file in columnar format. Let me know if you have difficulties. Thanks, James

jamesmrollins commented 5 years ago

Hi Savet, Will you see if you can open this file? Let me know if you have any difficulties.

Thanks, James

saveth commented 5 years ago

@jamesmrollins I don't have access to MTL 1.7. @staceypark can you export the file for psy from mtl 1.7 for me?

staceypark commented 5 years ago

@saveth Hi! Apologies I've been buried in other stuff. I'll send you a file and give you a login to the MTL 1.7 TEST in a couple hours as well.

staceypark commented 5 years ago

@saveth I was able to set you up with a login. The username is your VA email and password is the same one you've been using. Ran into some issues in pulling the export file however, which I reported to James on #257

staceypark commented 5 years ago

@jamesmrollins I just tried exporting a psychotherapy file from MTL 1.7 Test, but it gave me a .csv file instead of the .xls? Also got an error, which I reported in #257 . Not sure if that may have caused the .csv file issue though.

@saveth I'll still send you the .csv I pulled in case that's useful?

jamesmrollins commented 5 years ago

@staceypark @saveth The only model exporting .xls is the SP model in Test. We are attempting to resolve the time-step issue as the highest priority. After that is resolved, we will implement to export .xls to other models. Stay tuned! James

jamesmrollins commented 5 years ago

Good Morning @saveth @staceypark, the export .xls function is available for all models in Test.

staceypark commented 5 years ago

@saveth I ran and saved sample runs in all 5 modules here: https://github.com/lzim/teampsd/tree/master/quant_workgroup/mtl_viz

saveth commented 5 years ago

@staceypark I can't download the .xls file from the path you mentioned above. Nor can I seem to send the file to myself per my email. Help please.

staceypark commented 5 years ago

Hi @jamesmrollins Looks like I can export to .xls through both the export button in the Experiment Maintenance and through the Outputs on my VA machine using Chrome 🎉 😄

On my Mac, previously the Experiment Maintenance export button gave me a .csv file and the Outputs export gave me a TextEdit document. Now both export buttons give me a TextEdit document on my mac?

staceypark commented 5 years ago

@jamesmrollins The file format you sent to @saveth to work with does not match the exports I'm pulling from the sim. I've been trying to debug her code, and after changing the exports from ones today to the ones you sent, her code works again.

The exports are .xls files. I've tried changing them to .xlsx and that doesn't make the code run all the way through either. But using the .xlsx you sent makes it run?

I wonder if it's related to this issue: https://github.com/tidyverse/readxl/issues/480 Are we still forcing a different file extension to be .xls?

staceypark commented 5 years ago

Issue resolved - Download and resave as .xls