Story J: User needs more accurate end-user time reporting of events in Sim UI and Data UI

[jamesmrollins]

Time Logging Inaccurate

When the user logs into the Sim UI and begins a session, if they don't close the browser or log out, the time clock keeps ticking. This leads to inflated user session hours.

• [x] Automatic logout may create an inconvenience for the user.
• [x] Investigate using SmartLook API to gather event information via an API.
• [x] Devise reports available from Facilitator Dashboard.
• [x] Devise events and funnels that indicate a desirable interaction pattern with the Sim UI.
• [x] Consider API calls to R-notebook for further analysis.

[jamesmrollins]

At 11/17 Workgroups Leads meeting, decided to move this to January Epic

[jamesmrollins]

One way to improve accuracy of time hacks, is to automatically log out the user when the page lays fallow.

See example below from VA Citrix

[dkngenda]

Hi James, should we move this to the ICE BOX instead of closing it?

[jamesmrollins]

Sorry @dkngenda - I meant to close #2338 as it was a duplication and got it backwards.

[jamesmrollins]

As per conversation with @lzim and @lijenn SWG 1.31

1. Not sure having an auto-close is a useful feature. While it would improve data collection accuracy. But creates an inconvenience for our end users.
2. @jamesmrollins is going to investigate SmartLook to see if we can export actual interaction time with the Sim UI via an API.

[jamesmrollins]

@hirenp-waferwire - We need to investigate if the SmartLook people provide an API. If so, can we export all the time a user is engaged with their Sim UI?

[jamesmrollins]

@lijenn @mnallajerla @lzim

Results of SmartLook API investigation

• Information regarding SmartLook Rest API and Webhook available here
• SmartLook has an API that will provide detailed information regarding events, funnels and visitors.

• A visitor is identified by the user's IP address. If a user is signing in using a virtual machine, then the IP may be nonsense. Further, some networks may block the IP, so identifying visitors may be unreliable.
• An event is something the user does:
  • Checks a box
  • Clicks a button
  • Navigates to a page
• Funnel is a series of events that leads to the designer's intended conclusion. For example, if a retail sales website design a funnel where the user searches, finds and views the desired items, selects them, checks out, enters credit card information and submits a sales order.

Recommended Next Steps

• Brainstorm a funnel that contains the desired events we wish to track using the Wednesday Sim UI Meeting.
• Determine what reports we would like to see that would support the teams, the facilitators and the researchers.
• Report these events to the Sim Ui using the API and generate report pages.
• Determine if we need an API call to support an R-notebook.

[jamesmrollins]

Discussed at the 2/16 Sim UI Meeting.

REF: PowerBI Story 13 and #2504

• Review R01 and IRR Grant and see what we said we would look at in terms of monitoring usage.
• Do we have an open story for tracking user behavior? If not, we should.
• We should provide equal focus on user behavior in Power BI, and start there. There needs to be a parallel between the Data UI and Sim UI because you would want to crosswalk the behavior across platforms.
• Is there a built in Power BI capability for monitoring user behavior?

[jamesmrollins]

FYI @lzim @jeffhoerle @lijenn @mnallajerla

Reviewed IIR Grant today for potential measurements regarding user behaviors.

From Zimmerman_HSRD_AIMS_Strategy_SUMITTED.docx.

Page	Source	Potential Measure
p. 8	Table 3 - Increase Learning/General QI Capacity	Trending increase of team and individual interactions with Data UI and SIm UI after session_12
p. 14	"Encourages safe prototyping of ideas, experiential learning and accumulates in new staff QI capacity over time"	Trending increase in teams' Team Data bookmarking, export and 100% experiment completion after session_12.
p. 14	"Stakeholders make theory-based, causal attributions about the system and QI plans that are structured in a model, and validated with calibrated parameters from health system data.107,110 A significant advance beyond self-report of implementation ‘barriers and facilitators’ alone"	Measurement same as above.

[jamesmrollins]

Support Workgroups Meeting 2/23:

From @jeffhoerle :

PowerBI Capabilities

The PowerBI Platform does have basic reporting tools for user interactions through "View User Metrics Report." These reports can be output via xlsx.

• Number of daily visits.
• Report Opens
• Report Page Views
• Unique Viewers
• Report Open Trend.
• Tutorial Page Views
• List of users and how many times they have used it during requested period.
• Report Usage Across Workspace (each VISN relative to each other)
• Provide number of views by page. This would give us an indication of exports as a proxy.

Automation Available for PowerBI via API.

In theory, there is likely a PowerBI API that could move this information between PowerBI and the Research Repo, of which theResearch Repo is not inside the domain. That would enable the Data Scientist to do R notebook calls to PowerBI to automatically pull this data. However, API calls from outside the domain are currently not allowed.

SmartLook Capabilities Via API

• Page Views
• Unique viewers
• Number of views by page.

Sim UI Capabilities Via API

• Module visits.
• Number of attending logins.
• Open and close time.
• Experiment Completion Percentages.

[jamesmrollins]

FYI @lzim @lijenn @mnallajerla @jeffhoerle

Thank you @lijenn for finding the R01 reference displayed below and for directing my attention to the appropriate grant documents.

From TeamPSD > Grants > NIH > R01_Resubmission_Mar2018 > ZIM.RESUB.R01.4.10.18.AppImageDraft.Final, p. 166

Platform	Measure	Measure Available Y/N?	Alternative/comments
Power BI	# Learners	Yes	N/A
	# Unique Visits	Yes	N/A
	# Clicks	No	Page Visits
Sim UI	# Clicks	Yes	Page Visits, Experiment Completion %
	# Simulations	Yes	See question below
	# Sims Shared	Yes	See question below
	Sim Inputs and Outputs	Yes	total completed QHFD?

Questions - Updated from Sim UI Meeting 3/2

• Is the correlation between the Measure and the session number critical as called out in the table below? No. The session # information in R01 Table 9 is outdated relative to the current 12 session plan.
• Do we want data from the CRM? Weekly Email w/Data Report seems to indicate that. Yes, the CRM was meant to produce more data on the email data feeds. Cross reference #2445. Counts of emails sent by Team PSD, counts by frontline teams and average response times on emails.
• Is "# Simulations" really, "# Simulations Used or Applied?" For any MTL Blue team, count number of individual, shared or team saved experiments, and % completion of experiments principally for teams.NEED TO TRACK Modeling to Learn 3.0 Release changes in the worlds. For the # of Simulations: For any MTL Blue Team # of Individual, Shared, Team simulation "SAVED Experiments" % COMPLETED Experiments Q/H/F/D principally for Team Worlds.
• Is the # Sims Shared the number of team meetings with multiple participants - or - the invitation of a unique member to view results (similar to sharing a journal article link via a text message)? We don't have a SHARE metric after all available for the Data UI nor the Sim UI platforms.

From R01

From TeamPSD > Grants > HSRD > iir_resub3_2018_12 > zimmerman_2019_iir_irb_scientific_protocol.

• Reviewed grant documentation and did not find any specific mentions of user-level measures similar to the R01 cited above.
• The closest to user-level measures was in Aim 2c, with regard to Participatory Measures as a mediator for QI.MTL Assignment improving EBP Reach. However, Appendix 2 describes these participatory measures via the use of psychometric instruments (participant questionnaires) and QI Activity Reporting.
• Lack of specific user-level measures guidance notwithstanding, one could construe increased Data UI and Sim UI usage via experiment completion coupled with attendance reporting, beyond training as an indicator of participation.

[jamesmrollins]

Sim Ui Meeting 3/2

• @jeffhoerle provided documentation and demonstrated what is available out of the box.
• @dkngenda please review this thread for information regarding grant required reporting about user interactions and analysis of what is supported by the platforms. Let's discuss how we can feed you the data you need for consultation.
• Please consult this link for information regarding Power BI Reports.
• @mnallajerla looked for videos in quant_workgroup channel related to data pipelines & didn't see older ones and pinged @dkngenda.
• @mnallajerla will ADD to README.MD. Will also add links to any "how to" videos, like the one we just had in quant_research.
• @lijenn will add to our Meta Data CATALOG:

  • how_to_video

  • user_story

  • data_feeds

• MTL Red & MTL Blue
• Page Visits is the only standard observation across Data UI & Sim UI. So, this is the most important measure for comparing the two arms of the trial.

[jamesmrollins]

As per conversation with @dkngenda:

1. APIs to move Page Access by user, by team, information between PowerBI and Sim UI to the corresponding R notebook is preferred. We need to confirm that PowerBI can link the user to the page access, vice page access statistics by team only.
2. @jamesmrollins needs to investigate PowerBI API potential usage inside the domain, and if it will report the required information.
3. @jamesmrollins will consult with @hirenp-waferwire regarding APIs and what may be required.

[jamesmrollins]

FYI @jeffhoerle @dkngenda @lzim @lijenn @mnallajerla

Hi @hirenp-waferwire - Please investigate:

• Does Power BI have an API we can use to move user (page access) reporting, by team, by user to an R notebook. Note to @lzim: API data transfer, should it be possible, would occur within the domain.
• What are the specifics of accomplishing the same task with either the Epicenter or the SmartLook API similar to what we did with ZenHub/GitHub reporting?

[jamesmrollins]

FYI @lzim @lijenn @dkngenda @mnallajerla

@hirenp-waferwire - To access the SmartLook Rest API, we have to buy up to the Business level ($95/mo) AND pay a $269/mo fee to use it. This is too cost prohibitive. So, we should:

• [x] Evaluate our current Google Analytics to see if we can get the same page access count analytics. We have the cookies for this service embedded in the webpages.
• [ ] Investigate for the sake of knowing - can we put Google cookies in the Power BI publication pages to gather page count data.
• [ ] Investigate how we can gather page analytics safely and in an approved way inside the VHA.

[jamesmrollins]

Hi @lzim @lijenn @mnallajerla @dkngenda

@hirenp-waferwire has determined how to get page access counts from Google. However, we have to do each query by user for a given date range. We can get the usernames from Epicenter and pair with the query, but we would still need some bounding variables so the search can be more efficient. Also, it would seem we would want to aggregate page access counts by group, versus by user. Please verify - thank you.

[jamesmrollins]

FYI @lijenn @lzim

Hi @dkngenda - Attached is the instructions for using the Google Analytics API to get page access data. I also sent instruction and invitation emails to your va and gmail accounts. API.docx

[dkngenda]

@jamesmrollins

Tried the instructions out and got stuck on step 3. Briefly, this is what I attempted

1. Generated a redirect URI as stated in step 2 (see screenshot below)

2. Added this redirect uri into the link provided in step 3 of the instructions (see highlighted section in link below)

3. Copied this link into google chrome address bar but then I got this error message

Any ideas what am missing here?

Thanks

[jamesmrollins]

Hi @hirenp-waferwire - Please take a look above at @dkngenda issues with the API. Can you offer any advice? Let me know if you two would like me to set up a meeting.

[jamesmrollins]

FYI @lijenn @lzim

@hirenp-waferwire @dkngenda and I met last night to troubleshoot this API. There is an issue with the confluence of R code and the java that was used to develop the API. In other words, a webserver is required to use the API. Moreover, there is a multitude of data available since we installed the tracking cookies. We decided on the following:

1. David will navigate to Google Analytics and manually download available data and consult with @lzim to determine which data parameters are needed.
2. Hiren will investigate R code versions of the API, so they can be embedded directly into the R notebook. Failing an R notebook solution -
3. We will investigate hosting the API on Epicenter and moving CSV data from Epicenter to a data repo in the research repo.

[dkngenda]

Hi @lzim @jamesmrollins

I went through google analytics data and documented data that I thought would be useful for Research/Ops activities. You can find the excel file here on Teams. @lzim if you could take a look and comment on the variable list, we can refine it further. @jamesmrollins: I was actually able to download this data directly through powerbi (and there is a lot more there) but I could not find a variable that captures a teamid on google analytics ... but you maybe able to.

FYI @lijenn @mnallajerla

[jamesmrollins]

FYI @hirenp-waferwire @lzim

Hi @dkngenda

@hirenp-waferwire wrote a sample code of google analytics API in R language. He checked it and it works. Please note that the Client ID and Client Secret are available in the file attached to the email sent 5/9 to your gmail account. It is the MTL self-registration APP, so no need to change the secrets.

Below are reference links.

1. google analytics install
2. metrics explorer

[jamesmrollins]

@dkngenda - did the above R code work for you?

[dkngenda]

@jamesmrollins Yes I am able to pull google analytics data into the notebook. However, I can't pull individual /teams data. @lzim Looks like we are able to track data for each page. Do we have specific pages we want to collect user data for.

[jamesmrollins]

Hi @dkngenda - thanks for the feedback. I will discuss with @hirenp-waferwire - I believe we had a programming strategy where we could enable an individual/team query by leveraging the Epicenter API, then loop it through the Google API. This can be accomplished with java, so we just need to figure it out with R code ( I think).

[jamesmrollins]

@dkngenda we will prioritize this after we complete the Team PSD 3.0 website update.

FYI @lzim

[dkngenda]

@lzim @lijenn

Is there anyone granting/reviewing requests to access the TeamPSD_datapipeline_trackers power BI dashboard. I see some coming through my VA email. Not sure if they have been acted on.

FYI: @jamesmrollins

[jamesmrollins]

@hirenp-waferwire we can turn our attention back to this task. @dkngenda needs a way to loop the search function by user so he doesn't have to put in the userid individually. We discussed using the Epicenter API to provide the userids.

FYI @dkngenda - I don't know who is responsible for granting/reviewing requests for access.

[jamesmrollins]

Hi @dkngenda - Approximately 2 weeks ago @hirenp-waferwire provided you with an API to loop request user IDs with Google Analytics to download use information for your R notebook. Did this code work for you - can you give us feedback please so I can close this task or modify the API so it works for you? Thanks, James

FYI: @lzim

[dkngenda]

Hi @hirenp-waferwire - I am still not able to link google analytics to simiUI. See the screen shot below to campare the two datasets.

As you can see the user ID in the two datasets are different. Is there a way of sending the ID from the simUI to google analytics so I can use it to link the two datasets?

FYI @jamesmrollins @lzim

[jamesmrollins]

Hi @hirenp-waferwire - please investigate and resolve if possible. Let me know if you need assistance.

FYI: @dkngenda @lzim

[jamesmrollins]

hi @dkngenda - @hirenp-waferwire says to try using the MTL User View in Google Analytics:

The data are available in google analytics:

I'm not sure if this is answering your question, please let me know if you need more information.

[dkngenda]

Hi @jamesmrollins - Unfortunately am still not able to pull the user id through an API call in Rstudio. I just read this article online (scroll all the way to the bottom of the article) that says you cannot get userid through an API call. However, it seems that you can create your own custom google dimension then load the user into it, which I can then pull into Rstudio through an API call. See how to create a custom dimension. @hirenp-waferwire might make more sense of this

[jamesmrollins]

Ok, @hirenp-waferwire and I will check it out.

[dkngenda]

@jamesmrollins and @hirenp-waferwire

I realize that last time you guys sent me a message on my VA email on this issue. I jus wanted to let you know that it's much easier to get my attention if you ping me on GitHub rather than over my VA email. If there are any updates on this issue let me know. Thank you

[jamesmrollins]

Hi @dkngenda . I don't think we will be able to derive userid based information from Google Analytics prior to July 8, 2022. The problem is the Google Analytics cookie gathered page view information, but assigned a unique ID number to the page visitor. There is no way to tell who the user is from that. However, we have added a custom dimension to Google, so now it is accurately reporting User ID, and should be exploitable from the API. Moreover, the SmartLook platform appears to have deleted much of our historical information for reasons I have not been able to investigate. If I can somehow get a mass download of information from the platform, that would be our only hope.

FYI: @hirenp-waferwire @lzim

[dkngenda]

@jamesmrollins @hirenp-waferwire

What the name of the custom dimension you created?

[dkngenda]

@jamesmrollins @hirenp-waferwire

I saw your email on the status of this work. I was able to use the API information you provided to pull some data which you can't really tie to a specific identifiable user. If you run the IIR R notebook chunks under the SIM data section, you will be able to see the information I'm pulling. However, as per my comment on Aug 4 on this this, I inquired about the name of the google user dimension you created so I can see if I can pull information tied to a user. Did you provide that information elsewhere other than on this issue?

Thank you

[jamesmrollins]

Hi @dkngenda - We reconfigured the API so that it should now be pulling user information by user ID. However, I hear you saying that the information is not available and you are asking if there are modifications to the API that you need to make to pull the right information. I will ask @hirenp-waferwire about this so perhaps we can clear this up. He will return to work Sunday night 9/4 and I will inquire with him. Thanks for your patience - James

[jamesmrollins]

@dkngenda I followed up with SmartLook to see if there were data we could download and to my horror found that all my data from years past was removed. The adjusted the license so that they only retain 30 days of data. So their only use for us is immediate troubleshooting of users who encounter problems where we need a video of their interactions. Otherwise their license for enhanced data collection is cost prohibitive; especially since google analytics is free, as long as you can arrange the API correctly.

FYI: @KimCase123 @lzim

[hirenp-waferwire]

Hi @dkngenda - The custom dimension name is "dimension1" for users' identifiers. The following image is a reference to how we used it for testing purposes. View Id is 265013385.

[jamesmrollins]

Hi @dkngenda - Can you verify if that dimension call above has solved the issue of getting data from Google Analytics? If you are satisfied with the API, then I can close this issue.

[dkngenda]

@jamesmrollins @hirenp-waferwire This is now working! Thanks for creating the UserId dimension

@lzim Below is an extract of the data as pulled into the R notebook. Are they other details, you would want to pull into this dataset?

[jamesmrollins]

Great news! Thanks @dkngenda and @hirenp-waferwire !