search

medic / cht-core

The CHT Core Framework makes it faster to build responsive, offline-first digital health apps that equip health workers to provide better care in their communities. It is a central resource of the Community Health Toolkit.
https://communityhealthtoolkit.org
GNU Affero General Public License v3.0
438 stars 209 forks source link

Telemetry gathering for QA Release Test. #6661

Open newtewt opened 3 years ago

newtewt commented 3 years ago

What feature do you want to improve? We have telemtery that collects different things around the app the user interacts with. As part of our release testing, we collect some of these interactions manually with a stop watch. Here is the release test issue for possible further clarity.

Describe the improvement you'd like It would be great if we could collect these workflows through the app using telemetry that we can just harvest at the end of a run. These are the interactions we would need.

Time from logging on to finishing replication for first time log in with offline user.

Time from click the Reload button on the about page until the app is present. It goes to a refreshing page screen.

Time it takes to load the contacts tab from first navigation to the page.

Time it takes to load the reports tab from first navigation to the page.

Time it takes from clicking report on left hand side to it loading on the right hand side. Or in mobile view becomes the present page.

Time it takes from clicking submit on a form to returning to the app.

Describe alternatives you've considered Alternatives currently are to time it manually with a stop watch.

Additional context Add any other context or screenshots about the feature request here.

garethbowen commented 3 years ago

Some of this data is collected already in user telemetry. There may be functions we can add to the testing page to make this easier to access. @newtewt Can you have a look and see if we're collecting the right data, and if it's easy enough to access currently?

newtewt commented 3 years ago

Unless something is broken. I ran through these workflows locally and there are no telemetry items collected. I had 5 rows in the meta database. 3 were for read reports, 1 for the _design/medic-user and a feedback report.

garethbowen commented 3 years ago

Were you checking on the server? User telemetry is aggregated and replicated at the end of the month. Can you wind your clock forward and reload the app and then sync and check again?

We could add a button to the testing page to make this easier in future.

newtewt commented 3 years ago

I was looking at the indexed db for the _pouch_medic-user-ac2-meta database in the browser. Would they be saved somewhere else?

garethbowen commented 3 years ago

Before aggregation the telemetry records are stored in medic-user-ac2-telemetry-<uuid>.

newtewt commented 3 years ago

👍 I'll re-review what's collected and see what can be changed on this list.

newtewt commented 3 years ago

Reviewing the telemetry and the flows in the app I believe this is the correct flow to data mapping. @garethbowen can you review this for accuracy? I think after this review we might only need 1 new entry. Which I can then update this request.

Time from logging on to finishing replication for first time log in with offline user.

rules-engine:initialize
boot_time
boot_time:3:to_angular_bootstrap
boot_time:2:to_bootstrap
boot_time:1:to_first_code_execution

Time from click the Reload button on the about page until the app is present. It goes to a refreshing page screen.

Second iteration of 

rules-engine:initialize
boot_time
boot_time:3:to_angular_bootstrap
boot_time:2:to_bootstrap
boot_time:1:to_first_code_execution

Time it takes to load the contacts tab from first navigation to the page.

search:contacts:types

Time it takes to load the reports tab from first navigation to the page.

search:reports - reports page
search:reports:subjectIds

Time it takes from clicking report on left hand side to it loading on the right hand side. Or in mobile view becomes the present page.

I didn't find any telemetry for this one.

Time it takes from clicking submit on a form to returning to the app.

enketo:contacts:pregnancy:add:save caveat being the name of the report here.

newtewt commented 3 years ago

Also a side note. I think this would probably be easily automatable.

garethbowen commented 3 years ago

I think the tab loading times won't be accurate - search:contacts:types is only a small part of the overall loading time.

newtewt commented 3 years ago

Can we then add to the click and then page render or are you thinking about alternatives?

garethbowen commented 3 years ago

I think we should add the extra points. It will be a little limited because while we can time how long it takes to prepare the data, I don't know of any way to measure actually painting it to the screen. But it will be very reproducible and cover 90% of what we care about.

I'm internally debating whether we should gather this data in production or have some kind of "test" mode which gathers additional points. This sounds like useful data to gather from production instances too (eg: it could be impacted by poorly configured contact summary) so maybe we should just record all page load times.

newtewt commented 3 years ago

That sounds good to me @garethbowen. I'm not sure I understand where the point starts and ends well enough to be able to define it in a this ticket so that a developer could add it. I also have some time now that I could possibly add it if we had it defined well. Would you want to take a few minutes to chat about it?

garethbowen commented 3 years ago

I think we should add telemetry in the angular router so we record the rendering phase for all pages, but this will change significantly with Angular v10 so we may need to hold off until that is done.