Analyse partners telemetry closely

medic / cht-core

The CHT Core Framework makes it faster to build responsive, offline-first digital health apps that equip health workers to provide better care in their communities. It is a central resource of the Community Health Toolkit.

https://communityhealthtoolkit.org

GNU Affero General Public License v3.0

468 stars 217 forks source link

Analyse partners telemetry closely #5993

Closed garethbowen closed 4 years ago

garethbowen commented 5 years ago

We have been provided with a batch of raw telemetry data from a production instance. Do some analysis of the data. Keep in mind that this is a small set of data (it's from a single partner, over a single month, and a relatively small set of users) so we have to be a little careful about drawing conclusions.

Some brief analysis has been done which is enough to warrant a deeper investigation.

Possible outcomes for this work:

issues to improve performance of the project configuration
issues to improve performance of our app
issues to improve telemetry
scripts to make it easier to analyse telemetry data
documentation describing app performance and device recommendations

Spend around a day on this, then evaluate and discuss whether further effort is warranted.

kennsippell commented 5 years ago

I remember looking at the form completion rates for Muso was an interesting topic. The number of forms that get completed / opened. I'm not sure the outcome would be useful for the core team, but could really help as a useful indicator of quality for the partner as they design their forms.

garethbowen commented 4 years ago

Pushing back to 3.9.0 as this is a fair amount of work and I don't want to delay 3.8.0.

yembrick commented 4 years ago

@garethbowen we have a similar ticket to be worked on very soon thats being tracked in the Learning Roadmap https://github.com/medic/medic-impact/issues/180

That feels like an appropriate place for coordinating this analysis. If you agree, perhaps we could close this ticket and discuss there? @kennsippell has just assigned @senseibara to be the TL working on it. It would be great to have you participate!

kennsippell commented 4 years ago

My thoughts (but obviously this is up to the product team): It seems valuable to have somebody from the product team with dedicating time to look at telemetry data. Especially since this issue tracks a "single day", I'd recommend against closing it. I think their "product-focused" analysis is valuable, and it seems prudent to ensure that somebody from the product team (authors and stewards of the telemetry framework) have time set aside to dig in.

Spent around a day on this, then evaluate and discuss whether further effort is warranted.

yembrick commented 4 years ago

Hey @kennsippell I definitely support and encourage someone from the product team to prioritize time on this issue, for the reasons you said above. I definitely don't want to stop this work, or even change it, just update how we are coordinating on it. From a documentation staindpoint, overall, as we get our product backlogs and learning roadmap set up and in wider use, Im trying to pay attention to reducing duplication across. As many of the product and learning roadmap tickets will involve people from both teams, we should be comfortable assigning tickets and working on things across teams/roadmaps. As the bulk of our anslyses live on the learning roadmap, it makes sense to me for this to be there, solely from a tracking perspective. Hopefully that would also allow us to better coordinate the other telemetry analysis being discussed. Thoughts?

MaxDiz commented 4 years ago

Hi @yembrick Let's add this to our call agenda next week. I think we could benefit from a more concrete plan for how to manage between teams

yembrick commented 4 years ago

sounds good!

helizabetholsen commented 4 years ago

Checking in on the follow-up conversation referenced by @yembrick and @MaxDiz re: overlap between R&L telemetry analysis and proposed metrics and product team exploration / use of telemetry data. This could be something for us to discuss @garethbowen during our call early next week.

garethbowen commented 4 years ago

Key stats from the product perspective...

Doc count mean: 3182. Quite high, could use purging.
Boot time mean: 238s
DB warming time component: 78%. This metric has been reworked in later versions but highlights how much time it takes to query map/reduce views
Boot count mean: 44. App was booted more than once a day on average
Report search by subject ID count mean: 1356. Queried when selecting contacts.
DB map/reduce query performance is roughly linear with the number of docs.

The DB warming taking 78% of the boot time, and the 1300+ report searches, highlights to me the importance of performance optimisation of the map/reduce queries in PouchDB.

I've used the time allocated for this issue so I'm going to close it off. If anyone has any specific questions let me know and I can dig in further!

kennsippell commented 4 years ago

@garethbowen What project's telemetry data did you look at and over what time span? What version of the core do these numbers represent?

garethbowen commented 4 years ago

Muso, August 2019, 3.5.0 & 3.6.0.