search

medic / care-teams

For Product Management
0 stars 0 forks source link

contact_detail apdex: Define benchmarking scenario #72

Closed michaelkohn closed 3 months ago

michaelkohn commented 3 months ago
  1. Which config(s) will be used
  2. Which device(s)
  3. How many docs / data distribution for the user
  4. Detailed clickpath for the scenario
latin-panda commented 3 months ago

Initial data analysis I made related to this work

Configs

We will run performance testing on the config we are familiar with, and covering 3 major regions:

Devices

The phone use by Kenya deployment:

User type

Data

For CHWs

Iterations

We will run the automation tests using at least 1 user and simulate 5 days, each day running 10 times the suite.

Clickpath

As an offline user, access the application and do the following navigation for each contact type:

latin-panda commented 3 months ago

@michaelkohn I still have to do the 4 Detailed click-path for the scenario, but you can have a look and let me know your thoughts

The Nepal data you see there is what I could see from Raphael's test in our test instance for that config. I haven't properly analyzed that full data (the other ticket works) because it wasn't syncing to Postgresql, so I added the questions marks.

michaelkohn commented 3 months ago

Thanks @latin-panda, great start. I don't know what this will all look like when we get to the point of publishing it, but this is really useful in starting to shake out all the details 🙏🏼

Config

  1. It might be the case that the apdex score is already above .94 for some configs and not for others, that's totally fine. We don't need our baseline to represent every config scenario. Obviously it's most impactful if it helps more people, but it's also impactful if it just helps 1 big project.
  2. When identifying the baseline config, I'd imagine we'll want to be able to have a snapshot of it saved with our tests so that it can be recreated even if we end up changing the config in the future
  3. I don't think we need to document every nuance of the baseline config, but I do think it would be useful to briefly describe it in words to include some high level details like "contact summary has 5 fields on it, there are 2 condition cards", etc...

Devices

  1. I'd imagine we'll want to provide the specs / versions of the testing phone

User type

  1. During our call today, I mentioned that the "Supervisor" user should also be offline. I don't think the current test suite includes supervisors anyway and I'm OK with not including them for now... you mentioned it as a stretch goal which is fine, I'm also OK with removing it for now.

Data

  1. We'll want to know how many reports there are per contact

Clickpath scenario

  1. I imagine these will just be very simple like "Tap on a household from Contact Page List View" or "Tap on person from Contact Page Contact Detail View (Household)"
  2. And like you have set out in the current performance duration section, I think we should evaluate each level separately. For example (using a typical CHW Area/Household/Person hierarchy), baseline for... CHW Area, Contact of CHW Area (this is probably just the user's contact/self), Household, Contact of the Household.

Data

  1. I think we should be focusing on the apdex score for the given scenario (CHW Area, Contact of CHW Area, Household, Contact of Household). It's useful to see the counts because it gives a sense of scale to the tests but...
  2. Apdex treats tolerable and frustrated differently in the calculation, so we need to differentiate tolerable from frustrated. For example... your apdex score will be different if you have 5 tolerable and 0 frustrated vs. 0 tolerable and 5 frustrated.
latin-panda commented 3 months ago

I'd imagine we'll want to be able to have a snapshot of it saved with our tests so that it can be recreated even if we end up changing the config in the future

@michaelkohn Do you mean like the Apdex result so we can compare later?

latin-panda commented 3 months ago

@michaelkohn I have updated the info based on your comments. Do you think something is missing? If we are good, I will close the ticket as completed.

Togo Info will have to wait until next week when the tests are ready

michaelkohn commented 3 months ago

I'd imagine we'll want to be able to have a snapshot of it saved with our tests so that it can be recreated even if we end up changing the config in the future

@michaelkohn Do you mean like the Apdex result so we can compare later?

Nope, i meant keeping a snapshot of the actual config from when the tests were executed... I wouldn't expect it to change in the short term, but I can imagine in the future we may need to change the config to either accommodate some cht-core change or an important change to a project's config.

Do you think something is missing?

Performance baseline

I actually think we should remove this section since it is covered in https://github.com/medic/care-teams/issues/69.

Clickpath

Thinking about this some more, do you think it's useful to add details like "load a contacts list that has 100 hh's in it, select the 5th household"....

latin-panda commented 3 months ago

Nope, i meant keeping a snapshot of the actual config from when the tests were executed.

Yes, we have a snapshot of every config in the Fast UI folder.

I've applied the last feedback and closed this as complete.