search

medic / care-teams

For Product Management
0 stars 0 forks source link

contact_list apdex: Establish baseline #90

Closed michaelkohn closed 2 months ago

michaelkohn commented 2 months ago
  1. Document clickpath for measuring contact_list
  2. Document the "BEFORE" apdex score for this scenario
Benmuiruri commented 2 months ago

Background

Configs

The baseline "BEFORE" apdex score will be obtained from running the test suite in these three configs

Devices

To be confirmed

User type

Data

For CHWs

Iterations

We will execute the automation tests for 1 user and simulate 5 days, with the suite running 10 times each day.

Clickpath for measuring contact_list

According to the test plan documented in this comment. We expect to have 3 entries of contact_list:load for each test run. As an offline user, access the application and perform the following navigation:

  1. I can view the CHW's Area
    • Open CHT app
    • Visit the Contacts Page List view
    • (contact_list:load is recorded once)
    • (contact_list:query is recorded once)
  2. I can view a person within the household
    • Open CHT app
    • (contact_list:load is recorded once)
    • (contact_list:query is recorded once)
  3. I can submit a report for a newly created person
    • Open CHT app
    • Visit the Contacts Page List view
    • (contact_list:load is recorded once)
    • (contact_list:query is recorded once)
    • Tap on a household
    • Create a new person
    • (contact_list:query is recorded once)
    • Create a report for that person
    • Go back to the Contacts Page List View
    • Search for the person you just created
    • (contact_list:query is recorded once)

At the end of the test (Running the test 10 times for 5 days) we expect to see:

  1. 150 entries of contact_list:load
  2. 250 entries of contact_list:query
Benmuiruri commented 2 months ago

Hi @michaelkohn

  1. is the above clickpath and summary sufficient and correct ? Or is there something else I should include ?
  2. Is the "BEFORE" apdex score for this scenario the one from the test party data or will it be the value after running the test suite as outlined in the click path ?
Benmuiruri commented 2 months ago

Hi @michaelkohn, due to the issues I am facing with the automated testing I decided to get a baseline using manual testing.

I ran the manual testing 30 times how I outlined it:

  1. Open app, which loads Messages tab by default (Or Tasks for Togo and Mali),
  2. Click on the Contacts tab.
  3. Scroll to the bottom of the contacts to load all contacts
  4. Click one household
  5. Click back button to contacts list
  6. Search for a contact
  7. Click back button to contacts list
  8. Close the app and close it from background as well.
  9. Open the app again and repeat steps 1-8, 30 times in the same day, then sync.

Apdex Score = ((Satisfied Count) + (Tolerating Count / 2)) / Total Count

Kenya - Phone Tecno Pop 7 - contact_list:load = 0.77, contact_list:query = 0.88 ```json "contact_list:load": { "sum": 267504, "min": 2092, "max": 162416, "count": 30, "sumsqr": 26868328860 }, "contact_list:load:apdex:frustrated": { "sum": 162416, "min": 162416, "max": 162416, "count": 1, "sumsqr": 26378957056 }, "contact_list:load:apdex:satisfied": { "sum": 42145, "min": 2092, "max": 2939, "count": 17, "sumsqr": 106401163 }, "contact_list:load:apdex:tolerable": { "sum": 62943, "min": 3008, "max": 8022, "count": 12, "sumsqr": 382970641 }, "contact_list:query": { "sum": 528342, "min": 1158, "max": 162013, "count": 113, "sumsqr": 41920686882 }, "contact_list:query:apdex:frustrated": { "sum": 298102, "min": 14690, "max": 162013, "count": 3, "sumsqr": 41201725470 }, "contact_list:query:apdex:satisfied": { "sum": 133062, "min": 1158, "max": 2897, "count": 90, "sumsqr": 208120866 }, "contact_list:query:apdex:tolerable": { "sum": 97178, "min": 3471, "max": 7512, "count": 20, "sumsqr": 510840546 }, ```
Mali - Phone Tecno Pop 7 - contact_list:load = 1.00, contact_list:query = 0.99 (Mali CHWs don't have reports) ```json "contact_list:load": { "sum": 43652, "min": 295, "max": 2745, "count": 31, "sumsqr": 76434764 }, "contact_list:load:apdex:satisfied": { "sum": 43652, "min": 295, "max": 2745, "count": 31, "sumsqr": 76434764 }, "contact_list:query": { "sum": 111828, "min": 112, "max": 39022, "count": 90, "sumsqr": 1613727640 }, "contact_list:query:apdex:frustrated": { "sum": 39022, "min": 39022, "max": 39022, "count": 1, "sumsqr": 1522716484 }, "contact_list:query:apdex:satisfied": { "sum": 72806, "min": 112, "max": 2335, "count": 89, "sumsqr": 91011156 }, ```
Nepal - Phone Tecno Pop 7 - contact_list:load = 0.97, contact_list:query = 0.98 ```json "contact_list:load": { "sum": 85369, "min": 680, "max": 38758, "count": 30, "sumsqr": 1584216519 }, "contact_list:load:apdex:frustrated": { "sum": 38758, "min": 38758, "max": 38758, "count": 1, "sumsqr": 1502182564 }, "contact_list:load:apdex:satisfied": { "sum": 46611, "min": 680, "max": 2609, "count": 29, "sumsqr": 82033955 }, "contact_list:query": { "sum": 229962, "min": 159, "max": 84752, "count": 148, "sumsqr": 8853099192 }, "contact_list:query:apdex:frustrated": { "sum": 123075, "min": 38323, "max": 84752, "count": 2, "sumsqr": 8651553833 }, "contact_list:query:apdex:satisfied": { "sum": 94346, "min": 159, "max": 2393, "count": 144, "sumsqr": 119663414 }, "contact_list:query:apdex:tolerable": { "sum": 12541, "min": 4997, "max": 7544, "count": 2, "sumsqr": 81881945 } ```
Togo - Phone Tecno Pop 7 - contact_list:load = 0.59, contact_list:query = 0.90 ```json "contact_list:load": { "sum": 122034, "min": 964, "max": 19578, "count": 29, "sumsqr": 808812564 }, "contact_list:load:apdex:frustrated": { "sum": 19578, "min": 19578, "max": 19578, "count": 1, "sumsqr": 383298084 }, "contact_list:load:apdex:satisfied": { "sum": 9144, "min": 964, "max": 2071, "count": 6, "sumsqr": 15516228 }, "contact_list:load:apdex:tolerable": { "sum": 93312, "min": 3212, "max": 6279, "count": 22, "sumsqr": 409998252 }, "contact_list:query": { "sum": 321022, "min": 178, "max": 78021, "count": 145, "sumsqr": 7601213600 }, "contact_list:query:apdex:frustrated": { "sum": 121479, "min": 19243, "max": 78021, "count": 3, "sumsqr": 7043935715 }, "contact_list:query:apdex:satisfied": { "sum": 110372, "min": 178, "max": 2753, "count": 120, "sumsqr": 176227142 }, "contact_list:query:apdex:tolerable": { "sum": 89171, "min": 3046, "max": 6663, "count": 22, "sumsqr": 381050743 }, ```
Screenshot 2024-04-29 at 16 38 45
michaelkohn commented 2 months ago

Nice @Benmuiruri. Thanks for the clear data... it's really interesting to see how much worse Togo is, I wonder why??

I assume the Mali and Nepal users you set up are only seeing 2-3 rows in the contact_list (due to where they are in the hierarchy), so it's not surprising they are scoring very high. Since we have a good dataset of, well, bad data with Kenya and Togo, I'm fine if you just proceed comparing the results of before / after code changes for Kenya and Togo only since there really isn't any room to improve Mali/Nepal anyway. The one thing that could be useful, but not totally necessary, would be to show that Mali/Nepal didn't get slower after the code changes, but it's not necessary since I imagine it's a very low chance it gets slower.