Apdex testing scripts executed on MoH KE configuration

[michaelkohn]

MoH KE : https://care-teams-4x.dev.medicmobile.org/

Execute our testing scripts using this project's configuration

• Run on a physical device while offline
• All scripts should be executed 10 times per day across 5 days (manually advancing the days)
• Confirm data is syncing to PostgreSQL for analysis

[ralfudx]

Test execution for MoH KE is now complete 🟢 Test Sequence: Run test suite (6x10 tests) - 21/03 Sync data - 22/03 Run test suite (6x10 tests) - 22/03 Sync data - 23/03 Run test suite (6x10 tests) - 23/03 Sync data - 24/03 Run test suite (6x10 tests) - 24/03 Sync data - 25/03 Run test suite (6x10 tests) - 25/03 Sync data - 26/03

[ralfudx]

It appears the tests where not executed for this sequence

Run test suite (6x10 tests) - 25/03
Sync data - 26/03

I'd rerun this using a backdated date and see if it works cc: @michaelkohn

[michaelkohn]

OK. I'm moving the status back since this one is not done yet.

[michaelkohn]

@ralfudx I notice that the runs are saying (6x10 tests). I know we are running these 10x for each user, but I thought there were actually 7 tests, as described here. Is this because certain tests are purposely skipped? Or some failed?

[ralfudx]

@michaelkohn Yeah this is because we initially skipped the flaky test should submit a report for a newly created person However, with this GH-ticket, this test has now been fixed and will be included in subsequent test executions

[ralfudx]

@michaelkohn further modifications will be needed for this before we can execute the tests - this takes us to Monday for the test run

[michaelkohn]

Thanks @ralfudx ... can you provide a brief explanation of the modifications? Anything you need from someone else or anything we can do to make sure it isn't further blocked on Monday?

[ralfudx]

Sure @michaelkohn these are the things we always need to do for each new config:

• Run entire test suite (7 tests) on the New config
• Identify all changing workflows and selectors
• Find the new selectors on the config's changing screen and/or create logic(method) for the new workflow -> (this is the hard part)
• Create new method for this test should submit a report for a newly created person as it's always different for each config -> (while this is the hardest part)
• Rerun the tests and ensure none is flaky
• Set-up test suite to run all 7 tests 10 times -> (this is basically a copy and paste thingy)

For this ticket (MoH-KE config) apart from changing some of the selectors used for some of the form fields, we also need to modify the tests to use the same screen scroll logic implemented for the NSSD config as that is more stable old get scrollView () { return $('android=new UiScrollable(new UiSelector().scrollable(true)).scrollToEnd(5, 10)'); }

new

get scrollView () {
        return $('android=new UiScrollable(new UiSelector().scrollable(true)).scrollToEnd(1)');
    }
and
async scrollUntilTextVisible(text) {
        return $(`android=new UiScrollable(new UiSelector().scrollable(true)).scrollTextIntoView("${text}")`);
    }

[michaelkohn]

Moving to next week's sprint. If you don't think this can be finished during the week of 1-April, please let me know first thing Monday.

[ralfudx]

Test execution for MoH-KE Config is now complete 🟢 @michaelkohn @latin-panda kindly have a look at the data when it completely syncs User - gladys Device - Samsung SM-G955U, Android 9.0, API level 28

Test Sequence:

• Run test suite (7x10 tests) - 04/10 68 passing, 2 Failing Sync data - 04/11 All reports synced

• Run test suite (7x10 tests) - 04/11 70 passing Sync data - 04/12 All reports synced

• Run test suite (7x10 tests) - 04/08 70 passing Sync data - 04/09 All reports synced

• Run test suite (7x10 tests) - 04/09 70 passing Sync data - 04/10 All reports synced

• Run test suite (7x8 tests) - 04/08 56 passing -> Sync data - 04/08 All reports synced

For these tests we filled the Defaulter Follow Up form to create a task

[latin-panda]

@michaelkohn @ralfudx I don't see this data coming into postgres, I open a questions here for Yuvraj

[latin-panda]

Yuvraj fixed the DB and now we have data in postgres. These are the results:

@michaelkohn @ralfudx, please check this information. Just to compare, this is the one from Nepal that seems to produce better data.

The script was adapted based on my understanding of the test cases planned here + saving a task form.

Dates: 8th April to 11th April (taken from this comment) Total days: 4

The query has all the keys of the telemetry we expect from the automation tests.

SQL for Verification

```sql WITH constants (days, start_date, end_date) AS (VALUES (4, '2024-04-08', '2024-04-11')), apdex_metrics AS( SELECT substring(metric FROM '^(.*):apdex:') AS metric, SUM(COUNT) AS COUNT FROM useview_telemetry_metrics, constants WHERE metric LIKE '%:apdex:%' AND period_start BETWEEN constants.start_date::DATE AND constants.end_date::DATE GROUP BY metric ), apdex_result as ( SELECT metric, CASE WHEN apdex_metrics.metric = 'contact_list:load' THEN 30 * constants.days WHEN apdex_metrics.metric = 'contact_list:query' THEN 40 * constants.days WHEN apdex_metrics.metric = 'contact_detail:d_community_health_volunteer_area:load' THEN 10 * constants.days WHEN apdex_metrics.metric = 'contact_detail:e_household:load' THEN 20 * constants.days WHEN apdex_metrics.metric = 'contact_detail:f_client:load' THEN 20 * constants.days WHEN apdex_metrics.metric = 'enketo:contacts:form:contact:f_client:create:add:render' THEN 10 * constants.days WHEN apdex_metrics.metric = 'enketo:contacts:form:contact:f_client:create:add:save' THEN 10 * constants.days WHEN apdex_metrics.metric = 'enketo:contacts:defaulter_follow_up:add:render' THEN 10 * constants.days WHEN apdex_metrics.metric = 'enketo:contacts:defaulter_follow_up:add:save' THEN 10 * constants.days WHEN apdex_metrics.metric = 'report_list:load' THEN 10 * constants.days WHEN apdex_metrics.metric = 'report_list:query' THEN 10 * constants.days WHEN apdex_metrics.metric = 'report_detail:defaulter_follow_up:load' THEN 10 * constants.days WHEN apdex_metrics.metric = 'tasks:load' THEN 20 * constants.days WHEN apdex_metrics.metric = 'tasks:refresh' THEN 10 * constants.days WHEN apdex_metrics.metric = 'enketo:tasks:commodity_count:add:render' THEN 10 * constants.days WHEN apdex_metrics.metric = 'enketo:tasks:patient_details_reminder:add:render' THEN 10 * constants.days WHEN apdex_metrics.metric = 'enketo:tasks:patient_details_reminder:add:save' THEN 10 * constants.days WHEN apdex_metrics.metric = 'message_list:load' THEN 10 * constants.days WHEN apdex_metrics.metric = 'messages_detail:load' THEN 10 * constants.days WHEN apdex_metrics.metric = 'analytics:targets:load' THEN 10 * constants.days WHEN apdex_metrics.metric = 'boot_time' THEN 10 * constants.days END AS expected_count, SUM(apdex_metrics.count) AS actual_count FROM apdex_metrics, constants GROUP BY apdex_metrics.metric, constants.days ) SELECT apdex_result.metric, apdex_result.expected_count, apdex_result.actual_count, CASE WHEN apdex_result.actual_count >= apdex_result.expected_count THEN 'TRUE' ELSE 'FALSE' END AS meet_expectation FROM apdex_result ORDER BY meet_expectation DESC, apdex_result.metric ASC ```

This is Apdex from the results

SQL for Apdex

```sql WITH apdex_telemetry_data AS ( SELECT substring(metric from '^(.*):apdex:') AS event_category, CASE WHEN metric LIKE '%:satisfied' THEN 'satisfied' WHEN metric LIKE '%:tolerable' THEN 'tolerable' WHEN metric LIKE '%:frustrated' THEN 'frustrated' END AS event_type, SUM(count) AS event_count FROM useview_telemetry_metrics WHERE metric LIKE '%:apdex:%' AND period_start BETWEEN '2024-04-08' AND '2024-04-11' GROUP BY event_category, event_type ), apdex_scores AS ( SELECT event_category, SUM(CASE WHEN event_type = 'satisfied' THEN event_count ELSE 0 END) AS satisfied_count, SUM(CASE WHEN event_type = 'tolerable' THEN event_count ELSE 0 END) AS tolerable_count, SUM(CASE WHEN event_type = 'frustrated' THEN event_count ELSE 0 END) AS frustrated_count, SUM(event_count) AS total_event_count FROM apdex_telemetry_data GROUP BY event_category ) SELECT event_category, satisfied_count, tolerable_count, frustrated_count, ROUND(((satisfied_count + (tolerable_count / 2.0)) / total_event_count)::numeric, 2) AS apdex_score FROM apdex_scores ORDER BY apdex_score asc; ```

[michaelkohn]

The tests have been executed but not 100% validated. We are planning on upgrading the testing framework and will need to run everything again (validation scripts and evaluation scripts). As a result, I'm closing this issue.

Evaluation Output