atsign-foundation / at_client_sdk

The Dart implementation of atSDK used for implementing Atsign's technology into other software
https://pub.dev/publishers/atsign.org/packages
BSD 3-Clause "New" or "Revised" License
1.47k stars 32 forks source link

build client performance test rigs #734

Open VJag opened 1 year ago

VJag commented 1 year ago

**Is your feature request related to a problem? Please describe. Build the tests that measure the performance bottlenecks

**Describe the solution you'd like Use design from #733

**Describe alternatives you've considered NA

**Additional context NA

purnimavenkatasubbu commented 1 year ago

See comment for more reference

  1. Sender, that sends a notification text (notify: text) continuously to a receiver without any expiry,
    sender_test

  2. The first receiver that only receives(without sync), and receiver_test

  3. a second receiver that listens to the notification from the sender and also updates a key for every 500ms i.e..,( with sync ) receiver2_test

Observations

The latency results were better for the receiver that only received notifications than the receiver that received notifications and updated a key simultaneously.

A graphical representation of the same can be found here.

https://docs.google.com/spreadsheets/d/1wY9eFI1OdqaexyR7Ho7ctT6IyNdsdIIB/edit#gid=63467969

Notification latency tests need to be re-tested once the following bug is fixed. https://github.com/atsign-foundation/at_client_sdk/issues/750

  1. Latency test with default values (3,3,5), without calling sync() manually and with a wait for 10 seconds.
  2. Latency test with default values(3,3,5) and by calling sync() in the test.
  3. Latency test with threshold values (1,3,3)
  4. Latency test with threshold values (2,3,3)
  5. Latency test with threshold values (1,3,2)
  6. Latency test with threshold values (1,2,1)
  7. Latency test with threshold values (1,2,2)

Test can be found in the following branch sync_latency_test

Observations The best latency time of around 1118ms on average, calculated for 50 iterations was for the following threshold values. syncRequestThreshold 1, syncRequestTriggerInSeconds 2 syncRunIntervalSeconds 1

Results for the above-mentioned latency scenarios can be found in the following sheet. https://docs.google.com/spreadsheets/d/13T4poZNMjIxSU2KM9EluaJ0B1TRiKXu-/edit#gid=1637466287

purnimavenkatasubbu commented 1 year ago

Re-tested with the skip_latestNotificationId. Now the latency results are better compared to the previous results. Results can be found in the following sheet.

Tab -1 Previous results(without the fix for skipping latestnotification Id) Tab 2 - After fix (Receiver that only listens to the notification) Tab -3 After fix (Receiver that listens to the notifications and updates the key )

https://docs.google.com/spreadsheets/d/1wY9eFI1OdqaexyR7Ho7ctT6IyNdsdIIB/edit#gid=1926778118

purnimavenkatasubbu commented 1 year ago

Latency tests have been re-verified again after the fix for "skipping latest notification Id ". The results have been better than the previous results that have been taken without the fix. Results can be found in the following sheet.

https://docs.google.com/spreadsheets/d/1wY9eFI1OdqaexyR7Ho7ctT6IyNdsdIIB/edit#gid=243961569

Tab 2 and 3

gkc commented 1 year ago

Moving to backlog as the core work in PR50 will be to continue to focus on sync tests, rearchitecting for secure element, future-proofing our encryption and of course bug fixing