search

atsign-foundation / at_client_sdk

The Dart implementation of atSDK used for implementing Atsign's technology into other software
https://pub.dev/publishers/atsign.org/packages
BSD 3-Clause "New" or "Revised" License
1.46k stars 31 forks source link

end2end_tests in at_client_sdk are flaky #483

Closed gkc closed 2 years ago

gkc commented 2 years ago

Describe the bug Sometimes the end2end_tests fail. Sometimes they succeed.

To Reproduce They're flaky so it's not consistent. However - noticing that end2end_tests had failed for #482, I re-ran them and they failed again. I re-ran them again, and they worked.

Expected behavior The tests should either succeed or fail consistently.

gkc commented 2 years ago

We need to invest time here, find out why these tests are flaky and fix whatever the problem is. Every time there is a spurious test failure, it chews up some of somebody's time.

VJag commented 2 years ago

@purnimavenkatasubbu I see that notify_test.dart has "'Notify a key with value to sharedWith atSign and listen for notification from sharedWith atSign' test and it is flaky.

I see that test assumes a single entry that matches the regex ".notifyList(regex: 'phone_$lastNumber')", however, it may not be true, as is the case when the tests failed. Below JSON has four entries that match the above regex.

[{id: 314d29c3-b216-4657-85b7-a8d2ed1adf41, from: @ cicd1, to: @ cicd2, key: @ cicd2:phone_44.wavi@cicd1, value: null, operation: update, epochMillis: 1649715698135 {id: 459224c1-7393-424a-8bd5-f0a18f56b8d4, from: @ cicd1, to: @ cicd2, key: @ cicd2:phone_4.wavi@cicd1, value: npe+I7mKqcsFSvxjBLGCwA==, operation: update, epochMillis: 1649761941400 {id: c756d6c2-508e-4bf1-954b-2d5882533a57, from: @ cicd1, to: @ cicd2, key: @ cicd2:phone_43@cicd1, value: null, operation: delete, epochMillis: 1649682494813 {id: cb6c99bc-de65-47af-8ceb-c472fbacd6f4, from: @ cicd1, to: @ cicd2, key: @ cicd2:phone_41.wavi@cicd1, value: null, operation: update, epochMillis: 1649745615550 {id: da71f885-6231-4658-9439-509f65ae0f41, from: @ cicd1, to: @ cicd2, key: @ cicd2:phone_43.wavi@cicd1, value: npe+I7mKqcsFSvxjBLGzww==, operation: update, epochMillis: 1649707813961 {id: dd8eaf85-070a-41ae-9f20-a141a32449d7, from: @ cicd1, to: @ cicd2, key: @ cicd2:phone_40.wavi@cicd1, value: npe+I7mKqcsFSvxjBLGwww==, operation: update, epochMillis: 1649742368152]

We can probably fix the flakiness in many different ways. A couple of ways that come to my mind are:

  1. Use "UUID" in place of a random number so that the chance of a collision is rare
  2. Don't expect the "0th" entry in JSON to be the one you are looking for, rather iterate and look for the key we have created
purnimavenkatasubbu commented 2 years ago

Sure @VJag. I will re-write the existing and also the new tests (e2e_tests) in the PR

VJag commented 2 years ago

Yes. Idea is to fix other flaky tests of similar nature. Thank you.

purnimavenkatasubbu commented 2 years ago

Fixed the issue as part of the PR

gkc commented 2 years ago

Great - please merge and then close this ticket - thank you!

purnimavenkatasubbu commented 2 years ago

Merged the PR to the trunk. Closing the issue.PR