Flaky tests

[rkeeves]

Hi!

First of all: Thanks for the repos!

I want to bring up the problem of dynamic behavior (ajax) which causes tests to be flaky. I mean in general, not your tests in this repo.

I'd be really interested in your solution, or thoughts on how to make tests reliable (again: just in general, not any specific test). In this repo I tried to describe problematic behaviors, and some pretty barbaric and clunky ways to interact with client side js to make the tests somewhat reliable by interacting with client side js.

The site I use for the tests is a showcase site, so it is pretty reliable compared to the projects which use its technology out in the wild, but at least it should give you a rough idea what the problem is all about. (I know that the jsf-based technology is pretty outdated, but there are still a lot of slowly dying projects still kicking around which use it.)

Another thing is the whole page object pattern... it feels to me like it does some things right, but it also kind of limiting. I don't know why, but I feel that there must be some other way...

I know that contacting you via a github issue is pretty awkward, so please feel free to delete/ignore this, and accept my apologies if you'd rather not talk.

But, even if we never speak: Thanks for the repos again!

Bye

[PietroSassone]

Hello!

Thanks for the question! It is interesting.

I have faced some flakiness in the past. But not with such problematic Ajax elements that you have described.

So I'll take a look again at your repo later to see what solution you found.

What I have used for flaky tests before:

• Configuring the Maven Failsafe plugin with Junit to rerun failed tests once. So maybe next time it succeeds.
• Implementing a method that would reload the page once if even after waiting, there's an issue with loading some element.
• Using fluent wait for a condition with awaitility. It's a great tool for that in my opinion. Example: https://github.com/PietroSassone/selenium-ta-demo/blob/main/src/test/java/com/example/selenium/ta/demo/steps/WebTablesStepDefinitions.java#L198
• I have learned a trick from others to cache WebElements that were once located by Selenium, like this: https://github.com/PietroSassone/selenium-ta-demo/blob/main/src/test/java/com/example/selenium/ta/demo/util/DemoElementLocator.java We used this one in special tests when we shut down the webdriver in the middle of a test. Then start a new with different settings one and continue the scenario without re-initializing the page objects. I'm not sure that this would help against detached elements as I have not tried it in such a situation. Maybe.

One more idea:

• Other than that I could think about some solution switching back and forth between frames. Like in case of captchas. Example: https://stackoverflow.com/questions/66226928/message-no-such-element-unable-to-locate-element-methodcss-selector-se Again, I'm not sure if that would work. Just an idea.

About the page object pattern: I have used it with reflection before to make it more flexible. I don't have a public example for that. Some say it's a bit unorthodox, but I don't think so, What I mean for example: in a big framework with loads of scenarios and test cases. We can easily implement a parent page object. Then have all page objects extend it. Then the parent could have a method which returns the method from any page object that matches the given name. Using Java reflection. Like: find & execute the method named "clickOn" + yourElementName. This is nicely pairable with a cucumber stepdef where yourElementName can be supplied from any scenario. This way if you have a lot of pageobjects, you can avoid explicitly calling all the pageA.cklicOkX() one by one, etc.

[rkeeves]

Hi!

Thanks for the answers!

awaitility

Simply: Yes!

ElementLocator

Yes again!

method that would reload the page

Yes. I'm thinking about a solution, where when the user writes this

$("#foo").should(idempotent).should(idempotent).execute(nonidempotent);

We inject dom/ajax/effects/detached/stale safeguards implicitly, or even delay the execution.

The other trouble (aside from how to inject the safeguards while still using Selenide in a civilized manner) is that the primefaces components are quite complex and I want to reduce code noise in the tests. Each component is so complex that they are kind of like mini pageobjects. For example, if you want to select a value you basically need three clicks on three different elements from two entirely different divs. And don't let me get started on div based filterable tables which have rows which can themselves be filterable paginated tables, while event-wise it is an ajax minefield.

So I guess I'll have to use Selenide's Command interface, and Conditions to make the testers' life more bearable. I mean it is kind of clunky to use $("#foo").shouldBe(Foo.customCondition()).execute(Foo.setFoo("bar")), but it works, (Foo is simply a utility class like Lists, Foo::setFoo is simply a factory method which supllies a Command instance, Foo::customCondition is factory method which supplies "some package private type which extends Condition" instance.) and it's kind of plug-and-play aka it's not like a framework with a lot of wiring and dependencies.

Retry

YES! This would be the sane thing.

But, this is not possible. (The application under test is a really old dinosaur, no db/soap/rest setup is possible. Tests have to setup themselves via the ui. It's horrible, ugly, angry and unmaint...let's just say it's a bit clunky to work with :)

frame switch

Not in this current problem, but still helpful! Thanks!

page object type resolution from grammar via reflection

Hm. I have to think about this... :)

Once again, thanks for the many helpful tips!

[rkeeves]

Hi again!

I just wanted to thank you for pointing me in the right directions with all your examples and tips! I'm currently setting up my local test environment to reproduce the flakyness in a controlled manner. Issues were nicely reproducible up to this point, I hope it stays this way :) And with springboot, hehe :)

Also: Turns out the guard's hook up is much more painless than I thought.

Bye rkeeves

[PietroSassone]

Hello and sorry for the late answer!

Thank you, I'm glad I could be at least of some assistance! I hope you manage to implement the solution you're looking for. Sounds interesting.