Flaky Percy tests due to keyboard focus

[andybalaam]

Test failure examples:

• Room Directory - filtered one result
• EventTile with reply chains on bubble layout
• EventTile with reply chains on IRC layout
• EventTile with reply chains on compact modern layout
• Hidden event line with zero padding on IRC layout

These all seem to fail because the (keyboard?) focus varies between test runs.

[luixxiul]

Hmm... is there not a way of debugging the focus issue locally? (though I guess there should not be, as we could just use it instead of their service)

[t3chguy]

(though I guess there should not be, as we could just use it instead of their service)

Sure, you can run the cypress tests locally, the same issue should be present there, though make sure to not have your cursor over the cypress window to avoid impacting the focus

[andybalaam]

Hmm... is there not a way of debugging the focus issue locally? (though I guess there should not be, as we could just use it instead of their service)

Yeah, we definitely want to fix these issues (and would appreciate help!) but at the moment I am reviewing and closing ~10-20 failures per day, meaning that I won't notice real errors, so disabling them is the less-bad option than just leaving them.

[luixxiul]

Yeah, we definitely want to fix these issues (and would appreciate help!) but at the moment I am reviewing and closing ~10-20 failures per day, meaning that I won't notice real errors, so disabling them is the less-bad option than just leaving them.

Ahh, my intention was not to ask you to debug the issues, but to ask how I can debug to stop creating flaky percy tests before creating PRs as most of them listed above were based on recent PRs I've pushed...

[luixxiul]

Sure, you can run the cypress tests locally, the same issue should be present there, though make sure to not have your cursor over the cypress window to avoid impacting the focus

Yes, it is surely possible to run cypress tests locally (I have tried to minimize risk of flaky tests by running them locally before creating a PR), but I could not manage to spot the flakiness :frowning: Maybe something happens in milliseconds.

[andybalaam]

Ahh, my intention was not to ask you to debug the issues, but to ask how I can debug to stop creating flaky percy tests before creating PRs as most of them listed above were based on recent PRs I've pushed...

Ah, that is great, thank you for trying! These failures seem really hard to debug, but we'll get there. Please don't give up creating the tests - they will be really helpful when we have made them work properly :-)

[andybalaam]

Sure, you can run the cypress tests locally, the same issue should be present there, though make sure to not have your cursor over the cypress window to avoid impacting the focus

Yes, it is surely possible to run cypress tests locally (I have tried to minimize risk of flaky tests by running them locally before creating a PR), but I could not manage to spot the flakiness frowning Maybe something happens in milliseconds.

I have tried similarly with the variable-height dropdown and I can't see anything locally :-(

[luixxiul]

I suspect that the flakiness of a reply / ReplyChain is caused by the fact that the color of vertical strokes changes randomly per each run (the test user is created randomly -> the color of the user name disambiguatedProfile changes randomly too -> the color of the strokes changes too as it depends on the user name color).

The same issue has been experienced for BaseAvatar too (please see: _BaseAvatar.pcss).

I am going to create a PR to (hopefully) address this.

ref: https://github.com/vector-im/element-web/issues/24952

[luixxiul]

Created the PR to apply the same color to the left borders of ReplyChain consistently: https://github.com/matrix-org/matrix-react-sdk/pull/10450

[andybalaam]

I haven't seen this for a while, so assuming it's fixed. Will re-open if not.

[andybalaam]

Nope, I forgot that I had disabled the related test.

[kerryarchibald]

https://percy.io/dfde73bd/matrix-react-sdk/builds/25491724/changed/1421036924?browser=edge&browser_ids=18%2C33%2C34%2C35&subcategories=approved&viewLayout=overlay&viewMode=new&width=1920&widths=400%2C800%2C1024%2C1920

[andybalaam]

Disabling:

• Selected EventTile of audio player (high contrast) on IRC layout
• Selected EventTile of audio player with a reply on IRC layout
• Selected EventTile of audio player (light theme, monospace font) on IRC layout

because they are flaky due to variable keyboard focus.

[andybalaam]

We are not using Percy any more. Closing this.