search

phetsims / aqua

Automatic QUality Assurance
MIT License
2 stars 4 forks source link

snapshot comparison fails when switching tabs while testing #131

Open zepumph opened 3 years ago

zepumph commented 3 years ago

This is mostly just a question, and for @jonathanolson. I'm curious about a behavior and would like to know if it is expected or potentially a bug to investigate.

I have noticed that in a couple of sims, friction included, if you switch tabs while running the snapshot comparison on a currently running snapshot, it will fail the test. Here is the error:

image

It is always quite minimal, but I think it has something to do with how javascript (and therefore clock ticks) do not occur on windows/tabs that are not "active" (at the forefront of the screen, on Windows at least).

@jonathanolson does this feel like something to investigate. Does it seem like something that could be mitigated. The atoms in the above picture are using a CanvasNode, does that change anything? Feel free to close BTW.

zepumph commented 3 years ago

For clarity: when I run off the same working copy, but don't switch tabs, the sha is the same and I do pass the snapshot comparison.

jonathanolson commented 3 years ago

Not sure exactly how to mitigate, I usually open a new window for it and leave it open (in an active tab), since we don't get the animation frames without it. We SHOULD get the same result however, so this seems like something worthy of investigation.

zepumph commented 3 years ago

Thanks! I'll note here when I run into it in other cases (or other sims). Perhaps I'll even promote to aqua if it gets general enough.