Dev Test should include a Chromebook performance test

[pixelzoom]

During RC1 testing, it was discovered that My Solar System performance is unacceptable on Chromebook. See https://github.com/phetsims/my-solar-system/issues/326. It's unfortunate that this was not discovered during dev testing, because addressing this in the RC phase may be expensive. But apparently Chromebook is typically not tested during dev testing. According to @KatieWoe:

It is generally considered optional for dev testing in general. We can move it to required for this test if you would like

Full thread from Slack#qa 3/22/23

@pixelzoom Good morning QA. Is Chromebook testing included for https://github.com/phetsims/qa/issues/921? Or is that something that we need to explicitly request? In general, it seems like some level of Chromebook testing should be done for dev tests, to identify performance issues. But ChromeOS is in the checklist for “Light testing, or optionally skip if time crunch”. For Calculus Grapher specifically, performance on Chromebooks is a concern. @KatieWoe It is generally considered optional for dev testing in general. We can move it to required for this test if you would like @pixelzoom Yes please, move it to required for Calculus Grapher. But in general… It seems like identifying performance issues is crucial during dev testing, not waiting until RC testing — because addressing performance issues often results in big code changes. So given that Chromebooks are the most performance-challenged platforms, why do we wait until RC testing to do a required test? @KatieWoe If there are major problems they often also show up on iPad, which is covered. We try to get to Chromebooks for performance testing, but since Chrome is tested on the bigger platforms it was deemed less necessary than the iPad testing.

While it may not be necessary to do a full Chromebook dev test, doing a performance test seems important. Should we modify the QA process to always include a Chromebook performance test?

[KatieWoe]

Commit above adjusts the platforms required in a dev test to include ChromeOS and remove Win 10 Chrome to optional since Win 11 has become more common.. @pixelzoom if this is sufficient feel free to close.

[pixelzoom]

I said:

While it may not be necessary to do a full Chromebook dev test, doing a performance test seems important.

If you want to add ChromeOS to the required category, that's fine. But what I sugggested was a quick performance test, just to vertify that performance is not a problem on the most challenged machines.

[KatieWoe]

While I was looking at it I realized that testing Win 10 and Win 11 Chrome was rather superfluous at this point and extra problems were more likely to show up on ChromeOS comparatively, so I think it is a good change.

[pixelzoom]

Sounds good @KatieWoe. Closing.