Safari, Interop 2025 & Updated score

@gsnedders: We had a lot of discussion. Had a plan on using WebKittest runner for iOS. Now looking at the possibilities of using safari instead. Brings up a different set of challenges. Possible to run some tests with safari today. Others with manual setup. Like inability to change the size of viewport. Most problems can be solved with manual setup. Not a long term solution but enough to see where things are today.
@jgraham It would be nice to have a better solution at the WPT level for the viewport problem
@gsnedders Could do things in an iframe. But is that better or worse?
@jgraham I suspect better
@jgraham Gecko has a special hack to make this work. Able to create a viewport of the right size
@gsnedders a few questions from people. If you scroll, do we disable dynamic viewport sizing. How does this affect keyboard overlays. As you start changing the size of the viewport things become more complicated
@jgraham: Agreed that we haven't really had a plan on how to test things that depend on viewport sizes. Short term, the question is getting anything for safari results. sounds like it is.
@gsnedders yes. There is a plan. How frequently and setup is TBD
@jgraham Won't run every X hours?
@gsnedders How often do we have to restore the device? More involved if using an actual device.
@past Since we are using simulators for Android, maybe lean towards to that for iOS.
@gsnedders Yes. But there may be a difference in the media pipeline.
@past Could we just add a separate label for tests that we think are good?
@gsnedders Question of how many tests you run and how much hardware you need. I think we are trying to figure out the fundamental approach we taking
@past With the labels, it will help us exclude the tests like media problem
@jgraham It is a conversation for Interop 2025. . Either specific focus areas or specific individual tests. We think the mobile scores are not representative of the actual mobile browser. Or they come with a lot of caveats. And there are some proposals that are that. Example of known differences: touch events. Works on iOS but not on desktop.
@past: Let me show the current mobile dashboard that was deployed yesterday. Enable via https://wpt.fyi/flags then navigate to https://wpt.fyi/interop-2024?mobile-view
@jgraham Would like to have all Browsers uploading first. The warning will not go away. We don't have a mobile Edge. Not sure if there's a plan for that. Maybe having separate mobile focus areas would make sense.
@past: It currently looks green. And if you even focus on the areas that only make sense for mobile, it would be better.
@jgraham: Maybe mobile input events and layout focus. Pick those that make sense. And go from there. But this conversation should go to the Interop group
@past How do we score our efforts for this year? We have a dashboard. Trying to gauge the iOS efforts.
@gsnedders Simulator should be doable by EOY but don't want to commit to that.
@jgraham Is safari going to be stable or pre-release. Firefox uses experimental. Chrome can choose stable or experimental, right?
@gsnedders For iOS, something we have to resolve internally.
@past In the meantime, we could introduce Chrome on iOS results. No idea how the score looks like. Something WebKit on iOS.
@jgraham Don't have a strong opinion. Don't think it makes loads of difference to the EOY investigation score. Apple has a plan. Not there yet. Maybe EOY that it is achievable with safari. Investigating the differences and catalog the differences. Have not reduced the differences between mobile and desktop to zero. Would say we have done, 2&4. Half of one and maybe 3. 62.5%. 75% if iOS WebKit results came onboard.
@past We don't have to decide right now, but 75% could be reached if we use Chrome on iOS results. Not a long term solution but a good start to get us to 75%.

Action items:

[ ] Have the conversation on the approach in the Interop meeting. @jgraham @past @gsnedders

web-platform-tests / interop-mobile-testing

Agenda 2024-10-22 #31

Safari, Interop 2025 & Updated score