Faulty results when testing with 12.2+

[sburnicki]

When testing with an iPad running 12.2, the test results are faulty.

This is what a waterfall of testing bbc.com looks like:

Important requests (as the HTML itself) are missing, and there also seems to be a timing issue, since the bars start before 0.

I couldn't reproduce this with webpagetest.org agents, but they seem to run iOS < 12.2 anyway.

[pmeenan]

If you try the iPhone 6 on the public instance you may see similar issues (it is running 12.3). I think the devtools changes in 12.2 are more involved than the first round of fixes covered.

On Tue, May 28, 2019 at 6:57 AM Stefan Burnicki notifications@github.com wrote:

When testing with an iPad running 12.2, the test results are faulty.

This is what a waterfall of testing bbc.com looks like: [image: image] https://user-images.githubusercontent.com/5701519/58483607-e1ffec00-8160-11e9-8d34-6da5c210b7d1.png

Important requests (as the HTML itself) are missing, and there also seems to be a timing issue, since the bars start before 0.

I couldn't reproduce this with webpagetest.org agents, but they seem to run iOS < 12.2 anyway.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/WPO-Foundation/wptagent/issues/265?email_source=notifications&email_token=AADMOBMB7QNJGFT5EI4LNWDPXU23LA5CNFSM4HQDVHP2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4GWHRRGA, or mute the thread https://github.com/notifications/unsubscribe-auth/AADMOBJITL3FLI4IKYUZVWDPXU23LANCNFSM4HQDVHPQ .

[nilskuhn]

Hi Patrick, can you say when you will find the time or plan to take a deeper look into this? Or do you have an idea of the complexity of a possible fix? Thanks in advance. Regards, Nils

[pmeenan]

It's going to be at least 3 weeks to a month before I get a chance to look. I'm traveling for the next 2 weeks and then my schedule is pretty hectic when I get back (and WebPageTest is a "weekends and evenings" project).

If you want to take a crack at it, the relevant iOS agent code is here.

My guess is it is going to involve watching what Safari's web inspector does when connected to a device doing the same activity to see if there are changes to the events that need to be made.

[BennyLi]

Did some work on this today with an iPad Air (3rd generation) an Software Version 12.2. In the debugging output I found this part:

...
13:53:53.945 - Connect to dev tools Error: HTTPConnectionPool(host='localhost', port=9222): Max retries exceeded with url: /json (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x74984e30>: Failed to establish a new connection: [Errno 111] Connection refused',))
...
13:53:56.042 - Starting new HTTP connection (1): localhost:9222
13:53:56.047 - http://localhost:9222 "GET /json HTTP/1.1" 200 258
13:53:56.053 - Dev Tools tabs: [{"title":"","url":"http:\/\/www.webpagetest.org\/","webSocketDebuggerUrl":"ws:\/\/localhost:9222\/devtools\/page\/1","appId":"PID:854","faviconUrl":"","thumbnailUrl":"\/thumb\/http:\/\/www.webpagetest.org\/","devtoolsFrontendUrl":""}]
13:53:56.057 - Connect to dev tools websocket Error: [Errno 111] Connection refused
13:53:56.060 - DevTools websocket connected
13:53:56.061 - Sending: {"params":{"waitForDebuggerOnStart":true,"autoAttach":true},"id":1,"method":"Target.setAutoAttach"}
13:53:56.062 - Sending: {"params":{},"id":2,"method":"Target.getTargets"}
13:53:56.072 - {"method":"Target.targetCreated","params":{"targetInfo":{"targetId":"page-2","type":"page"}}}
13:53:56.116 - {"error":{"code":-32601,"message":"'Target.setAutoAttach' was not found","data":[{"code":-32601,"message":"'Target.setAutoAttach' was not found"}]},"id":1}
13:53:56.117 - {"error":{"code":-32601,"message":"'Target.getTargets' was not found","data":[{"code":-32601,"message":"'Target.getTargets' was not found"}]},"id":2}
13:53:56.120 - >>> 11:exec
13:53:56.139 - <<< 11:OK
13:53:56.140 - Running test
...

Others do have problems with this too. For example:

• https://github.com/RemoteDebug/remotedebug-ios-webkit-adapter/issues/138
  • https://github.com/google/ios-webkit-debug-proxy/issues/294
  • https://github.com/RemoteDebug/remotedebug-ios-webkit-adapter/issues/138
• https://github.com/appium/appium-remote-debugger/commit/4a03ddc1f68b6c32f24a204dad2388eecd884670

I'll try to get more into this, but help is really appreciated!

[pmeenan]

Current plan is to backfill the request data using resource timing (and to re-align the start time based on matching the resource timing to the dev tools request times). Still not as good as fixing the underlying issue but will hopefully buy time until (hopefully) there is a real fix.

[pmeenan]

Here is the inspector API: https://trac.webkit.org/browser/webkit/trunk/Source/JavaScriptCore/inspector/protocol/Target.json

Looks like it is possible to pause targets when they are created, allowing us to attach to them and start monitoring requests before resuming. Adds more round trips than I'd like but should be better than things are now.

[pmeenan]

I added support to the agent for the pauseOnStart targets but the code only landed in WebKit in December so it will not work until an updated Safari rolls out: https://trac.webkit.org/changeset/253097/webkit/trunk/Source/JavaScriptCore/inspector/protocol/Target.json