catchpoint / WebPageTest.agent

Cross-platform WebPageTest agent
Other
213 stars 138 forks source link

Use the streaming netlog as the primary source for response bodies #557

Closed pmeenan closed 2 years ago

pmeenan commented 2 years ago

This turns on response body byte capturing when the streaming netlogs are active and uses that as the primary source for response bodies (falling back to the legacy dev tools method if a body isn't available from the netlog). It also reduces the overhead since it is using memory-mapped pipes for the file I/O from netlog (on a background thread) instead of bouncing through the dev tools protocol.

This applies both to the response bodies captured all of the time for optimization checks as well as the response bodies that are stored when explicitly requested.

We were seeing some edge cases where service worker scripts could not retrieve the response bodies through dev tools and this completely resolves that issue.

vibaldem commented 2 years ago

Makes sense that getting events directly from netlog instead of getting it passed via dev tools is more reliable. We've seen better variance with this approach than the old way of getting it through dev tools, and this seems like it should also help with that. I didn't find a way to use memory-mapped pipes in windows though, there doesn't seem to be an equivalent, so we just process it after the test.