search

rubycdp / ferrum

Headless Chrome Ruby API
https://ferrum.rubycdp.com
MIT License
1.74k stars 123 forks source link

Handle pending connections losing context on frame navigation #426

Open francisbeaudoin opened 9 months ago

francisbeaudoin commented 9 months ago

Details

Similar to what is being proposed in #420, this PR fixes intermittent Ferrum::TimeoutError as the driver awaits for requests to be completed.

What

It was observed that in some scenarios the driver is not always receiving and associated Network.loadingFailed for requests that were in-flight as the main frame navigates. This may be happening with either main frame requests or sub-frame ones.

I'm not sure if this is the right approach, but it appears to be working. Basically, it flags requests which are no longer relevant to the current page context as unknown which are then assumed to be #finished.

Relates to #420

route commented 9 months ago

@francisbeaudoin do you have an example of such a page in the public internet? I just want to check the CDP logs to better understand of what's going on. A test describing the issue would be more appropriate though.

francisbeaudoin commented 8 months ago

@route Thanks for following up.

I spent some time reproducing the issue by using a use case that was causing timeouts at the time, and found something interesting.

By using the headless browser instance opened by ferrum vs using a regular instance ran from the desktop, the network requests were not returning the same status. i.e. (canceled) vs (unknown). As far as I remember, the (unknown) ones were causing the library to raise a timeout as it couldn't infer if a request had completed.

I've identified the different behavior to the disable-site-isolation-trials option. If that option is removed thus enabled, Chrome shows a (unknown) network request as opposed to (canceled) status. In the current library state, the option is disabled (added 4 months ago) so it may not longer be an issue for this specific use case, but I suspect there are other scenarios where requests may end up in an unknown state thereby timing out.

This was reproduced by visiting this example, and toggling the disable-site-isolation-trials option.

With disable-site-isolation-trials disable-site-isolation-trials

Without disable-site-isolation-trials with-site-isolation-trials

route commented 8 months ago

Hmmmm, nice thank you! Let me experiment a bit with that

francisbeaudoin commented 6 months ago

👋 @route By any chance you had time to look into this? I'd love to stop using a forked version in my project.

route commented 6 months ago

Sorry not yet, was busy doing other things. Thanks for the reminder I will

francisbeaudoin commented 1 month ago

@route Sorry for bothering. Any plans to move forward with this PR?