Unrecoverable error from Selenium: timeout: Timed out receiving message from renderer: 300,000

[mcepl]

When I run fanfictionnet_ff_proxy on updating multiple (and I mean many, tens at least) stories, I always end up in:

Unrecoverable error from Selenium: timeout: Timed out receiving message from renderer: 300,000

See for this or this log.

Also, closer to the top I see this apparently recoverable backtrace:

2021-06-27 14:06:43.318 CEST DEBUG selenium.webdriver.remote.remote_connection POST http://127.0.0.1:35785/session/b1f207ac06374e3ce2c007b61c094c88/goog/cdp/execute {"cmd": "Page.addScriptToEvaluateOnNewDocument", "params": {"source": "\n\n                                   Object.defineProperty(window, 'navigator', {\n                                       value: new Proxy(navigator, {\n                                       has: (target, key) => (key === 'webdriver' ? false : key in target),\n                                       get: (target, key) =>\n                                           key === 'webdriver'\n                                           ? undefined\n                                           : typeof target[key] === 'function'\n                                           ? target[key].bind(target)\n                                           : target[key]\n                                       })\n                                   });\n                                    \n                                                            \n                        "}}

I am not sure even where does the backtrace comes from.

[nsapa]

I can try increasing the timeout but it is the worst kind of heinsebug.

The second log is from Undetected Chromedriver. It override some part of the DOM so CloudFare doesn't trigger.

[nsapa]

I'm on holiday so I've refactored a little. The proxy should stay operational after a Selenium exception now.

Can you retry?

[mcepl]

Much better, run fanficfare -u on couple of hundred EPubs, and got only once stalled process. However, when it breaks now, it really breaks ... process was locked for couple of hours before I noticed. However, the total is MUCH better. Thank you.

[nsapa]

That's strange. You killed the browser and the proxy fixed itself?

Version 0.5.1 have a docker config if you want to try with another userland (Debian).

[mcepl]

No, I haven't killed the browser. Just that everything got frozen and I don't know who is the culprit, all three programs (fanficfare, your proxy and chromium) were just doing nothing for couple of hours without any timeout of either of them.

[nsapa]

2021-08-14 14:40:23.514 CEST INFO mainloop Current URL = https://www.fanfiction.net/s/3894793/1/, page title = Harry Potter and the Distaff Side Chapter 1, a harry potter fanfic | FanFiction, mimetype = text/html
2021-08-14 14:45:35.482 CEST ERROR mainloop TimeOut while getting https://www.fanfiction.net/s/11950816/1/, resetting renderer with an internal page
2021-08-14 14:55:35.494 CEST ERROR root Unrecoverable error from Selenium: timeout: Timed out receiving message from renderer: 300,000
  (Session info: chrome=90.0.4430.85). Killing this instance...

The proxy take ~15 minutes to recover from a Selenium failure. I am not able to reproduce your issue.

I am closing since the initial issue have been fixed in 0.5.