search

bosondata / chrome-prerender

Render JavaScript-rendered page as HTML/PDF/mhtml/png/jpeg using Headless Chrome
MIT License
165 stars 20 forks source link

Use layerPainted event to update _last_active_time may lead to timeout. #13

Closed myrfy001 closed 6 years ago

myrfy001 commented 7 years ago

Currently, LayerTree.layerPainted event is used to update the _last_active_time variable.

But, as a common case, a blinking cursor in a focused textbox can lead to a continuous refreshing.

Thus, _wait_responses_ready will not work because the one second threshold won't be reached, which means a page without setting window.prerenderReady will wait until timeout.

messense commented 7 years ago

Thank you for reporting this, can you give an example page so I can test it?

knuhol commented 7 years ago

Is it the same? Let's try https://google.com:

2017-06-12 14:41:28,750 INFO prerender.chromerdp.navigate:205   Page 08a8d058-66e4-4bd6-ae57-6a5942381f6f [1] navigating to https://google.com
2017-06-12 14:41:58,765 INFO websockets.protocol.fail_connection:618   Failing the WebSocket connection: 1006 
2017-06-12 14:41:58,767 WARNING prerender.app._render:95    Temporary browser failure: , retry rendering https://google.com in 1s
2017-06-12 14:41:58,786 ERROR asyncio.serve:450   Task exception was never retrieved
future: <Task finished coro=<Page._evaluate_prerender_ready() done, defined at /Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py:219> exception=AttributeError("'NoneType' object has no attribute 'send'",)>
Traceback (most recent call last):
  File "/Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py", line 221, in _evaluate_prerender_ready
    res = await self.evaluate('window.prerenderReady == true')
  File "/Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py", line 215, in evaluate
    'params': {'expression': expr}
  File "/Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py", line 171, in send
    await self.websocket.send(json.dumps(payload))
AttributeError: 'NoneType' object has no attribute 'send'
2017-06-12 14:41:59,773 INFO prerender.chromerdp.navigate:205   Page 5702afd5-b794-4c0c-85f3-5817f2bcc59f [1] navigating to https://google.com
2017-06-12 14:42:28 - (sanic)[ERROR]: NoneType: None

2017-06-12 14:42:28,742 ERROR sanic.log:104   NoneType: None

2017-06-12 14:42:28,751 INFO websockets.protocol.fail_connection:618   Failing the WebSocket connection: 1006 
2017-06-12 14:42:28,751 WARNING prerender.app.handle_request:172   Got 504 for https://google.com in 60006ms
2017-06-12 14:42:28 - (sanic)[ERROR]: Connection lost before response written @ ('127.0.0.1', 62967)
2017-06-12 14:42:28,752 ERROR sanic.write_response:217   Connection lost before response written @ ('127.0.0.1', 62967)
myrfy001 commented 7 years ago

Sorry for a late reply. I missed the mail sent to me 11 days ago. Yes, the search engine is a good example.

messense commented 7 years ago

For https://google.com I can't reproduce the exception locally. What's your OS and Chrome version? @akarienta

knuhol commented 7 years ago

OS: macOS Sierra, version 10.12.5 (16F73) Chrome: Version 59.0.3071.86 (Official) (64bit)

messense commented 7 years ago

Oh, I haven't try Chrome stable yet. I am using Chrome Canary for development. Chrome stable might miss some features that unstable/canary version has.

knuhol commented 7 years ago

@messense ok makes sense, I will try Canary and let you know, thanks

myrfy001 commented 7 years ago

You can also try www.baidu.com， the biggest search engine in China. I ran on Sierra too.

messense commented 7 years ago

@myrfy001 I have no problem rendering www.baidu.com too.

2017-06-12 21:35:00,284 INFO prerender.chromerdp.navigate:205   Page d1945e18-ff0b-4111-95cf-96611fa58975 [1] navigating to https://www.baidu.com
2017-06-12 21:35:01,017 WARNING prerender.chromerdp._on_log_entry_added:316   https://www.baidu.com/:705 console javascript log warning: A Parser-blocking, cross site (i.e. different eTLD+1) script, https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/protocol/https/global/js/all_async_search_0596852.js, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message.See https://www.chromestatus.com/feature/5718547946799104 for more details.
2017-06-12 21:35:01,018 WARNING prerender.chromerdp._on_log_entry_added:316   https://www.baidu.com/:705 console javascript log warning: A Parser-blocking, cross site (i.e. different eTLD+1) script, https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/protocol/https/global/js/all_async_search_0596852.js, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message.See https://www.chromestatus.com/feature/5718547946799104 for more details.
2017-06-12 21:35:02,634 INFO prerender.app.handle_request:161   Got 200 for https://www.baidu.com in 2355ms
knuhol commented 7 years ago

@messense It's absolutely the same for me with Version 61.0.3128.0 (Official) canary (64bit)

knuhol commented 7 years ago

@messense I have also the same problem to render www.baidu.com:

2017-06-12 15:38:33,258 INFO prerender.chromerdp.navigate:205   Page 4f5a3630-5014-4d85-b5ef-249096334e24 [1] navigating to https://www.baidu.com
2017-06-12 15:38:33,932 WARNING prerender.chromerdp._on_log_entry_added:316   https://www.baidu.com/:705 console javascript log warning: A Parser-blocking, cross site (i.e. different eTLD+1) script, https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/protocol/https/global/js/all_async_search_0596852.js, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message.See https://www.chromestatus.com/feature/5718547946799104 for more details.
2017-06-12 15:38:33,933 WARNING prerender.chromerdp._on_log_entry_added:316   https://www.baidu.com/:705 console javascript log warning: A Parser-blocking, cross site (i.e. different eTLD+1) script, https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/static/protocol/https/global/js/all_async_search_0596852.js, is invoked via document.write. The network request for this script MAY be blocked by the browser in this or a future page load due to poor network connectivity. If blocked in this page load, it will be confirmed in a subsequent console message.See https://www.chromestatus.com/feature/5718547946799104 for more details.
2017-06-12 15:39:02 - (sanic)[ERROR]: NoneType: None

2017-06-12 15:39:02,242 ERROR sanic.log:104   NoneType: None

2017-06-12 15:39:02,250 INFO websockets.protocol.fail_connection:618   Failing the WebSocket connection: 1006 
2017-06-12 15:39:02,251 WARNING prerender.app.handle_request:172   Got 504 for https://www.baidu.com in 60011ms
2017-06-12 15:39:02 - (sanic)[ERROR]: Connection lost before response written @ ('127.0.0.1', 64642)
2017-06-12 15:39:02,252 ERROR sanic.write_response:217   Connection lost before response written @ ('127.0.0.1', 64642)
2017-06-12 15:39:02,398 ERROR asyncio.serve:450   Task exception was never retrieved
future: <Task finished coro=<Page._evaluate_prerender_ready() done, defined at /Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py:219> exception=AttributeError("'NoneType' object has no attribute 'send'",)>
Traceback (most recent call last):
  File "/Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py", line 221, in _evaluate_prerender_ready
    res = await self.evaluate('window.prerenderReady == true')
  File "/Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py", line 215, in evaluate
    'params': {'expression': expr}
  File "/Users/idolezalova/.pyenv/versions/3.6.1/lib/python3.6/site-packages/prerender/chromerdp.py", line 171, in send
    await self.websocket.send(json.dumps(payload))
AttributeError: 'NoneType' object has no attribute 'send'

@myrfy001 Do you have the same error?

messense commented 7 years ago

@akarienta How did you launch Chrome Canary? I am using:

/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary --remote-debugging-port=9222

Make sure you didn't already opened a Chrome Canary before running the command in terminal.

messense commented 7 years ago

To debug it, start it with DEBUG=1 prerender

knuhol commented 7 years ago

@messense I go with command from your README:

/Applications/Google Chrome Canary.app/Contents/MacOS/Google\ Chrome\ Canary --headless --remote-debugging-port=9222 --disable-gpu "about:blank"
knuhol commented 7 years ago

@messense Ahahah, it's ok with your command: 

/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary --remote-debugging-port=9222

So maybe just let's update the README file 😄

BTW it works in the last stable Chrome as well!

messense commented 7 years ago

Good to know. The readme command is for Linux server.

knuhol commented 7 years ago

@messense Ah I see. Very well then, thank you very much for your help!

messense commented 7 years ago

BTW, I would expect Chrome Canary to work with --headless, I guess It's broken for now. But it's unstable version anyway, breakages are kind of normal.

knuhol commented 7 years ago

I have the same problem again (with both google.com and baidu.com), even when using the command 

/Applications/Google\ Chrome\ Canary.app/Contents/MacOS/Google\ Chrome\ Canary --remote-debugging-port=9222

I tried also the --headless flag. It seems like a problem with Chrome, it works on some runs, on others it doesn't.

messense commented 6 years ago

Removed layerPainted events in https://github.com/bosondata/chrome-prerender/commit/280ca7939a2ecc3ded207cfe82cbd503612c1ed4