search

bosondata / chrome-prerender

Render JavaScript-rendered page as HTML/PDF/mhtml/png/jpeg using Headless Chrome
MIT License
165 stars 20 forks source link

https problem on FreeBSD #32

Open beyondcreed opened 7 years ago

beyondcreed commented 7 years ago

I am running into issue when trying to render https based URL's. http works fine. Also chrome in headless can deal with https and spits out DOM, for example when I run: chrome --headless --disable-gpu --dump-dom https://www.apple.com

My prerender curl request is as follows:

curl -v http://myserver.com:8000/https://www.apple.com

Below is output from prerender:

[root@node295 ~]# prerender 2017-08-16 18:37:40 - (sanic)[DEBUG]: \u2584\u2584\u2584\u2584\u2584 \u2580\u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2584\u2584\u2584 _ \u2584\u2584\u2584\u2584\u2584 \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2584 / \ \u2580\u2580\u2580\u2580\u2588\u2588\u2588\u2588\u2588\u258c \u2580\u2590\u2584 \u2580\u2590\u2588 | Gotta go fast! | \u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2584\u2584 \u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2584\u2588\u2588 | ___/ \u2580\u2584\u2584\u2584\u2584\u2584 \u2580\u2580\u2588\u2584\u2580\u2588\u2550\u2550\u2550\u2550\u2588\u2580 |/ \u2580\u2580\u2580\u2584 \u2580\u2580\u2588\u2588\u2588 \u2580 \u2584\u2584 \u2584\u2588\u2588\u2588\u2580\u2580\u2588\u2588\u2584\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2584 \u2584\u2580\u2580\u2580\u2580\u2580\u2580\u2588\u258c \u2588\u2588\u2580\u2584\u2584\u2584\u2588\u2588\u2580\u2584\u2588\u2588\u2588\u2580 \u2580\u2580\u2588\u2588\u2588\u2588 \u2584\u2588\u2588 \u2584\u2580\u2580\u2580\u2584\u2588\u2588\u2584\u2580\u2580\u258c\u2588\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2592\u2592\u2588\u2588\u2588 \u258c\u2584\u2584\u2580 \u258c \u2590\u2580\u2588\u2588\u2588\u2588\u2590\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2592\u2590\u2588\u2588\u258c \u2580\u2584\u2584\u2584\u2584\u2580 \u2580\u2580\u2588\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2584\u2588\u2588\u2580 \u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2580 \u2584\u2584\u2588\u2588\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2580\u2588 \u2584\u2588\u2588\u2580 \u2580\u2580\u2580 \u2588 \u2584\u2588 \u2590\u258c \u2584\u2584\u2584\u2584\u2588\u258c \u2580\u2588\u2584\u2584\u2584\u2584\u2580\u2580\u2584 \u258c \u2590 \u2580\u2580\u2584\u2584\u2584\u2580 \u2580\u2580\u2584\u2584\u2580

2017-08-16 18:37:40 - (sanic)[INFO]: Goin' Fast @ http://0.0.0.0:8000 2017-08-16 18:37:40,712 INFO prerender.chromerdp.new_page:47 Created new page 990b1656-05b1-4c93-9909-3299d31579b3 2017-08-16 18:37:40,723 INFO prerender.chromerdp.new_page:47 Created new page 083237b6-e7bf-4e44-97ea-a91312e636f1 2017-08-16 18:37:40,728 INFO prerender.chromerdp.new_page:47 Created new page b81c0ec4-e4a8-489c-a1be-ecd869aa861a 2017-08-16 18:37:40,735 INFO prerender.chromerdp.new_page:47 Created new page cf6d747f-d71f-4519-a74b-6a4f43e29353 2017-08-16 18:37:40,739 INFO prerender.chromerdp.new_page:47 Created new page 5272d7fd-44ed-45ff-b2c3-448fb214e29e 2017-08-16 18:37:40,747 INFO prerender.chromerdp.new_page:47 Created new page 5aac2038-452a-4b56-8e0b-a9a6fc03a8ed 2017-08-16 18:37:40,849 INFO prerender.chromerdp.new_page:47 Created new page 5d9d66ef-c68b-45b8-8e2a-711fa070a972 2017-08-16 18:37:40,863 INFO prerender.chromerdp.new_page:47 Created new page 5402ba0f-3ebd-4579-bc18-f20743deaf4e 2017-08-16 18:37:40,895 INFO prerender.chromerdp.new_page:47 Created new page ec42cd61-49ca-4319-82e4-42cffa53cd8f 2017-08-16 18:37:41,009 INFO prerender.chromerdp.new_page:47 Created new page 019aa698-ae0d-4e08-987c-63547ecd134d 2017-08-16 18:37:41,015 INFO prerender.chromerdp.new_page:47 Created new page 64d86146-6092-4b72-bbd4-c8af5e0acc66 2017-08-16 18:37:41,039 INFO prerender.chromerdp.new_page:47 Created new page fd4bb779-ee77-4ad3-8468-94348c2fda3c 2017-08-16 18:37:41,141 INFO prerender.chromerdp.new_page:47 Created new page 92221c66-8fdc-4494-a0d8-4e432fe805ee 2017-08-16 18:37:41,145 INFO prerender.chromerdp.new_page:47 Created new page 54c984c2-f04e-4b84-a267-7fb4babf9430 2017-08-16 18:37:41,198 INFO prerender.chromerdp.new_page:47 Created new page a6f6d228-4d64-4caf-906d-5ed701d019f4 2017-08-16 18:37:41,203 INFO prerender.chromerdp.new_page:47 Created new page 77111530-da40-48fc-9259-da08aa97972f 2017-08-16 18:37:41 - (sanic)[INFO]: Starting worker [95309] 2017-08-16 18:37:41,204 INFO sanic.serve:526 Starting worker [95309] 2017-08-16 18:42:07,854 INFO prerender.chromerdp.navigate:218 Page 990b1656-05b1-4c93-9909-3299d31579b3 [1] navigating to https://www.apple.com 2017-08-16 18:42:37,858 INFO websockets.protocol.fail_connection:618 Failing the WebSocket connection: 1006 2017-08-16 18:42:37,859 WARNING prerender.app._render:99 Temporary browser failure: , retry rendering https://www.apple.com in 1s 2017-08-16 18:42:37,871 ERROR asyncio.serve:527 Task exception was never retrieved future: <Task finished coro=<Page._evaluate_prerender_ready() done, defined at /usr/local/lib/python3.6/site-packages/prerender/chromerdp.py:232> exception=AttributeError("'NoneType' object has no attribute 'send'",)> Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 234, in _evaluate_prerender_ready res = await self.evaluate('window.prerenderReady == true') File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 228, in evaluate 'params': {'expression': expr} File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 178, in send await self.websocket.send(json.dumps(payload)) AttributeError: 'NoneType' object has no attribute 'send' 2017-08-16 18:42:38,865 INFO prerender.chromerdp.navigate:218 Page 083237b6-e7bf-4e44-97ea-a91312e636f1 [1] navigating to https://www.apple.com 2017-08-16 18:43:07 - (sanic)[ERROR]: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout raise RequestTimeout('Request Timeout') sanic.exceptions.RequestTimeout: Request Timeout

2017-08-16 18:43:07,848 ERROR sanic.log:104 Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout raise RequestTimeout('Request Timeout') sanic.exceptions.RequestTimeout: Request Timeout

2017-08-16 18:43:07,853 INFO websockets.protocol.fail_connection:618 Failing the WebSocket connection: 1006 2017-08-16 18:43:07,853 WARNING prerender.app.handle_request:201 Got 504 for https://www.apple.com in 60001ms 2017-08-16 18:43:07 - (sanic)[ERROR]: Connection lost before response written @ ('96.50.156.110', 18006) 2017-08-16 18:43:07,854 ERROR sanic.write_response:267 Connection lost before response written @ ('96.50.156.110', 18006) 2017-08-16 18:43:07,890 ERROR asyncio.serve:527 Task exception was never retrieved future: <Task finished coro=<Page._evaluate_prerender_ready() done, defined at /usr/local/lib/python3.6/site-packages/prerender/chromerdp.py:232> exception=AttributeError("'NoneType' object has no attribute 'send'",)> Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 234, in _evaluate_prerender_ready res = await self.evaluate('window.prerenderReady == true') File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 228, in evaluate 'params': {'expression': expr} File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 178, in send await self.websocket.send(json.dumps(payload)) AttributeError: 'NoneType' object has no attribute 'send'

beyondcreed commented 7 years ago

Hi. I upgraded to Chromium 60.0.3112.101. However same problem still persists. I run the chrome as follows: chrome --blink-settings=imagesEnabled=false --headless --remote-debugging-port=9222 --disable-gpu "about:blank"

Then, I run prerender as: prerender DEBUG=1. After 2 requests, the prerender will stop working. I may be wrong, but it does not seem like this is chrome issue, instead it looks like something is wrong with prerender code as multiple chrome versions are not working but when I dump DOM with chrome directly it works without a problem (I can do it more then 2 times).

On console error log I am seeing

2017-08-19 15:01:53,544 INFO prerender.chromerdp.navigate:216 Page 77d8643b-7129-420f-9fd2-3000b52648f8 [1] navigating to https://www.thrinacia.com/blog/post/express-checkout-feature 2017-08-19 15:02:53 - (sanic)[ERROR]: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout raise RequestTimeout('Request Timeout') sanic.exceptions.RequestTimeout: Request Timeout

2017-08-19 15:02:53,829 ERROR sanic.log:104 Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout raise RequestTimeout('Request Timeout') sanic.exceptions.RequestTimeout: Request Timeout

2017-08-19 15:02:53,829 WARNING prerender.app.handle_request:201 Got 504 for https://www.thrinacia.com/blog/post/express-checkout-feature in 60286ms

beyondcreed commented 7 years ago

I am also on slack if you need to discuss more. Thanks.

messense commented 7 years ago

@beyondcreed I left message on slack.