search

bosondata / chrome-prerender

Render JavaScript-rendered page as HTML/PDF/mhtml/png/jpeg using Headless Chrome
MIT License
165 stars 20 forks source link

https problem on FreeBSD #32

Open beyondcreed opened 6 years ago

beyondcreed commented 6 years ago

I am running into issue when trying to render https based URL's. http works fine. Also chrome in headless can deal with https and spits out DOM, for example when I run: chrome --headless --disable-gpu --dump-dom https://www.apple.com

My prerender curl request is as follows:

curl -v http://myserver.com:8000/https://www.apple.com

Below is output from prerender:

[root@node295 ~]# prerender 2017-08-16 18:37:40 - (sanic)[DEBUG]: \u2584\u2584\u2584\u2584\u2584 \u2580\u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2584\u2584\u2584 _ \u2584\u2584\u2584\u2584\u2584 \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2584 / \ \u2580\u2580\u2580\u2580\u2588\u2588\u2588\u2588\u2588\u258c \u2580\u2590\u2584 \u2580\u2590\u2588 | Gotta go fast! | \u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2584\u2584 \u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2584\u2588\u2588 | ___/ \u2580\u2584\u2584\u2584\u2584\u2584 \u2580\u2580\u2588\u2584\u2580\u2588\u2550\u2550\u2550\u2550\u2588\u2580 |/ \u2580\u2580\u2580\u2584 \u2580\u2580\u2588\u2588\u2588 \u2580 \u2584\u2584 \u2584\u2588\u2588\u2588\u2580\u2580\u2588\u2588\u2584\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2584 \u2584\u2580\u2580\u2580\u2580\u2580\u2580\u2588\u258c \u2588\u2588\u2580\u2584\u2584\u2584\u2588\u2588\u2580\u2584\u2588\u2588\u2588\u2580 \u2580\u2580\u2588\u2588\u2588\u2588 \u2584\u2588\u2588 \u2584\u2580\u2580\u2580\u2584\u2588\u2588\u2584\u2580\u2580\u258c\u2588\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2592\u2592\u2588\u2588\u2588 \u258c\u2584\u2584\u2580 \u258c \u2590\u2580\u2588\u2588\u2588\u2588\u2590\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2592\u2590\u2588\u2588\u258c \u2580\u2584\u2584\u2584\u2584\u2580 \u2580\u2580\u2588\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2584\u2588\u2588\u2580 \u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2580 \u2584\u2584\u2588\u2588\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2580\u2588 \u2584\u2588\u2588\u2580 \u2580\u2580\u2580 \u2588 \u2584\u2588 \u2590\u258c \u2584\u2584\u2584\u2584\u2588\u258c \u2580\u2588\u2584\u2584\u2584\u2584\u2580\u2580\u2584 \u258c \u2590 \u2580\u2580\u2584\u2584\u2584\u2580 \u2580\u2580\u2584\u2584\u2580

2017-08-16 18:37:40 - (sanic)[INFO]: Goin' Fast @ http://0.0.0.0:8000 2017-08-16 18:37:40,712 INFO prerender.chromerdp.new_page:47 Created new page 990b1656-05b1-4c93-9909-3299d31579b3 2017-08-16 18:37:40,723 INFO prerender.chromerdp.new_page:47 Created new page 083237b6-e7bf-4e44-97ea-a91312e636f1 2017-08-16 18:37:40,728 INFO prerender.chromerdp.new_page:47 Created new page b81c0ec4-e4a8-489c-a1be-ecd869aa861a 2017-08-16 18:37:40,735 INFO prerender.chromerdp.new_page:47 Created new page cf6d747f-d71f-4519-a74b-6a4f43e29353 2017-08-16 18:37:40,739 INFO prerender.chromerdp.new_page:47 Created new page 5272d7fd-44ed-45ff-b2c3-448fb214e29e 2017-08-16 18:37:40,747 INFO prerender.chromerdp.new_page:47 Created new page 5aac2038-452a-4b56-8e0b-a9a6fc03a8ed 2017-08-16 18:37:40,849 INFO prerender.chromerdp.new_page:47 Created new page 5d9d66ef-c68b-45b8-8e2a-711fa070a972 2017-08-16 18:37:40,863 INFO prerender.chromerdp.new_page:47 Created new page 5402ba0f-3ebd-4579-bc18-f20743deaf4e 2017-08-16 18:37:40,895 INFO prerender.chromerdp.new_page:47 Created new page ec42cd61-49ca-4319-82e4-42cffa53cd8f 2017-08-16 18:37:41,009 INFO prerender.chromerdp.new_page:47 Created new page 019aa698-ae0d-4e08-987c-63547ecd134d 2017-08-16 18:37:41,015 INFO prerender.chromerdp.new_page:47 Created new page 64d86146-6092-4b72-bbd4-c8af5e0acc66 2017-08-16 18:37:41,039 INFO prerender.chromerdp.new_page:47 Created new page fd4bb779-ee77-4ad3-8468-94348c2fda3c 2017-08-16 18:37:41,141 INFO prerender.chromerdp.new_page:47 Created new page 92221c66-8fdc-4494-a0d8-4e432fe805ee 2017-08-16 18:37:41,145 INFO prerender.chromerdp.new_page:47 Created new page 54c984c2-f04e-4b84-a267-7fb4babf9430 2017-08-16 18:37:41,198 INFO prerender.chromerdp.new_page:47 Created new page a6f6d228-4d64-4caf-906d-5ed701d019f4 2017-08-16 18:37:41,203 INFO prerender.chromerdp.new_page:47 Created new page 77111530-da40-48fc-9259-da08aa97972f 2017-08-16 18:37:41 - (sanic)[INFO]: Starting worker [95309] 2017-08-16 18:37:41,204 INFO sanic.serve:526 Starting worker [95309] 2017-08-16 18:42:07,854 INFO prerender.chromerdp.navigate:218 Page 990b1656-05b1-4c93-9909-3299d31579b3 [1] navigating to https://www.apple.com 2017-08-16 18:42:37,858 INFO websockets.protocol.fail_connection:618 Failing the WebSocket connection: 1006 2017-08-16 18:42:37,859 WARNING prerender.app._render:99 Temporary browser failure: , retry rendering https://www.apple.com in 1s 2017-08-16 18:42:37,871 ERROR asyncio.serve:527 Task exception was never retrieved future: <Task finished coro=<Page._evaluate_prerender_ready() done, defined at /usr/local/lib/python3.6/site-packages/prerender/chromerdp.py:232> exception=AttributeError("'NoneType' object has no attribute 'send'",)> Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 234, in _evaluate_prerender_ready res = await self.evaluate('window.prerenderReady == true') File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 228, in evaluate 'params': {'expression': expr} File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 178, in send await self.websocket.send(json.dumps(payload)) AttributeError: 'NoneType' object has no attribute 'send' 2017-08-16 18:42:38,865 INFO prerender.chromerdp.navigate:218 Page 083237b6-e7bf-4e44-97ea-a91312e636f1 [1] navigating to https://www.apple.com 2017-08-16 18:43:07 - (sanic)[ERROR]: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout raise RequestTimeout('Request Timeout') sanic.exceptions.RequestTimeout: Request Timeout

2017-08-16 18:43:07,848 ERROR sanic.log:104 Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout raise RequestTimeout('Request Timeout') sanic.exceptions.RequestTimeout: Request Timeout

2017-08-16 18:43:07,853 INFO websockets.protocol.fail_connection:618 Failing the WebSocket connection: 1006 2017-08-16 18:43:07,853 WARNING prerender.app.handle_request:201 Got 504 for https://www.apple.com in 60001ms 2017-08-16 18:43:07 - (sanic)[ERROR]: Connection lost before response written @ ('96.50.156.110', 18006) 2017-08-16 18:43:07,854 ERROR sanic.write_response:267 Connection lost before response written @ ('96.50.156.110', 18006) 2017-08-16 18:43:07,890 ERROR asyncio.serve:527 Task exception was never retrieved future: <Task finished coro=<Page._evaluate_prerender_ready() done, defined at /usr/local/lib/python3.6/site-packages/prerender/chromerdp.py:232> exception=AttributeError("'NoneType' object has no attribute 'send'",)> Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 234, in _evaluate_prerender_ready res = await self.evaluate('window.prerenderReady == true') File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 228, in evaluate 'params': {'expression': expr} File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 178, in send await self.websocket.send(json.dumps(payload)) AttributeError: 'NoneType' object has no attribute 'send'

messense commented 6 years ago

WARNING prerender.app._render:99 Temporary browser failure: , retry rendering https://www.apple.com in 1s

Seems like that prerender can't connect to your headless Chrome instance.

What's your Chrome version? How did you start it?

beyondcreed commented 6 years ago

wow, fast response :+1:

it works fine with http based URL, for example if I do

curl -v http://myserver.com:8000/http://www.somesite.com

then it works fine, just doesn't like https

chrome is Chromium 59.0.3071.115, it's started like this chrome --headless --remote-debugging-port=9222 --disable-gpu "about:blank" as per your instructions in README

messense commented 6 years ago

Might related to #13

messense commented 6 years ago

Can you try prerender https://www.apple.com/mac/ ?

beyondcreed commented 6 years ago

unfortantely this does not help, we need it to run with --headless as this is in server environment for rendering our SPA's

messense commented 6 years ago

The problem is with how we use layerPainted event, not --headless, I tried prerender https://www.apple.com/mac/ and it worked.

https://www.apple.com/ has something producing layerPainted event constantly causing prerender timeout.

beyondcreed commented 6 years ago

ok I inferred it was --headless switch from someone mentioning it on the ticket link you sent. so what you are saying its layerPainted?, but how come I can do this chrome --headless --disable-gpu --dump-dom https://www.apple.com it works fine.

also other https targets have same problem, for example if I do

curl -v http://myserver.com:8000/https://www.google.com same problem

it seems something else is problematic with https and how prerender works with this possibly

messense commented 6 years ago

Because in prerender it tries to infer when the page has done rendering. There are several methods:

  1. evaluate window.prerenderReady == true
  2. count finished network requests
  3. wait for DOM tree has no changes in 1s

The last one can be problematic when the page constantly modify DOM like Apple homepage do. (If you SPA don't do that it should work fine)

messense commented 6 years ago

That said, I would recommend you to try to prerender your SPA instead of Apple/Google page.

beyondcreed commented 6 years ago

our SPA does not work as well, we were hoping we would be able to try it out instead of using our existing PhantomJS + Prerender.IO setup, which works fine.

messense commented 6 years ago

So you SPA has the same issue?

beyondcreed commented 6 years ago

We may have to look at puppeteer instead, https://github.com/GoogleChrome/puppeteer#readme

but we will likely need to re-implement some of functionality which you already have here

beyondcreed commented 6 years ago

Yes our SPA has same issue

messense commented 6 years ago

@beyondcreed Try comment out https://github.com/bosondata/chrome-prerender/blob/master/prerender/chromerdp.py#L152 to see if it helps. Thanks!

I got it work for Apple homepage.

messense commented 6 years ago

Since layerPainted events are problematic, I removed it in master. Try:

pip install -U https://github.com/bosondata/chrome-prerender/archive/master.zip

to test it.

beyondcreed commented 6 years ago

If I comment line 152 in chromerdp.py same problem. Will try your new version now as well.

messense commented 6 years ago

If you encountered the same issue again, please post the logs here. Thanks!

beyondcreed commented 6 years ago

ok trying now

Collecting https://github.com/bosondata/chrome-prerender/archive/master.zip Downloading https://github.com/bosondata/chrome-prerender/archive/master.zip \ 20kB 1.5MB/s Requirement already up-to-date: websockets in /usr/local/lib/python3.6/site-packages (from prerender==0.8.1) Requirement already up-to-date: aiohttp in /usr/local/lib/python3.6/site-packages (from prerender==0.8.1) Requirement already up-to-date: ujson in /usr/local/lib/python3.6/site-packages (from prerender==0.8.1) Requirement already up-to-date: sanic>=0.5.4 in /usr/local/lib/python3.6/site-packages (from prerender==0.8.1) Requirement already up-to-date: raven in /usr/local/lib/python3.6/site-packages (from prerender==0.8.1) Requirement already up-to-date: raven-aiohttp in /usr/local/lib/python3.6/site-packages (from prerender==0.8.1) Requirement already up-to-date: diskcache in /usr/local/lib/python3.6/site-packages (from prerender==0.8.1) Requirement already up-to-date: chardet in /usr/local/lib/python3.6/site-packages (from aiohttp->prerender==0.8.1) Requirement already up-to-date: multidict>=2.1.4 in /usr/local/lib/python3.6/site-packages (from aiohttp->prerender==0.8.1) Requirement already up-to-date: async_timeout>=1.2.0 in /usr/local/lib/python3.6/site-packages (from aiohttp->prerender==0.8.1) Requirement already up-to-date: yarl>=0.11 in /usr/local/lib/python3.6/site-packages (from aiohttp->prerender==0.8.1) Requirement already up-to-date: httptools>=0.0.9 in /usr/local/lib/python3.6/site-packages (from sanic>=0.5.4->prerender==0.8.1) Requirement already up-to-date: uvloop>=0.5.3 in /usr/local/lib/python3.6/site-packages (from sanic>=0.5.4->prerender==0.8.1) Requirement already up-to-date: aiofiles>=0.3.0 in /usr/local/lib/python3.6/site-packages (from sanic>=0.5.4->prerender==0.8.1) Requirement already up-to-date: contextlib2 in /usr/local/lib/python3.6/site-packages (from raven->prerender==0.8.1) Installing collected packages: prerender Found existing installation: prerender 0.8.1 Uninstalling prerender-0.8.1: Successfully uninstalled prerender-0.8.1 Running setup.py install for prerender ... done Successfully installed prerender-0.8.1

beyondcreed commented 6 years ago

prerender DEBUG=1 2017-08-16 19:59:22 - (sanic)[DEBUG]: \u2584\u2584\u2584\u2584\u2584 \u2580\u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2584\u2584\u2584 _ \u2584\u2584\u2584\u2584\u2584 \u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2584 / \ \u2580\u2580\u2580\u2580\u2588\u2588\u2588\u2588\u2588\u258c \u2580\u2590\u2584 \u2580\u2590\u2588 | Gotta go fast! | \u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2584\u2584 \u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2584\u2588\u2588 | ___/ \u2580\u2584\u2584\u2584\u2584\u2584 \u2580\u2580\u2588\u2584\u2580\u2588\u2550\u2550\u2550\u2550\u2588\u2580 |/ \u2580\u2580\u2580\u2584 \u2580\u2580\u2588\u2588\u2588 \u2580 \u2584\u2584 \u2584\u2588\u2588\u2588\u2580\u2580\u2588\u2588\u2584\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2584 \u2584\u2580\u2580\u2580\u2580\u2580\u2580\u2588\u258c \u2588\u2588\u2580\u2584\u2584\u2584\u2588\u2588\u2580\u2584\u2588\u2588\u2588\u2580 \u2580\u2580\u2588\u2588\u2588\u2588 \u2584\u2588\u2588 \u2584\u2580\u2580\u2580\u2584\u2588\u2588\u2584\u2580\u2580\u258c\u2588\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2592\u2592\u2588\u2588\u2588 \u258c\u2584\u2584\u2580 \u258c \u2590\u2580\u2588\u2588\u2588\u2588\u2590\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2592\u2590\u2588\u2588\u258c \u2580\u2584\u2584\u2584\u2584\u2580 \u2580\u2580\u2588\u2588\u2588\u2588\u2592\u2592\u2592\u2592\u2584\u2588\u2588\u2580 \u2580\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2580 \u2584\u2584\u2588\u2588\u2580\u2588\u2588\u2588\u2588\u2588\u2588\u2580\u2588 \u2584\u2588\u2588\u2580 \u2580\u2580\u2580 \u2588 \u2584\u2588 \u2590\u258c \u2584\u2584\u2584\u2584\u2588\u258c \u2580\u2588\u2584\u2584\u2584\u2584\u2580\u2580\u2584 \u258c \u2590 \u2580\u2580\u2584\u2584\u2584\u2580 \u2580\u2580\u2584\u2584\u2580

2017-08-16 19:59:22 - (sanic)[INFO]: Goin' Fast @ http://0.0.0.0:8000 2017-08-16 19:59:22,751 INFO prerender.chromerdp.new_page:49 Created new page fc6548e1-041e-4501-b99b-645c8a871d19 2017-08-16 19:59:22,760 INFO prerender.chromerdp.new_page:49 Created new page cd1a4b9b-2283-456e-b8af-f2e4c6729000 2017-08-16 19:59:22,767 INFO prerender.chromerdp.new_page:49 Created new page f1ccfb0b-0f85-4f18-8f16-6fd915fb6867 2017-08-16 19:59:22,775 INFO prerender.chromerdp.new_page:49 Created new page 29b4a610-e7d4-4110-b3fb-a6aa7c564814 2017-08-16 19:59:22,784 INFO prerender.chromerdp.new_page:49 Created new page 87cd694a-ebd7-43d3-9b5e-b54222692228 2017-08-16 19:59:22,787 INFO prerender.chromerdp.new_page:49 Created new page 64d444f4-744b-4085-a7d5-fa5ff4e9d1d9 2017-08-16 19:59:22,916 INFO prerender.chromerdp.new_page:49 Created new page 733978e8-6978-4cfb-ab49-d954322e9214 2017-08-16 19:59:23,050 INFO prerender.chromerdp.new_page:49 Created new page 9474f707-e168-494f-a21e-5121cd111cb4 2017-08-16 19:59:23,059 INFO prerender.chromerdp.new_page:49 Created new page b2e2de74-88c4-4461-807c-39087af49586 2017-08-16 19:59:23,065 INFO prerender.chromerdp.new_page:49 Created new page 7d2f682c-29e7-4f28-b424-5f0dcf4b85c6 2017-08-16 19:59:23,107 INFO prerender.chromerdp.new_page:49 Created new page 0f0ef699-a9af-4b6d-b350-7535dfca66b5 2017-08-16 19:59:23,116 INFO prerender.chromerdp.new_page:49 Created new page b2bd0362-fe01-4469-9d87-5df557037c75 2017-08-16 19:59:23,143 INFO prerender.chromerdp.new_page:49 Created new page dc69a01d-b80f-4768-8073-1c6450df8e84 2017-08-16 19:59:23,242 INFO prerender.chromerdp.new_page:49 Created new page 0a4a3701-5c11-4a17-a029-7abac531581c 2017-08-16 19:59:23,316 INFO prerender.chromerdp.new_page:49 Created new page de021820-7009-4608-a142-05d7b373992c 2017-08-16 19:59:23,325 INFO prerender.chromerdp.new_page:49 Created new page cb288c6c-3184-4e1b-a4f3-fbc54a956261 2017-08-16 19:59:23 - (sanic)[INFO]: Starting worker [95687] 2017-08-16 19:59:23,326 INFO sanic.serve:526 Starting worker [95687] 2017-08-16 19:59:30,409 INFO prerender.chromerdp.navigate:216 Page fc6548e1-041e-4501-b99b-645c8a871d19 [1] navigating to https://origin.thrinacia.com 2017-08-16 20:00:00,418 INFO websockets.protocol.fail_connection:618 Failing the WebSocket connection: 1006 2017-08-16 20:00:00,419 WARNING prerender.app._render:99 Temporary browser failure: , retry rendering https://origin.thrinacia.com in 1s 2017-08-16 20:00:00,447 ERROR asyncio.serve:527 Task exception was never retrieved future: <Task finished coro=<Page._evaluate_prerender_ready() done, defined at /usr/local/lib/python3.6/site-packages/prerender/chromerdp.py:230> exception=AttributeError("'NoneType' object has no attribute 'send'",)> Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 232, in _evaluate_prerender_ready res = await self.evaluate('window.prerenderReady == true') File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 226, in evaluate 'params': {'expression': expr} File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 176, in send await self.websocket.send(json.dumps(payload)) AttributeError: 'NoneType' object has no attribute 'send' 2017-08-16 20:00:01,423 INFO prerender.chromerdp.navigate:216 Page cd1a4b9b-2283-456e-b8af-f2e4c6729000 [1] navigating to https://origin.thrinacia.com 2017-08-16 20:00:30 - (sanic)[ERROR]: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout raise RequestTimeout('Request Timeout') sanic.exceptions.RequestTimeout: Request Timeout

2017-08-16 20:00:30,415 ERROR sanic.log:104 Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout raise RequestTimeout('Request Timeout') sanic.exceptions.RequestTimeout: Request Timeout

2017-08-16 20:00:30,419 INFO websockets.protocol.fail_connection:618 Failing the WebSocket connection: 1006 2017-08-16 20:00:30,420 WARNING prerender.app.handle_request:201 Got 504 for https://origin.thrinacia.com in 60012ms 2017-08-16 20:00:30 - (sanic)[ERROR]: Connection lost before response written @ ('96.50.156.110', 15034) 2017-08-16 20:00:30,420 ERROR sanic.write_response:267 Connection lost before response written @ ('96.50.156.110', 15034) 2017-08-16 20:00:30,423 ERROR asyncio.serve:527 Task exception was never retrieved future: <Task finished coro=<Page._evaluate_prerender_ready() done, defined at /usr/local/lib/python3.6/site-packages/prerender/chromerdp.py:230> exception=AttributeError("'NoneType' object has no attribute 'send'",)> Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 232, in _evaluate_prerender_ready res = await self.evaluate('window.prerenderReady == true') File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 226, in evaluate 'params': {'expression': expr} File "/usr/local/lib/python3.6/site-packages/prerender/chromerdp.py", line 176, in send await self.websocket.send(json.dumps(payload)) AttributeError: 'NoneType' object has no attribute 'send'

beyondcreed commented 6 years ago

still a problem

messense commented 6 years ago

You might want to upgrade your Chromium to latest version to avoid any issue with devtools protocol the old version might have.

beyondcreed commented 6 years ago

I believe that is latest I have in my packages, let me check

beyondcreed commented 6 years ago

Chromium 59.0.3071.115 is latest I have http://www.freshports.org/www/chromium/

messense commented 6 years ago

Thanks! I have reproduced the problem locally, I'll try to find out what is causing it.

messense commented 6 years ago

I see your website set prerenderReady to false but never set it to true which might caused the issue because we prefer to read window.prerenderReady over other methods.

https://github.com/bosondata/chrome-prerender/blob/master/prerender/chromerdp.py#L242

beyondcreed commented 6 years ago

ok, btw your current version is working with https://www.apple.com

beyondcreed commented 6 years ago

I will let front-end developer know about prerenderReady. If there is any other issues they will follow up on this thread. thanks.

messense commented 6 years ago

@beyondcreed Try the current master, it should work with your website

beyondcreed commented 6 years ago

very nice, it worked.

beyondcreed commented 6 years ago

thanks for fixing that quickly

messense commented 6 years ago

For more accurate result, you should set prerenderReady to true in JS when you know it's done rendering.

beyondcreed commented 6 years ago

hm, just tried it on our main site, https://www.thrinacia.com/blog/post/express-checkout-feature

but it waits very long, then returns with error, debug is:

2017-08-16 20:43:29,951 INFO prerender.chromerdp.navigate:216 Page 135db0bc-8e9d-495b-b50e-10f9f4905de8 [1] navigating to https://www.thrinacia.com/blog/post/express-checkout-feature 2017-08-16 20:44:29 - (sanic)[ERROR]: Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout raise RequestTimeout('Request Timeout') sanic.exceptions.RequestTimeout: Request Timeout

2017-08-16 20:44:29,971 ERROR sanic.log:104 Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/sanic/server.py", line 143, in connection_timeout raise RequestTimeout('Request Timeout') sanic.exceptions.RequestTimeout: Request Timeout

2017-08-16 20:44:29,971 WARNING prerender.app.handle_request:201 Got 504 for https://www.thrinacia.com/blog/post/express-checkout-feature in 60028ms 2017-08-16 20:44:29 - (network)[INFO][96.50.156.110:57234]:

messense commented 6 years ago

Because counting network requests isn't very reliable when you have some external service like disqus loading lots of stuffs.

Turn on DEBUG=1 you will see that the page sends over 160 network requests. Manually set prerenderReady to true should resolve this issue.

beyondcreed commented 6 years ago

yeah, I didn't write this code, talking to front-end devs to see if we can resolve, thanks for info

beyondcreed commented 6 years ago

it works ok in our existing phantomJS setup with prerender.io, but here it hangs, likely due to what you are saying.

messense commented 6 years ago

@beyondcreed I tried it locally, it works sometimes, hangs sometimes, depends on how long it takes to finish all the network requests. prerender has default timeout set to 30s, you can set envvar PRERENDER_TIMEOUT=xx to increase/decrease it

messense commented 6 years ago

I think prerender.io can render it is because they disabled image loading while we didn't. #20

beyondcreed commented 6 years ago

the problem is some sites like linkedin, facebook, etc. don't wait long, max out at 10s when crawling, so output generally needs to be pre-rendered and served relatively fast. I tried it on linkedin and it times out. going to look at #20

messense commented 6 years ago

I looked at your website, I think it has too many javascript and css files to load, might be worth to use tools like webpack to pack vendor js/css together to reduce number of requests.

beyondcreed commented 6 years ago

we already pack it (minify and build checksums), also static resources are served off CDN.

I am trying chrome with --blink-settings=imagesEnabled=false

beyondcreed commented 6 years ago

still times out

messense commented 6 years ago

we already pack it (minify and build checksums),

image

You might be doing something wrong, this is a lot of scripts to load, ideally they should be lazy loaded on demand.

messense commented 6 years ago

On my laptop, it takes Chrome at least 10+s to load the page already. (with cache disabled)

image

beyondcreed commented 6 years ago

hi, I didn't write the code, but front-end developers who did, pack it using gulp process and generate checksums so we can bust cache (you can see code is already minified if you curl the URL, e.g. spaces are removed, etc.). Front-end dev mentioned due to legacy angular framework being used code could not be easily lazily loaded, we tried it before quite a few times and there were many issues with this.

beyondcreed commented 6 years ago

phantomjs with prerender.io renders it quite fast, so I am just surprised to see this timing out, that is all

beyondcreed commented 6 years ago

also if you are loading resources from overseas (china, etc.) it may take longer to load due to cdn nodes not located there, on my chrome load is at 2.51s

messense commented 6 years ago

I understand. I have done lazy loading legacy angular framework (1.x) with webpack before, so if you guys have issues I might be able to help. :-)

with --blink-settings=imagesEnabled=false I can render it within 12s so I think it should be better for your network?

messense commented 6 years ago

https://github.com/bosondata/chrome-prerender/commit/22233d40b0c78a89897942bba5ddc891ab6803f6#diff-b987f64b4374fdef9d36193a6f11b3ddR252

Becasue we sleep 0.2s there, manually setting prerenderReady to true might save you 2s (10 iterations).

beyondcreed commented 6 years ago

ok, I can message you privately on this, may need to discuss things further. I'll email you on how we can connect

beyondcreed commented 6 years ago

ok sent you email and slack invite, thanks