psf / requests-html

Pythonic HTML Parsing for Humans™
http://html.python-requests.org
MIT License
13.73k stars 976 forks source link

Javascript rendering error (pyppeteer.errors.PageError: net::ERR_CERT_SYMANTEC_LEGACY) #174

Closed cquark7 closed 5 years ago

cquark7 commented 6 years ago

I am using requests-html library to render JavaScript on a webpage, but I received this error: pyppeteer.errors.PageError: net::ERR_CERT_SYMANTEC_LEGACY

Traceback (most recent call last):

    File "F:/Users/user/scripts/test.py", line 24, in <module>
        page = page.html.render()

    File "F:\Miniconda\lib\site-packages\requests_html.py", line 583, in render
        content, result, page = self.session.loop.run_until_complete(_async_render(url=self.url, script=script, sleep=sleep, wait=wait, content=self.html, reload=reload, scrolldown=scrolldown, timeout=timeout, keep_page=keep_page))

    File "F:\Miniconda\lib\asyncio\base_events.py", line 466, in run_until_complete
        return future.result()

    File "F:\Miniconda\lib\site-packages\requests_html.py", line 545, in _async_render
        await page.goto(url, options={'timeout': int(timeout * 1000)})

    File "F:\Miniconda\lib\site-packages\pyppeteer\page.py", line 644, in goto
        raise PageError(result)

pyppeteer.errors.PageError: net::ERR_CERT_SYMANTEC_LEGACY

How can I fix this error in requests-html?

cquark7 commented 6 years ago

When I open the same website normally using Google Chrome it shows "privacy error". In order to access the site I have to click on "advanced" and then "proceed anyway". This answer https://superuser.com/a/1036062 describes how a developer can disable such errors in Chrome/Chromium, but I don't know how to do it using requests-html

L1ghtn1ng commented 6 years ago

@cquark7 The issue is the ssl cert is not valid, in requests lib there is a verify= param to toggle the ssl check so use that and you should be good

0xmayday commented 6 years ago

the issue with calling .render is that it uses pyppeteer to make the request, which bypasses the verify=False, so you'll see certificate errors when trying to call render. There is an ignoreHTTPSErrors bool argument in pyppeteer, but I can't figure out how to force that within requests-html...

oldani commented 5 years ago

Hi @DavidMay121

On the last version released yesterday, you can just pass verify=False when initializing your session.

CayoM commented 4 years ago

this doesnt seem to be working anymore. The parameter seem to be ignored and there is nothing about this parameter in the official documentation. Any news on this?

isobarbaric commented 2 years ago

I agree with ^, this doesn't seem to be a thing anymore. There isn't any mention of the verify parameter in the docs either. Are there any updates on this?