psf / requests-html

Pythonic HTML Parsing for Humans™
http://html.python-requests.org
MIT License
13.7k stars 975 forks source link

r.html.render() does not update cookies? Manual extracting of cookies possible? #109

Open patarapolw opened 6 years ago

patarapolw commented 6 years ago

These lines, actually

self.session = Requests(username, password).session
self.session.get('https://www.wanikani.com/dashboard').html.render()
self.session.get('https://community.wanikani.com/latest').html.render()

Which is from this project, where I simulate by disabling/enabling Javascript.

kennethreitz commented 6 years ago

not yet.

kennethreitz commented 6 years ago

in the meantime, try html.render(reload=False)

shinglyu commented 6 years ago

So does this mean HTTP Basic Auth is not working when using render() ?

liiight commented 6 years ago

Any update on this issue? Thanks!

pash99 commented 5 years ago

Extraction of cookies from rendered page:

import requests_html
import asyncio

my_url = "https://example.net"

with requests_html.HTMLSession() as s:
    r = s.get(my_url)
    r.html.render(keep_page=True)
    l = asyncio.get_event_loop()
    cookies = l.run_until_complete(r.html.page.cookies())
    userAgent = l.run_until_complete(s.browser.userAgent())
    # Make sure to close the page
    l.run_until_complete(r.html.page.close())
Exid87 commented 5 years ago

Wondering if there's an update to this issue. I'm able to extract cookies and get the page I need fine. Once I make the call to render the page, the cookies disappear and the page renders my login page instead.

jonathanmishler commented 4 years ago

If you install from the repo, the requests_html.HTMLResponse.html.render() function is updated to include a new parameter send_cookies_session: bool=False. I used r.html.render(send_cookies_session=True) and the webpage was no longer giving me an error about not accepting cookies.

Franky1 commented 4 years ago

I downloaded the latest head revision from git and installed it with pip. I tried it with three different parameters:

  1. r.html.render(send_cookies_session=True)
  2. r.html.render(keep_page=True)
  3. r.html.render(reload=False)

None of these worked, i am always set back to the login page after calling the render() method...

Code-Layman commented 3 years ago

the problem remains. i use cookies to get url and then render it defaultly to get the html. but it returns only the html of login page. r.html.render(cookies=my_cookies) or r.html.render(send_cookies_session=True) returns error. that is render() got an unexpected keyword argument.

hotdog5225 commented 2 years ago

any solution to this problem now? resp.html.render() always without cookies.....

agpldev commented 1 year ago

Any update on this issue?

Mr-DRP commented 7 months ago

Any Update ?

bogburner commented 6 months ago

Bump to this. send_cookies_session bool, reload=False, and including cookies in the render() call (as detailed in readthedocs) return unexpected keyword arguments or fail to render the authed page (in the case of reload).