psf / requests-html

Pythonic HTML Parsing for Humans™
http://html.python-requests.org
MIT License
13.74k stars 976 forks source link

How to render JS to generate fingerprint for cookie? #300

Open northtree opened 5 years ago

northtree commented 5 years ago

This website uses JS to set cookie.

How could I run the JS to mock as browser to avoid 429 error?

from requests_html import HTMLSession

with HTMLSession() as s:
  url = 'https://www.realestate.com.au/auction-results/nsw'
  r = s.get(url)
  print(r.status_code)
  print(r.text)

  r.html.render()
  print(r.text)
northtree commented 5 years ago

The requests from Chrome were captured as following,

  1. request nsw (status 429)
  2. request three JS
  3. request fingerprint XHR which will return new key for the cookie
  4. request nsw again (status 200)
Screen Shot 2019-06-20 at 12 04 41 pm