rajatomar788 / pywebcopy

Locally saves webpages to your hard disk with images, css, js & links as is.
https://rajatomar788.github.io/pywebcopy/
Other
520 stars 105 forks source link

How to change session header settings when getting a 403 Forbiden error #110

Closed Alipser closed 1 year ago

Alipser commented 1 year ago

I can't figure out how to change the header to set the user-agent

rajatomar788 commented 1 year ago

There are two ways to do this,

First if you want to set everything up by yourself then for that the actual WebPage element is located in the pywebcopy.core submodule. So you would do something like this

from pywebcopy.core import WebPage

and then create an instance and so on.

But secondly if you want to only change a few variables then you can just copy the insides of the save_webpage function and do something like this


from pywebcopy.config import get_config
config = get_config(url, project_folder, project_name, bypass_robots, debug, delay, threaded)
webpage = config.create_page()

# here you have the ready webpage instance to do anything
webpage.session.header = anything
webpage.get(url)

webpage.save_complete(pop=open_in_browser)