Closed fumbles closed 1 year ago
You can do it as below
from pywebcopy.config import get_config
config = get_config(url, project_folder, project_name, bypass_robots, debug, delay, threaded)
webpage = config.create_page()
# here change the ssl verification settings
webpage.session.verify = False
webpage.get(url)
webpage.save_complete(pop=open_in_browser)
Thanks! I got it to work with crawler to traverse the whole site. I'm getting an error when encountering the 3 shockwave scripts that are in there but otherwise it's great....on my list to sort out.
LocationParseError: clsid:D27CDB6E-AE6D-11cf-96B8-444553540000
from pywebcopy.configs import get_config
from pywebcopy import warnings
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
url=url
project_folder=path
project_name=name
bypass_robots=True
debug=False
open_in_browser=True
delay=None
threaded=False
config = get_config(url, project_folder, project_name, bypass_robots, debug, delay, threaded)
crawler = config.create_crawler()
crawler.session.verify = False # for SSL verification
crawler.get(url)
if threaded:
warnings.warn(
"Opening in browser is not supported when threading is enabled!")
open_in_browser = False
crawler.save_complete()
You should set debug=True
to get the log of actual events happening to assess the error.
I do not see a way to account for the invalid certificate.