siznax / wptools

Wikipedia tools (for Humans): easily extract data from Wikipedia, Wikidata, and other MediaWikis
MIT License
575 stars 79 forks source link

how do I specify a mediawiki URL that requires an SSL certificate? #133

Closed fredzannarbor closed 5 years ago

fredzannarbor commented 6 years ago
>>> page = wptools.page**(wiki='xxx')**
xxx (random) 🍭
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/.local/lib/python3.5/site-packages/wptools/page.py", line 80, in __init__
    self.get_random()
  File "/home/ubuntu/.local/lib/python3.5/site-packages/wptools/page.py", line 693, in get_random
    self._get('random', show, proxy, timeout)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/wptools/core.py", line 175, in _get
    response = req.get(qstr, qobj.status)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/wptools/request.py", line 76, in get
    return self.curl_perform(crl)
  File "/home/ubuntu/.local/lib/python3.5/site-packages/wptools/request.py", line 84, in curl_perform
    crl.perform()
pycurl.error: (60, 'server certificate verification failed. CAfile: /home/ubuntu/.local/lib/python3.5/site-packages/certifi/cacert.pem CRLfile: none')
siznax commented 5 years ago

Thanks for trying wptools, @fredzannarbor!

We use certifi for certificate verification with pycurl.

wptools/wptools/request.curl_setup():

        crl = pycurl.Curl()
        crl.setopt(pycurl.USERAGENT, user_agent())
        crl.setopt(pycurl.FOLLOWLOCATION, True)
        crl.setopt(pycurl.CAINFO, certifi.where())

If you want to provide your own CA bundle, we can expose CAINFO or CAPATH (see curl_easy_setopt docs).

Is that what you're looking for, or something else? How would you expect it to work?

siznax commented 5 years ago

Feel free to re-open if you want to continue the conversation @fredzannarbor.