CJWorkbench / loadurl

Workbench module that downloads CSV, Excel or JSON files from a website.
0 stars 0 forks source link

SSL Certificate Verification Error #6

Open arky opened 4 years ago

arky commented 4 years ago

Loading https://ncov.moh.gov.vn results in following SSL error:

Can't connect: Cannot connect to host ncov.moh.gov.vn:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)')]
1

SSL Certificate Check: http://www.sslchecker.com/sslchecker?su=cda559548bf6fced7ba19d14d82d118d

arky commented 4 years ago

This site also triggers the SSL verification error https://www.covid19.gov.la/

http://www.sslchecker.com/sslchecker?su=776db90ef766afdf2ef0c9efe55f6c16

adamhooper commented 4 years ago

Both sites have SSL errors. (sslchecker shows the errors, but its design makes them hard to see.)

Google Chrome and Firefox go and download the intermediate certificates themselves: they don't need to look at these intermediate certificates. But Workbench doesn't do that: our (Python) SSL library can't make HTTP requests.

Long story short: we don't handle misconfigured SSL servers. That's frustrating when other tools can securely validate the server.

I don't know of a solution within the Python ecosystem. Maybe we should implement our fetching with a different tool....

arky commented 4 years ago

Thank you @adamhooper for looking into this problem. Unfortunately most of the sites in southeast asia (esp. public agencies) seems to be mis-configured. :(

Requests has a 'verify' boolean flag to turn off SSL checks.That seems to work the best for this tool.

Perhaps a checkbox '[ ] Ignore SSL errors' that toggles 'verify_ssl=False' could be stopgap measure over custom resolver or certificate bundle solution?