Add support for providing certificate files for the websites

[bhishanpdl]

🚀 The feature

Problem

I am working in a computer working behind a proxy wall and I need to have website certificate file to send request to any website. Is there a way to pass the key like 'cert' or 'cert_file' to the function app.add(url,cert='my_certificate_for_given_website.pem'?

I can see bunch of kwargs in app.add but I am not sure how we can pass the parameter verify to module requests or whichever module is used to download the website contents.

Code

import os
from embedchain import App

# Replace this with your OpenAI key
os.environ["OPENAI_API_KEY"] = my_oenai_key

app = App()
app.add("https://www.forbes.com/profile/elon-musk") # It gives SSL error here.
app.add("https://en.wikipedia.org/wiki/Elon_Musk")
app.query("What is the net worth of Elon Musk today?")

Error Got

File [~\venv\py311embedchain\Lib\site-packages\requests\adapters.py:517](http://localhost:8888/lab/tree/aa_Learn/LLM/OpenAI/Some_modules/module_embedchain/~/venv/py311embedchain/Lib/site-packages/requests/adapters.py#line=516), in HTTPAdapter.send(self, request, stream, timeout, verify, cert, proxies)
    513         raise ProxyError(e, request=request)
    515     if isinstance(e.reason, _SSLError):
    516         # This branch is for urllib3 v1.22 and later.
--> 517         raise SSLError(e, request=request)
    519     raise ConnectionError(e, request=request)
    521 except ClosedPoolError as e:

SSLError: HTTPSConnectionPool(host='www.forbes.com', port=443): Max retries exceeded with url: /profile/elon-musk (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)')))
2024-03-18 12:45:28,235 - 3616 - _common.py-_common:105 - INFO: Backing off send_request(...) for 0.0s (requests.exceptions.SSLError: HTTPSConnectionPool(host='us-api.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)'))))
2024-03-18 12:45:28,693 - 3616 - _common.py-_common:105 - INFO: Backing off send_request(...) for 1.9s (requests.exceptions.SSLError: HTTPSConnectionPool(host='us-api.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)'))))
2024-03-18 12:45:31,102 - 3616 - _common.py-_common:105 - INFO: Backing off send_request(...) for 1.1s (requests.exceptions.SSLError: HTTPSConnectionPool(host='us-api.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)'))))
2024-03-18 12:45:32,789 - 3616 - _common.py-_common:120 - ERROR: Giving up send_request(...) after 4 tries (requests.exceptions.SSLError: HTTPSConnectionPool(host='us-api.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1006)'))))

Motivation, pitch

I am working on a computer which is working behind the proxy firewall and I need to have website certificate to download the contents of any website (eg. using request.get).

Currently, I am not sure how to pass the certificate to the app function: app.add(url), without passing the certificate file, I can not use the module.

I assume that many programmers working behind the proxy and it will be beneficial to all of us.

[TravisRoad]

same err here

[penguyen72]

Any updates on this?

[axoxymous-coder]

Similar error encountered:

requests.exceptions.SSLError: HTTPSConnectionPool(host='www.forbes.com', port=443): Max retries exceeded with url: /profile/elon-musk (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))) 2024-07-05 15:31:25,245 - 13040119808 - _common.py-_common:105 - INFO: Backing off send_request(...) for 0.4s (requests.exceptions.SSLError: HTTPSConnectionPool(host='us-api.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)')))) 2024-07-05 15:31:26,179 - 13040119808 - _common.py-_common:105 - INFO: Backing off send_request(...) for 1.0s (requests.exceptions.SSLError: HTTPSConnectionPool(host='us-api.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)')))) 2024-07-05 15:31:27,783 - 13040119808 - _common.py-_common:105 - INFO: Backing off send_request(...) for 3.6s (requests.exceptions.SSLError: HTTPSConnectionPool(host='us-api.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)')))) 2024-07-05 15:31:32,004 - 13040119808 - _common.py-_common:120 - ERROR: Giving up send_request(...) after 4 tries (requests.exceptions.SSLError: HTTPSConnectionPool(host='us-api.i.posthog.com', port=443): Max retries exceeded with url: /batch/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))))