search

openshift-psap / llm-eval-test

Custom wrapper around lm-evaluation-harness and Unitxt
MIT License
1 stars 0 forks source link

Cannot run against self-signed endpoints #19

Open sjmonson opened 1 week ago

sjmonson commented 1 week ago

Running llm-eval-test against a self-signed HTTPS endpoint fail with the following message:

  File "/var/home/core/.local/lib/python3.9/site-packages/lm_eval/models/api_models.py", line 335, in model_call
    response = requests.post(
  File "/var/home/core/.local/lib/python3.9/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/var/home/core/.local/lib/python3.9/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/var/home/core/.local/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/var/home/core/.local/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/var/home/core/.local/lib/python3.9/site-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='example.com', port=443): Max retries exceeded with url: /v1/completions (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1129)')))
Requesting API:   0%|
sjmonson commented 1 week ago

This is an issue in lm-evaluation-harness and has been fixed upstream (see https://github.com/EleutherAI/lm-evaluation-harness/pull/2440).