psf / requests

A simple, yet elegant, HTTP library.
https://requests.readthedocs.io/en/latest/
Apache License 2.0
52.19k stars 9.33k forks source link

Request not throwing requests.exceptions.InvalidSchema on custom HTTPAdapter #6703

Closed defkev closed 6 months ago

defkev commented 6 months ago

Mounting a custom HTTPAdapter (e.g. for retry) using an unsupported protocol (e.g. typo) doesn't throw requests.exceptions.InvalidSchema like the default adapter?!

Expected Result

Should throw requests.exceptions.InvalidSchema

Actual Result

Traceback (most recent call last):
  File "python\lib\site-packages\requests\adapters.py", line 456, in send
    conn = self.get_connection(request.url, proxies)
  File "python\lib\site-packages\requests\adapters.py", line 358, in get_connection
    conn = self.poolmanager.connection_from_url(url)
  File "python\lib\site-packages\urllib3\poolmanager.py", line 299, in connection_from_url
    return self.connection_from_host(
  File "python\lib\site-packages\urllib3\poolmanager.py", line 246, in connection_from_host
    return self.connection_from_context(request_context)
  File "python\lib\site-packages\urllib3\poolmanager.py", line 258, in connection_from_context
    raise URLSchemeUnknown(scheme)
urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme htxp

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 21, in <module>
    with session.get(url.geturl()) as r:
  File "python\lib\site-packages\requests\sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
  File "python\lib\site-packages\requests\sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "python\lib\site-packages\requests\sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "python\lib\site-packages\requests\adapters.py", line 458, in send
    raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Not supported URL scheme htxp

Reproduction Steps

from urllib.parse import urlparse
import requests
import urllib3

url = urlparse('htxp://example.com/test')

retries = requests.packages.urllib3.util.retry.Retry(
    status_forcelist = tuple(range(500, 600))
)
adapter = requests.adapters.HTTPAdapter(max_retries = retries)
session = requests.Session()
session.mount('%s://' % (url.scheme), adapter)

try:
    with session.get(url.geturl()) as r: # Should throw requests.exceptions.InvalidSchema
        print(r.status_code)
except requests.exceptions.InvalidSchema as e: # Not thrown!
    print(e)
except urllib3.exceptions.URLSchemeUnknown as e: # Instead this does but can't catch it?
    print(e)
except requests.exceptions.InvalidURL as e: # Thrown by above exception
    print(e)

System Information

$ python -m requests.help
{
  "requests": {
    "version": "2.28.1"
  },
  "urllib3": {
    "version": "1.26.13"
  }
}
sigmavirus24 commented 6 months ago

This is one hundred percent your own fault. InvalidSchema is documented as a problem when we cannot find a suitable adapter to use for a request and adapters are found by looking at the longest prefix. You registered one with htxp so you then told Requests you were registering an adapter to handle that scheme. After that, we were unable to use the adapter you configured to complete the request. Finally you're concerned with an exception we catch and then raise a new exception from, there you're being tripped up by the way exceptions are printed in python 3. You're also handling the correct exception when we can't manage that request for you

defkev commented 6 months ago

The typo (htxp) was just an example, i am seeing the exact same behavior with valid schemes (e.g. rtmp://) unsupported by urllib3 also throwing requests.exceptions.InvalidURL

The million dollar question is: How do i catch urllib3.exceptions.URLSchemeUnknown in the stack if the URL is in fact NOT invalid (htxp:// vs rtmp://) if requests throws requests.exceptions.InvalidURL for both without adding my own protocol validator?

sigmavirus24 commented 6 months ago

If you're writing your own totally custom adapters for those protocols then why do you care about catching an exception from urllib3?

If you're just using the HTTPAdapter from us for unsupported protocols why do you want to handle an exception from urllib3?

None of this makes any sense as a use case and it seems you're not reading the documentation and expecting me to give you solutions to problems that aren't problematic. This is now a request for help that belongs on StackOverflow.