search

encode / httpx

A next generation HTTP client for Python. 🦋
https://www.python-httpx.org/
BSD 3-Clause "New" or "Revised" License
13.12k stars 835 forks source link

SyncHTTP11Connection ImportError on Python 3.9 when using ThreadPoolExecutor #1393

Closed senpos closed 3 years ago

senpos commented 3 years ago

Checklist

Describe the bug

When using httpx with ThreadPoolExecutor on Python 3.9 I get an exception about the import error / circular import of SyncHTTP11Connection.

I can't reproduce it on Python 3.8.

To reproduce

from concurrent.futures import ThreadPoolExecutor, as_completed

import httpx

_MAX_WORKERS = 4
_TASKS = 10

_HTTPBIN_BASE_URL = "https://httpbin.org/"

def do_post(client: httpx.Client):
    response = client.post("post")
    return response

def main():
    client = httpx.Client(timeout=5.0, base_url=_HTTPBIN_BASE_URL)

    with ThreadPoolExecutor(max_workers=_MAX_WORKERS) as executor:
        futures = [executor.submit(do_post, client) for _ in range(_TASKS)]

        for future in as_completed(futures):
            response = future.result()
            print(f"{response.status_code=}")

if __name__ == "__main__":
    main()

With _MAX_WORKERS=4 I get the error, but sometimes, when I change its value it works fine. Sometimes.

Expected behavior

It should not throw an exception regarding circular imports when using ThreadPoolExecutor. :-)

Actual behavior

Well, it throws it. :D

Debugging material

Traceback with HTTPX_LOG_LEVEL=trace ```bash $ HTTPX_LOG_LEVEL=trace python main.py TRACE [2020-11-19 18:15:10] httpx._config - load_ssl_context verify=True cert=None trust_env=True http2=False TRACE [2020-11-19 18:15:10] httpx._config - load_verify_locations cafile=/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/certifi/cacert.pem TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443) TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - created connection= TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - adding connection to pool= TRACE [2020-11-19 18:15:10] httpcore._sync.connection - open_socket origin=(b'https', b'httpbin.org', 443) timeout={'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0} TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443) TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - created connection= TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - adding connection to pool= TRACE [2020-11-19 18:15:10] httpcore._sync.connection - open_socket origin=(b'https', b'httpbin.org', 443) timeout={'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0} TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443) TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - created connection= TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - adding connection to pool= TRACE [2020-11-19 18:15:10] httpcore._sync.connection - open_socket origin=(b'https', b'httpbin.org', 443) timeout={'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0} TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443) TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - created connection= TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - adding connection to pool= TRACE [2020-11-19 18:15:10] httpcore._sync.connection - open_socket origin=(b'https', b'httpbin.org', 443) timeout={'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0} TRACE [2020-11-19 18:15:10] httpcore._sync.connection - create_connection socket= http_version='HTTP/1.1' TRACE [2020-11-19 18:15:10] httpcore._sync.connection - create_connection socket= http_version='HTTP/1.1' TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - remove from pool connection= TRACE [2020-11-19 18:15:10] httpcore._sync.connection - create_connection socket= http_version='HTTP/1.1' TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - removing connection from pool= TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - remove from pool connection= TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443) TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - removing connection from pool= TRACE [2020-11-19 18:15:10] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - created connection= TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - adding connection to pool= TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_data=Data(<0 bytes>) TRACE [2020-11-19 18:15:10] httpcore._sync.connection - open_socket origin=(b'https', b'httpbin.org', 443) timeout={'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0} TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443) TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - created connection= TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - adding connection to pool= TRACE [2020-11-19 18:15:10] httpcore._sync.connection - open_socket origin=(b'https', b'httpbin.org', 443) timeout={'connect': 5.0, 'read': 5.0, 'write': 5.0, 'pool': 5.0} DEBUG [2020-11-19 18:15:10] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK" TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - receive_event=Data(<395 bytes>) TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - receive_event=EndOfMessage(headers=) TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443) TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - reusing idle http11 connection= TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - reuse connection= TRACE [2020-11-19 18:15:10] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_data=Data(<0 bytes>) TRACE [2020-11-19 18:15:10] httpcore._sync.connection - create_connection socket= http_version='HTTP/1.1' TRACE [2020-11-19 18:15:10] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_data=Data(<0 bytes>) DEBUG [2020-11-19 18:15:10] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK" TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - receive_event=Data(<395 bytes>) TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - receive_event=EndOfMessage(headers=) TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443) TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - reusing idle http11 connection= TRACE [2020-11-19 18:15:10] httpcore._sync.connection_pool - reuse connection= TRACE [2020-11-19 18:15:10] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:10] httpcore._sync.http11 - send_data=Data(<0 bytes>) DEBUG [2020-11-19 18:15:11] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK" TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=Data(<395 bytes>) TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=EndOfMessage(headers=) TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443) TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - reusing idle http11 connection= TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - reuse connection= TRACE [2020-11-19 18:15:11] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_data=Data(<0 bytes>) TRACE [2020-11-19 18:15:11] httpcore._sync.connection - create_connection socket= http_version='HTTP/1.1' TRACE [2020-11-19 18:15:11] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_data=Data(<0 bytes>) TRACE [2020-11-19 18:15:11] httpcore._sync.connection - create_connection socket= http_version='HTTP/1.1' TRACE [2020-11-19 18:15:11] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_data=Data(<0 bytes>) DEBUG [2020-11-19 18:15:11] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK" TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=Data(<395 bytes>) TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=EndOfMessage(headers=) TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - get_connection_from_pool=(b'https', b'httpbin.org', 443) TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - reusing idle http11 connection= TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - reuse connection= TRACE [2020-11-19 18:15:11] httpcore._sync.connection - connection.request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_request method=b'POST' url=(b'https', b'httpbin.org', None, b'/post') headers=[(b'Host', b'httpbin.org'), (b'Content-Length', b'0'), (b'Accept', b'*/*'), (b'Accept-Encoding', b'gzip, deflate'), (b'Connection', b'keep-alive'), (b'User-Agent', b'python-httpx/0.16.1')] TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - send_data=Data(<0 bytes>) DEBUG [2020-11-19 18:15:11] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK" TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=Data(<395 bytes>) TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=EndOfMessage(headers=) TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE DEBUG [2020-11-19 18:15:11] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK" TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=Data(<395 bytes>) TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=EndOfMessage(headers=) TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE DEBUG [2020-11-19 18:15:11] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK" TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=Data(<395 bytes>) TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=EndOfMessage(headers=) TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE DEBUG [2020-11-19 18:15:11] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK" TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=Data(<395 bytes>) TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - receive_event=EndOfMessage(headers=) TRACE [2020-11-19 18:15:11] httpcore._sync.http11 - response_closed our_state=DONE their_state=DONE Traceback (most recent call last): File "/tmp/httpx-threadpool-demo/main.py", line 29, in main() File "/tmp/httpx-threadpool-demo/main.py", line 24, in main response = future.result() File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result return self.__get_result() File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result raise self._exception File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run result = self.fn(*self.args, **self.kwargs) File "/tmp/httpx-threadpool-demo/main.py", line 13, in do_post response = client.post("post") File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpx/_client.py", line 992, in post return self.request( File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpx/_client.py", line 733, in request return self.send( File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpx/_client.py", line 767, in send response = self._send_handling_auth( File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpx/_client.py", line 805, in _send_handling_auth response = self._send_handling_redirects( File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpx/_client.py", line 837, in _send_handling_redirects response = self._send_single_request(request, timeout) File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpx/_client.py", line 861, in _send_single_request (status_code, headers, stream, ext) = transport.request( File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 218, in request response = connection.request( File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 93, in request self._create_connection(self.socket) File "/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 153, in _create_connection from .http11 import SyncHTTP11Connection ImportError: cannot import name 'SyncHTTP11Connection' from partially initialized module 'httpcore._sync.http11' (most likely due to a circular import) (/tmp/httpx-threadpool-demo/venv/lib/python3.9/site-packages/httpcore/_sync/http11.py) TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - removing connection from pool= TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - removing connection from pool= TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - removing connection from pool= TRACE [2020-11-19 18:15:11] httpcore._sync.connection_pool - removing connection from pool= ```

It looks like it is actually doing requests, but fails afterwards?

Environment

Additional context

I get the same error on the Windows version of Python 3.9 too.

florimondmanca commented 3 years ago

Woah, that's umm, weird.

senpos commented 3 years ago

Hi @florimondmanca!

  1. Yes
  2. Yes, with 4 workers; sometimes with 2-3 workers.
  3. Not sure, sorry. I am not too familiar with it go get an idea if something is related to this

It works fine with requests.

```python from concurrent.futures import ThreadPoolExecutor, as_completed import requests _MAX_WORKERS = 4 _TASKS = 10 _HTTPBIN_BASE_URL = "https://httpbin.org/" def do_post(client: requests.Session): response = client.post(_HTTPBIN_BASE_URL + "post") return response def main(): client = requests.Session() with ThreadPoolExecutor(max_workers=_MAX_WORKERS) as executor: futures = [executor.submit(do_post, client) for _ in range(_TASKS)] for future in as_completed(futures): response = future.result() print(f"{response.status_code=}") if __name__ == "__main__": main() ```
florimondmanca commented 3 years ago

So, it's quite interesting.

I ran your script, and at first it worked fine. Now if I run it again, it fails consistently. If I turn on debug logging I can see it's making some of the requests (generally 7), doesn't show the output, then fails.

If I add a print("getting result") just before the call to future.result(), we can see it's hanging on it, and then we start to see requests terminating, then a crash before the future returns.

$ HTTPX_LOG_LEVEL=debug python debug/bug.py
getting result
DEBUG [2020-11-20 09:30:08] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:08] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:08] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:08] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:09] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:09] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:09] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:09] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
DEBUG [2020-11-20 09:30:09] httpx._client - HTTP Request: POST https://httpbin.org/post "HTTP/1.1 200 OK"
Traceback (most recent call last):
  File "/Users/florimond/Developer/python-projects/httpx/debug/bug.py", line 30, in <module>
    main()
  File "/Users/florimond/Developer/python-projects/httpx/debug/bug.py", line 25, in main
    response = future.result()
  File "/Users/florimond/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/Users/florimond/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/Users/florimond/.pyenv/versions/3.9.0/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/florimond/Developer/python-projects/httpx/debug/bug.py", line 13, in do_post
    response = client.post("post")
  File "/Users/florimond/Developer/python-projects/httpx/httpx/_client.py", line 992, in post
    return self.request(
  File "/Users/florimond/Developer/python-projects/httpx/httpx/_client.py", line 733, in request
    return self.send(
  File "/Users/florimond/Developer/python-projects/httpx/httpx/_client.py", line 767, in send
    response = self._send_handling_auth(
  File "/Users/florimond/Developer/python-projects/httpx/httpx/_client.py", line 805, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/Users/florimond/Developer/python-projects/httpx/httpx/_client.py", line 837, in _send_handling_redirects
    response = self._send_single_request(request, timeout)
  File "/Users/florimond/Developer/python-projects/httpx/httpx/_client.py", line 861, in _send_single_request
    (status_code, headers, stream, ext) = transport.request(
  File "/Users/florimond/Developer/python-projects/httpx/venv/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 200, in request
    response = connection.request(
  File "/Users/florimond/Developer/python-projects/httpx/venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 88, in request
    self._create_connection(self.socket)
  File "/Users/florimond/Developer/python-projects/httpx/venv/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 136, in _create_connection
    from .http11 import SyncHTTP11Connection
ImportError: cannot import name 'SyncHTTP11Connection' from partially initialized module 'httpcore._sync.http11' (most likely due to a circular import) (/Users/florimond/Developer/python-projects/httpx/venv/lib/python3.9/site-packages/httpcore/_sync/http11.py)

Crucially I've tested on 3.8, and I'm not able to reproduce there (even if on the same versions of HTTPX / HTTPCore). So I'm tempted to mark this as a 3.9 bug for now, though it may also be that something changed in 3.9 that revealed a problem we might have with thread-safe execution.

florimondmanca commented 3 years ago

Ah, also a funny bit of info: this only reproduces on http11, not http2. Eg Client(..., http2=True) makes the bug go away. 🤔

florimondmanca commented 3 years ago

This bug likely comes from: https://bugs.python.org/issue35943 - Python 3.9 changed a bug which allowed imports to return partially initialized module.

This discussion ticket relates similar behavior seen with multiprocessing: https://discuss.python.org/t/differences-between-3-8-and-3-9-in-importing-module/5520/2

This discussion^ mentions that "multiprocessing has a lot of local imports, so multiprocessing is not thread-safe". So… That local from .http11 import SyncHTTP11Connection import might be the culprit — I didn't know it was a source of lack of thread-safety.

florimondmanca commented 3 years ago

Yup — moving the local import of SyncHTTP11Connection to a global one (which is fine — we don't need a local import there, since h11 is always installed) fixes it on my side.