Unable to use httpx backend because of missing SyncConnectionPool in httpcore

[anomit]

I have had a successful installation of IPFS-Toolkit that has the following versions:

Successfully installed IPFS-Toolkit-0.3.1 httpcore-0.14.7 httpx-0.22.0

When I attempt to establish a connection with a reusable connection pool....

from IPFS_API import ipfshttpclient

ipfs_client = ipfshttpclient.connect(addr='/ip4/172.xx.xx.xx/tcp/5001', session=True)

I get the following exception:

Traceback (most recent call last):
  File "test_ipfs_http_api.py", line 9, in <module>
    ipfs_client = ipfshttpclient.connect(addr='/ip4/172.xx.xx.xx/tcp/5001', session=True)
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/ipfshttpclient2/client/__init__.py", line 119, in connect
    client = Client(
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/ipfshttpclient2/client/base.py", line 406, in __init__
    self._client.open_session()
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/ipfshttpclient2/http_common.py", line 429, in open_session
    self._session = self._make_session()
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/ipfshttpclient2/http_httpx.py", line 119, in _make_session
    connection_pool = httpcore.SyncConnectionPool(
AttributeError: module 'httpcore' has no attribute 'SyncConnectionPool'

Taking a first hand look at a few StackOverflow threads where developers run into quite a few missing attribute exceptions regarding httpcore, it seems like putting a version freeze on the httpx and httpcore requirements would be a good idea here.

[emendir]

Here is another possible cause for the problem: I've had trouble before with ipfshttpclient not able to choose the correct httpx/httpcore library on certain operating systems/virtual environments.

As far as I understand it, some environments require the use of the httpx library, others the httpcore instead. ipfshttpclient tries to work out which library it should use during runtime, but doesn't always make the correct choice and then runs into a misleading error.

Could you therefore report what operating system and python version you're using?

[emendir]

If you would like to test your hypothesis about the cause of the problem being the versions of the httpx and httpcore libraries, here are the versions I am currently using which work for me on Ubuntu 20 with python3.8.10: httpx: 0.20.0 httpcore: 0.13.7 If you face no issues when you install those versions of the libraries, your hypothesis will be correct and I would implement the dependency version freeze right away. Otherwise, we'll have to keep troubleshooting.

[anomit]

As far as I understand it, some environments require the use of the httpx library, others the httpcore instead. ipfshttpclient tries to work out which library it should use during runtime,

I forgot to mention this issue arises when I explicitly wish to switch to the httpx backend instead of requests by passing the following env flag. I believe you meant that. httpcore is used by httpx, and not requests.

export PY_IPFS_HTTP_CLIENT_PREFER_HTTPX=yes; python test_ipfs_http_api.py

This issue arises for me on

Ubuntu 20.04.3 LTS (GNU/Linux 5.11.0-1022-aws x86_64) Python 3.8.7 in a virtualenv Installed versions: IPFS-Toolkit-0.3.1, httpcore-0.14.7, httpx-0.22.0

I dug in a little into the present master tree for httpcore, and I found these imports exposed. Maybe the SyncConnectionPool is named as just ConnectionPool now?

from ._sync import (
    ConnectionInterface,
    ConnectionPool,
    HTTP2Connection,
    HTTP11Connection,
    HTTPConnection,
    HTTPProxy,
    SOCKSProxy,
)

I did a clean install, purged pip cache, and explicitly specified versions pip install httpx==0.20.0 httpcore==0.13.7. It just resulted in a new set of exceptions.

Traceback (most recent call last):
  File "test_ipfs_http_api.py", line 9, in <module>
    ipfs_client = ipfshttpclient.connect(addr='/ip4/172.xx.xx.xx/tcp/5001', session=False)
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/ipfshttpclient2/client/__init__.py", line 127, in connect
    assert_version(client.apply_workarounds()["Version"])
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/ipfshttpclient2/client/__init__.py", line 237, in apply_workarounds
    version_info = self.version()
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/ipfshttpclient2/client/base.py", line 229, in wrapper2
    result = func(*args, **kwargs)
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/ipfshttpclient2/client/miscellaneous.py", line 204, in version
    return self._client.request('/version', decoder='json', **kwargs)
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/ipfshttpclient2/http_common.py", line 583, in request
    closables, res = self._request(
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/ipfshttpclient2/http_httpx.py", line 177, in _request
    res: httpx.Response = session.stream(
  File "/home/ubuntu/.pyenv/versions/3.8.7/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/httpx/_client.py", line 833, in stream
    response = self.send(
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/httpx/_client.py", line 877, in send
    response = self._send_handling_auth(
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/httpx/_client.py", line 905, in _send_handling_auth
    response = self._send_handling_redirects(
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/httpx/_client.py", line 942, in _send_handling_redirects
    response = self._send_single_request(request)
  File "/home/ubuntu/.pyenv/versions/auditprotocol38/lib/python3.8/site-packages/httpx/_client.py", line 978, in _send_single_request
    response = transport.handle_request(request)
TypeError: handle_request() missing 4 required positional arguments: 'url', 'headers', 'stream', and 'extensions'

[emendir]

Thanks for the details, I managed to replicate your steps with the same results.

Right, so using slightly older versions of the httpx and httpcore libs didn't help. I suppose we could test a range of different versions in the hope of finding one the works, and putting a version freeze on those libs then, or we could the ipfshttpclient code to work with the latest versions of those libs. The latter sounds more appealing.

The issue is in the ipfshttpclient2 module, which is an modified version of the ipfshttpclient library. It would be best to fix the issue there, because I don't really want my ipfshttpclient2 to be a fork which people start using because the official ipfshttpclient hasn't been updated in almost a year. That would be missing the point of open-source work.

Therefore I recommend you work on fixing this issue not in the context of this IPFS-Toolkit project, but of ipfshttpclient library. Let me know as soon as you do and I'll integrate the fix into IPFS-Toolkit.

PS: I did a pull request there once but it didn't go through cause of some automated test fails whose report I didn't really understand. I'll have to learn how to do that properly soon, then I can get rid of the ipfshttpclient2 module in my project and import the official ipfshttpclient module.