Docker client failing to connect : requests.exceptions.InvalidURL: Not supported URL scheme http+docker

[SahilChaudhari]

Running Docker inside chroot results in "requests.exceptions.InvalidURL: Not supported URL scheme http+docker" due to Docker client failing to connect to Docker daemon.

The workspace was working fine till 20th May, On 21st May build started failing with below issue.

Steps to reproduce the issue:

1.export MIRROR_SNAPSHOT=y && NOJESSIE=1 NOSTRETCH=1 make target/sonic-pensando.tar

+ sudo cp files/build_templates/docker_image_ctl.j2 ./fsroot-pensando/usr/share/sonic/templates/docker_image_ctl.j2
+ sudo LANG=C DOCKER_HOST= chroot ./fsroot-pensando /usr/local/bin/generate_shutdown_order.py
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 532, in send
    conn = self._get_connection(request, verify, proxies=proxies, cert=cert)
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 400, in _get_connection
    conn = self.poolmanager.connection_from_host(
  File "/usr/local/lib/python3.9/dist-packages/urllib3/poolmanager.py", line 304, in connection_from_host
    return self.connection_from_context(request_context)
  File "/usr/local/lib/python3.9/dist-packages/urllib3/poolmanager.py", line 326, in connection_from_context
    raise URLSchemeUnknown(scheme)
urllib3.exceptions.URLSchemeUnknown: Not supported URL scheme http+docker

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/usr/local/lib/python3.9/dist-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/usr/local/lib/python3.9/dist-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 534, in send
    raise InvalidURL(e, request=request)
requests.exceptions.InvalidURL: Not supported URL scheme http+docker

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/generate_shutdown_order.py", line 15, in <module>
    main()
  File "/usr/local/bin/generate_shutdown_order.py", line 8, in main
    manager = PackageManager.get_manager()
  File "/usr/local/lib/python3.9/dist-packages/sonic_package_manager/manager.py", line 1003, in get_manager
    docker_api = DockerApi(docker.from_env(), ProgressManager())
  File "/usr/local/lib/python3.9/dist-packages/docker/client.py", line 96, in from_env
    return cls(
  File "/usr/local/lib/python3.9/dist-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/usr/local/lib/python3.9/dist-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: Not supported URL scheme http+docker
+ clean_sys
+ sudo chroot ./fsroot-pensando umount /sys/fs/cgroup/blkio /sys/fs/cgroup/cpu /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/cpuacct /sys/fs/cgroup/cpuset /sys/fs/cgroup/devices /sys/fs/cgroup/freezer /sys/fs/cgroup/hugetlb /sys/fs/cgroup/memory /sys/fs/cgroup/net_cls /sys/fs/cgroup/net_cls,net_prio /sys/fs/cgroup/net_prio /sys/fs/cgroup/perf_event /sys/fs/cgroup/pids /sys/fs/cgroup/rdma /sys/fs/cgroup/systemd /sys/fs/cgroup /sys
umount: /sys/fs/cgroup/cpu: no mount point specified.
umount: /sys/fs/cgroup/cpu,cpuacct: no mount point specified.
umount: /sys/fs/cgroup/cpuacct: no mount point specified.
umount: /sys/fs/cgroup/net_cls: no mount point specified.
umount: /sys/fs/cgroup/net_cls,net_prio: no mount point specified.
umount: /sys/fs/cgroup/net_prio: no mount point specified.
umount: /sys/fs/cgroup/systemd: no mount point specified.
+ true
+ true
+ sudo LANG=C chroot ./fsroot-pensando umount /proc
+ true
[  FAIL LOG END  ] [ target/sonic-pensando.tar ]
make: *** [slave.mk:1226: target/sonic-pensando.tar] Error 1
make[1]: *** [Makefile.work:542: target/sonic-pensando.tar] Error 2
make[1]: Leaving directory '/home/sonic/ssdk/sonic-buildimage'
make: *** [Makefile:41: target/sonic-pensando.tar] Error 2

can someone please help?

[praveen-ramamoorthi]

This was due to a recent break https://github.com/docker/docker-py/issues/3256.

Please use the following temporary workaround to proceed with the build. Add pip install requests package and pin version to <2.32.0 in build_debian.sh

sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install 'docker==6.1.1' sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install 'requests<2.32.0'

[SahilChaudhari]

Thanks it worked for me.

[dgsudharsan]

This needs to be fixed in code. Reopening the ticket until sonic code is modified

[gechiang]

A PR is raised to fix this already but failing PR test. submitter is working on resolving the test issue. https://github.com/sonic-net/sonic-utilities/pull/3328

[k-v1]

BTW, requests is also installed in docker-sonic-mgmt-framework. https://github.com/sonic-net/sonic-buildimage/blob/7d7332a6448c53856978db845685892507d27abe/dockers/docker-sonic-mgmt-framework/Dockerfile.j2#L11 Not sure, should we also use version 2.31.0 here or not.

[jusherma]

I see the fix committed to sonic-net/sonic-utilities, but I don't see sonic-buildimage pointing to a version of sonic-utilities that has the fix. Any ETA on when this will be fixed?

[gechiang]

I see the fix committed to sonic-net/sonic-utilities, but I don't see sonic-buildimage pointing to a version of sonic-utilities that has the fix. Any ETA on when this will be fixed?

I see there is one outstanding PR for sonic-utilities submodule update : https://github.com/sonic-net/sonic-buildimage/pull/18828 Once that gets in, or it failed and it forces to pick up the next head which will include the fix you wanted...

[jusherma]

I see there is one outstanding PR for sonic-utilities submodule update : #18828 Once that gets in, or it failed and it forces to pick up the next head which will include the fix you wanted...

Thanks. Since this is for a build failure, will this be backported to 202305, 202311, 202405?

[gechiang]

I see there is one outstanding PR for sonic-utilities submodule update : #18828 Once that gets in, or it failed and it forces to pick up the next head which will include the fix you wanted...

Thanks. Since this is for a build failure, will this be backported to 202305, 202311, 202405?

Please post your question to the PR and ask the Branch owner for 202305/202311 to see if this can be back ported to those branches. 202405 is not yet cut so it will have it for sure.

[jusherma]

Instead of having the fix in sonic-utilities, why not commit the above proposed fix directly to build_debian.sh? Then we don't need to wait on updates to submodule pointers, etc.

diff --git a/build_debian.sh b/build_debian.sh
index 6e80ab5cb..5d596e9a6 100755
--- a/build_debian.sh
+++ b/build_debian.sh
@@ -554,6 +554,7 @@ sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install 'wheel

 # docker Python API package is needed by Ansible docker module as well as some SONiC applications
 sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install 'docker==6.1.1'
+sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install 'requests<2.32.0'

[volodymyrsamotiy]

@yxieca, the same build issue is observed on 202205 branch, could you please help to check if we need to cherry-pick this fix to 202205 - https://github.com/sonic-net/sonic-utilities/pull/3328?

[collivier]

Instead of having the fix in sonic-utilities, why not commit the above proposed fix directly to build_debian.sh? Then we don't need to wait on updates to submodule pointers, etc.

diff --git a/build_debian.sh b/build_debian.sh
index 6e80ab5cb..5d596e9a6 100755
--- a/build_debian.sh
+++ b/build_debian.sh
@@ -554,6 +554,7 @@ sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install 'wheel

 # docker Python API package is needed by Ansible docker module as well as some SONiC applications
 sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install 'docker==6.1.1'
+sudo https_proxy=$https_proxy LANG=C chroot $FILESYSTEM_ROOT pip3 install 'requests<2.32.0'

Not really. Just adding an upper bound to requests may fix this case but the problem would quickly reappear because of the clear lack of pinning the indirect dependencies installed by pip. To avoid these common issues related to pip, I would highly recommend the use of upper-constraints as OpenStack (or OPNFV) has done for years.

I just sent an email to the mailing lists, but it doesn't look like broadcasted yet (@yxieca is there a manual mail validation ?)

[collivier]

I see the fix committed to sonic-net/sonic-utilities, but I don't see sonic-buildimage pointing to a version of sonic-utilities that has the fix. Any ETA on when this will be fixed?

I see there is one outstanding PR for sonic-utilities submodule update : #18828 Once that gets in, or it failed and it forces to pick up the next head which will include the fix you wanted...

same comments as https://github.com/sonic-net/sonic-buildimage/issues/19028#issuecomment-2144617202

[saiarcot895]

I've opened #19181 which upgrades docker to 7.1.0, which has the fix for requests 2.32.0. This should fix this issue fully.