fabric / fabric

Simple, Pythonic remote execution and deployment.
http://fabfile.org
BSD 2-Clause "Simplified" License
14.91k stars 1.94k forks source link

Cleanup of forward_local fails when requests fails #2290

Open daniel-falk opened 11 months ago

daniel-falk commented 11 months ago

Describe the bug The cleanup for the forward_local context manager fails if using requests within the context manager and an exception is raised from requests. Even if I catch the exception within the context manager, and the context manager returns normally, a ThreadException is raised in the context managers cleanup. This then leaves the forwarded port in use such that it is not possible to use it again.

To Reproduce Use this code to create a tunnel to a server with a self signed certificate. The requests request will fail to validate the certificate and raise an exception. This exception is caught, the execution continues and then a new exception is raised in the context manager cleanup. The port is never released.

import fabric
import requests
from time import sleep

sshargs = {
    "host": "192.168.0.195",
    "user": "root",
    "connect_kwargs": {
        "password": "root"
    }
}
local_port = 32345

with fabric.Connection(**sshargs).forward_local(local_port, remote_port=443):
   sleep(1) # Wait for the tunnel to start working
   try:
       auth = requests.auth.HTTPDigestAuth(sshargs["user"], sshargs["connect_kwargs"]["password"])
       response = requests.get(
         f"https://127.0.0.1:{local_port}", auth=auth, verify=True
       )
       print("RESPONSE: ", response)
       print(response.text)
   except requests.exceptions.SSLError as e:
       print("GOT ERROR: ", e)
   print("Done")

The output of the command is:

GOT ERROR:  HTTPSConnectionPool(host='127.0.0.1', port=32345): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)')))
Done
Traceback (most recent call last):

  File ~/.pyenv/versions/3.10.13/lib/python3.10/site-packages/spyder_kernels/py3compat.py:356 in compat_exec
    exec(code, globals, locals)

  File ~/.config/spyder-py3/temp.py:19
    with fabric.Connection(**sshargs).forward_local(local_port, remote_port=443):

  File ~/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py:142 in __exit__
    next(self.gen)

  File ~/.pyenv/versions/3.10.13/lib/python3.10/site-packages/fabric/connection.py:1003 in forward_local
    raise wrapper.value

  File ~/.pyenv/versions/3.10.13/lib/python3.10/site-packages/invoke/util.py:209 in run
    self._run()  # type: ignore

  File ~/.pyenv/versions/3.10.13/lib/python3.10/site-packages/fabric/tunnels.py:102 in _run
    raise ThreadException(exceptions)

ThreadException: 
Saw 1 exceptions within threads (ConnectionResetError):

Thread args: {}

Traceback (most recent call last):

  File "/home/danielfa/.pyenv/versions/3.10.13/lib/python3.10/site-packages/invoke/util.py", line 209, in run
    self._run()  # type: ignore

  File "/home/danielfa/.pyenv/versions/3.10.13/lib/python3.10/site-packages/fabric/tunnels.py", line 130, in _run
    empty_sock = self.read_and_write(

  File "/home/danielfa/.pyenv/versions/3.10.13/lib/python3.10/site-packages/fabric/tunnels.py", line 151, in read_and_write
    data = reader.recv(chunk_size)

ConnectionResetError: [Errno 104] Connection reset by peer

Note how "Done" is printed before the exception, so the exception is in the context manager code.

The next time I try to use the port in the same way, I get OSError: [Errno 98] Address already in use.

This does not seem to happen every time, but at least half of the times.

Expected behaviour

Environment I don't think the issue is specific to a narrow environment as we have seen this on multiple Linux environments, but here's what I used to replicate it now:

Python libraries:

daniel-falk commented 11 months ago

To clarify, I think it is related to the fact that requests uses a thread pool to execute the requests, which is not obvious at a first glance.