search

pytest-dev / pytest-xdist

pytest plugin for distributed testing and loop-on-failures testing modes.
https://pytest-xdist.readthedocs.io
MIT License
1.46k stars 232 forks source link

Network connectitivity issues with xdist #940

Open chrysle opened 1 year ago

chrysle commented 1 year ago

I am trying to run a subset of tests with pytest of which many require an internet connection, to install packages from pip.

I have verified the test suite passes, and enabled test parallelization using pytest-xdist. I am using auto-detection for the number of processes. Now I experience that various tests are failing because of broken internet connection. An extract from a log file containing the pip error:

PIP STDOUT
----------
Looking in indexes: http://username:****@127.0.0.1:8080/simple
Collecting zest-releaser==7.1.0
Downloading http://127.0.0.1:8080/packages/zest.releaser-7.1.0-py3-none-any.whl (93 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 93.7/93.7 kB 250.7 MB/s eta 0:00:00
Requirement already satisfied: setuptools in /tmp/pytest-of-user/pytest-0/popen-gw3/session_shareddir0/lib/python3.10/site-packages (from zest-releaser==7.1.0) (65.6.3)
Collecting requests
Downloading http://127.0.0.1:8080/packages/requests-2.28.1-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 207.4 MB/s eta 0:00:00
Collecting colorama
Downloading http://127.0.0.1:8080/packages/colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting twine>=1.6.0
Downloading http://127.0.0.1:8080/packages/twine-4.0.2-py3-none-any.whl (36 kB)
Collecting urllib3>=1.26.0
Downloading http://127.0.0.1:8080/packages/urllib3-1.26.13-py2.py3-none-any.whl (140 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.6/140.6 kB 229.1 MB/s eta 0:00:00
Collecting importlib-metadata>=3.6
Downloading http://127.0.0.1:8080/packages/importlib_metadata-5.1.0-py3-none-any.whl (21 kB)
Collecting keyring>=15.1
Downloading http://127.0.0.1:8080/packages/keyring-23.11.0-py3-none-any.whl (36 kB)
Collecting rich>=12.0.0
Downloading http://127.0.0.1:8080/packages/rich-12.6.0-py3-none-any.whl (237 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 237.5/237.5 kB 277.7 MB/s eta 0:00:00
Collecting requests-toolbelt!=0.9.0,>=0.8.0
Downloading http://127.0.0.1:8080/packages/requests_toolbelt-0.10.1-py2.py3-none-any.whl (54 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.5/54.5 kB 132.3 MB/s eta 0:00:00
Collecting readme-renderer>=35.0
Downloading http://127.0.0.1:8080/packages/readme_renderer-37.3-py3-none-any.whl (14 kB)
Collecting rfc3986>=1.4.0
Downloading http://127.0.0.1:8080/packages/rfc3986-2.0.0-py2.py3-none-any.whl (31 kB)
Collecting pkginfo>=1.8.1
Downloading http://127.0.0.1:8080/packages/pkginfo-1.9.2-py3-none-any.whl (26 kB)
Collecting idna<4,>=2.5
Downloading http://127.0.0.1:8080/packages/idna-3.4-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 107.2 MB/s eta 0:00:00
Collecting charset-normalizer<3,>=2
Downloading http://127.0.0.1:8080/packages/charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting certifi>=2017.4.17
Downloading http://127.0.0.1:8080/packages/certifi-2022.9.24-py3-none-any.whl (161 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 161.1/161.1 kB 265.6 MB/s eta 0:00:00
Collecting zipp>=0.5
Downloading http://127.0.0.1:8080/packages/zipp-3.11.0-py3-none-any.whl (6.6 kB)
Collecting jeepney>=0.4.2
Downloading http://127.0.0.1:8080/packages/jeepney-0.8.0-py3-none-any.whl (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.4/48.4 kB 108.2 MB/s eta 0:00:00
Collecting jaraco.classes
Downloading http://127.0.0.1:8080/packages/jaraco.classes-3.2.3-py3-none-any.whl (6.0 kB)
Collecting SecretStorage>=3.2
Downloading http://127.0.0.1:8080/packages/SecretStorage-3.3.3-py3-none-any.whl (15 kB)
Collecting bleach>=2.1.0
Downloading http://127.0.0.1:8080/packages/bleach-5.0.1-py3-none-any.whl (160 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 160.9/160.9 kB 169.0 MB/s eta 0:00:00
Collecting Pygments>=2.5.1
Downloading http://127.0.0.1:8080/packages/Pygments-2.13.0-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 170.9 MB/s eta 0:00:00
Collecting docutils>=0.13.1
Downloading http://127.0.0.1:8080/packages/docutils-0.19-py3-none-any.whl (570 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 570.5/570.5 kB 124.7 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of rfc3986 to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of requests-toolbelt to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of readme-renderer to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of pkginfo to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of keyring to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of importlib-metadata to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of idna to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of charset-normalizer to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of certifi to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of colorama to determine which version is compatible with other requirements. This could take a while.
Collecting colorama

PIP STDERR
----------
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))': /simple/commonmark/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b878b80>: Failed to establish a new connection: [Errno 111] Connection refused')': /simple/commonmark/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b878d90>: Failed to establish a new connection: [Errno 111] Connection refused')': /simple/commonmark/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b878fa0>: Failed to establish a new connection: [Errno 111] Connection refused')': /simple/commonmark/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b840c70>: Failed to establish a new connection: [Errno 111] Connection refused')': /simple/commonmark/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b7f7fa0>: Failed to establish a new connection: [Errno 111] Connection refused')': /packages/colorama-0.4.3-py2.py3-none-any.whl
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b854970>: Failed to establish a new connection: [Errno 111] Connection refused')': /packages/colorama-0.4.3-py2.py3-none-any.whl
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b856740>: Failed to establish a new connection: [Errno 111] Connection refused')': /packages/colorama-0.4.3-py2.py3-none-any.whl
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b855cc0>: Failed to establish a new connection: [Errno 111] Connection refused')': /packages/colorama-0.4.3-py2.py3-none-any.whl
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b8560e0>: Failed to establish a new connection: [Errno 111] Connection refused')': /packages/colorama-0.4.3-py2.py3-none-any.whl
ERROR: Could not install packages due to an OSError: HTTPConnectionPool(host='127.0.0.1', port=8080): Max retries exceeded with url: /packages/colorama-0.4.3-py2.py3-none-any.whl (Caused by NewConnectionError('<pip._vendor.urllib3.connection.HTTPConnection object at 0x7fb91b857fd0>: Failed to establish a new connection: [Errno 111] Connection refused'))
RonnyPfannschmidt commented 1 year ago

Are resource Limits in place on your setup,this Looks like ddos mitigation triggering on parallel tests

Pip is usually a subprocess,so it seems safe to assume the connection back pressure is related