googleapis / python-spanner

Apache License 2.0
134 stars 86 forks source link

testing: unit tests are slow, noisy and make network calls making it very hard to develop offline #1208

Open odeke-em opened 3 weeks ago

odeke-em commented 3 weeks ago

It is quite cumbersome and slow developing for this library because even just the unit tests require a bulky setup of fetching dependencies afresh every single time and even worse makes it impossible to properly test. Unit tests should be hermetic in most cases but the experience here is very noisy producing lots of output then requires network calls too.

Environment details

Steps to reproduce

nox -s unit-3.7

Stack trace

nox > Running session unit-3.7(protobuf_implementation='python')
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/unit-3-7-protobuf_implementation-python
nox > python -m pip install mock asyncmock pytest pytest-cov pytest-asyncio -c /Users/emmanuelodeke/go/src/github.com/googleapis/python-spanner/testing/constraints-3.7.txt
nox > Command python -m pip install mock asyncmock pytest pytest-cov pytest-asyncio -c /Users/emmanuelodeke/go/src/github.com/googleapis/python-spanner/testing/constraints-3.7.txt failed with exit code 1:
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fb1705318d0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/mock/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fb16e2f9190>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/mock/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fb16e2f99d0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/mock/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fb16e2f9e90>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/mock/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fb16e3172d0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known')': /simple/mock/
ERROR: Could not find a version that satisfies the requirement mock (from versions: none)
ERROR: No matching distribution found for mock
nox > Session unit-3.7(protobuf_implementation='python') failed.
nox > Running session unit-3.7(protobuf_implementation='upb')
nox > Creating virtual environment (virtualenv) using python3.7 in .nox/unit-3-7-protobuf_implementation-upb
nox > python -m pip install mock asyncmock pytest pytest-cov pytest-asyncio -c /Users/emmanuelodeke/go/src/github.com/googleapis/python-spanner/testing/constraints-3.7.txt
^Cnox > Interrupted...
nox > Session unit-3.7(protobuf_implementation='upb') interrupted.

/cc @harshachinta @surbhigarg92

odeke-em commented 1 week ago

Also whenever I run the tests locally, I consistently get these unrelated but failing tests even on a fresh clone

=========================== short test summary info ============================
FAILED tests/unit/test_client.py::TestClient::test_constructor_credentials_wo_create_scoped - AssertionError: <google.auth.credentials.AnonymousCredentials object at 0x1...
FAILED tests/unit/test_client.py::TestClient::test_constructor_custom_client_info - AssertionError: <google.auth.credentials.AnonymousCredentials object at 0x1...
FAILED tests/unit/test_client.py::TestClient::test_constructor_custom_client_options_dict - AssertionError: <google.auth.credentials.AnonymousCredentials object at 0x1...
FAILED tests/unit/test_client.py::TestClient::test_constructor_custom_client_options_obj - AssertionError: <google.auth.credentials.AnonymousCredentials object at 0x1...
FAILED tests/unit/test_client.py::TestClient::test_constructor_custom_query_options_client_config - AssertionError: <google.auth.credentials.AnonymousCredentials object at 0x1...
FAILED tests/unit/test_client.py::TestClient::test_constructor_custom_query_options_env_config - AssertionError: <google.auth.credentials.AnonymousCredentials object at 0x1...
FAILED tests/unit/test_client.py::TestClient::test_constructor_default_scopes - AssertionError: <google.auth.credentials.AnonymousCredentials object at 0x1...
FAILED tests/unit/test_client.py::TestClient::test_constructor_implicit_credentials - AssertionError: <google.auth.credentials.AnonymousCredentials object at 0x1...
FAILED tests/unit/test_client.py::TestClient::test_constructor_route_to_leader_disbled - AssertionError: <google.auth.credentials.AnonymousCredentials object at 0x1...
FAILED tests/unit/test_client.py::TestClient::test_constructor_w_directed_read_options - AssertionError: <google.auth.credentials.AnonymousCredentials object at 0x1...
FAILED tests/unit/test_client.py::TestClient::test_copy - AssertionError: <google.auth.credentials.AnonymousCredentials object at 0x1...
FAILED tests/unit/test_client.py::TestClient::test_credentials_property - AssertionError: <google.auth.credentials.AnonymousCredentials object at 0x1...