Make test suite independent of .git directory

yzgyyang commented 3 years ago

By simply removing the .git directory, tests will begin to fail (I got "12 failed, 38 passed, 20 skipped in 3.19s"). It seems like the suite is dependent on the .git directory of the source.

=============================================== short test summary info ================================================
FAILED tests/test.py::TestUploader::test_bowerrc - AttributeError: 'FileNotFoundError' object has no attribute 'retur...
FAILED tests/test.py::TestUploader::test_bowerrc_none - AttributeError: 'FileNotFoundError' object has no attribute '...
FAILED tests/test.py::TestUploader::test_disable_search - AttributeError: 'FileNotFoundError' object has no attribute...
FAILED tests/test.py::TestUploader::test_discovers - AttributeError: 'FileNotFoundError' object has no attribute 'ret...
FAILED tests/test.py::TestUploader::test_include_env - AttributeError: 'FileNotFoundError' object has no attribute 'r...
FAILED tests/test.py::TestUploader::test_none_found - AttributeError: 'FileNotFoundError' object has no attribute 're...
FAILED tests/test.py::TestUploader::test_not_jacoco - AttributeError: 'FileNotFoundError' object has no attribute 're...
FAILED tests/test.py::TestUploader::test_prefix - AttributeError: 'FileNotFoundError' object has no attribute 'return...
FAILED tests/test.py::TestUploader::test_read_token_file - AttributeError: 'FileNotFoundError' object has no attribut...
FAILED tests/test.py::TestUploader::test_returns_none - SystemExit: 0
FAILED tests/test.py::TestUploader::test_run_coverage_fails - AttributeError: 'FileNotFoundError' object has no attri...
FAILED tests/test.py::TestUploader::test_send - AttributeError: 'FileNotFoundError' object has no attribute 'returncode'
================================= 12 failed, 38 passed, 20 skipped, 1 warning in 8.09s =================================

Use Case: This happens if one downloads a release tarball (which doesn't include .git) and try to run the test suite. The situation happens when we try to include codecov in the FreeBSD Ports Collection - according to the FreeBSD Policy [1]: Python ports must include a do-test target if the package includes tests. We are currently removing failed tests manually, but would love to have this issue fixed upstream, so that the test suite will no longer depend on the .git directory.

This is essentially the same problem reported in #100.

[1] https://wiki.freebsd.org/Python/PortsPolicy#Tests

thomasrockhu commented 3 years ago

@yzgyyang thanks for this. Do you happen to be running this in a public repository, and if so, can you link to a CI build?

yzgyyang commented 3 years ago

@yzgyyang thanks for this. Do you happen to be running this in a public repository, and if so, can you link to a CI build?

I don't have the CI build results on hand, but essentially by the following steps:

Download and unzip https://github.com/codecov/codecov-python/archive/v2.1.10.zip
Install test dependencies and run pytest tests/test.py (Since releases don't ship with the .git directory)

Or:

Clone the repo
Remove .git directory
Install test dependencies and run pytest tests/test.py

Since I am trying to commit the latest version to FreeBSD Ports 1, I am essentially using a patch to disable all tests that seem to fail when removed .git. 2

Haven't really dig into the tests, but IMO the tests shouldn't be dependent on the actual .git directory of the repo (maybe using a mocked one that ships with the test suite instead?).

codecov / codecov-python

Make test suite independent of .git directory #304