ClementJ18 / moddb

A Python scrapper to access ModDB mods, games and more as objects
https://moddb.readthedocs.io
MIT License
13 stars 8 forks source link

85 test failures (from `-k "not main"`) when offline #42

Closed mgorny closed 7 months ago

mgorny commented 1 year ago

When running the test suite in an offline environment, I'm still getting 38 failures an 47 errors from the baseline set. Some fail outright, while others hang. strace(1) suggests that they are stuck in a sleep() with very long interval — my educated guess is that pyrate-limiter is wrongly assuming that the failures are due to rate limit, and are repeatedly retrying with ever growing delays. I've used pytest-timeout to force them to timeout.

To reproduce:

$ python3.11 -m venv .venv
$ . .venv/bin/activate
# adding pytest-timeout to deal with hanging tests
$ pip install -r requirements.txt -r requirements-dev.txt pytest-timeout
$ sudo unshare -n su - "${USER}"  # or disconnect the network ;-)
# inside the unshared namespace
$ cd <moddb-checkout>
$ . .venv/bin/activate
$ export USERNAME=test PASSWORD=test SENDER_USERNAME=test SENDER_PASSWORD=test
$ python -m pytest -k "not main" --timeout=60 --record-mode=none

Summary:

=========================== short test summary info ============================
FAILED tests/test_base.py::TestLogin::test_login - requests.exceptions.Connec...
FAILED tests/test_base.py::TestLogin::test_bad_login - requests.exceptions.Co...
FAILED tests/test_engine.py::TestEnginePatched::test_get_articles[https://www.moddb.com/engines/sage-strategy-action-game-engine]
FAILED tests/test_engine.py::TestEnginePatched::test_get_games[https://www.moddb.com/engines/sage-strategy-action-game-engine]
FAILED tests/test_engine.py::TestEnginePatched::test_get_tutorials[https://www.moddb.com/engines/sage-strategy-action-game-engine]
FAILED tests/test_game.py::TestGame::test_get_addons[https://www.moddb.com/games/battle-for-middle-earth-ii-rise-of-the-witch-king]
FAILED tests/test_game.py::TestGame::test_get_articles[https://www.moddb.com/games/battle-for-middle-earth-ii-rise-of-the-witch-king]
FAILED tests/test_game.py::TestGame::test_get_files[https://www.moddb.com/games/battle-for-middle-earth-ii-rise-of-the-witch-king]
FAILED tests/test_game.py::TestGame::test_get_mods[https://www.moddb.com/games/battle-for-middle-earth-ii-rise-of-the-witch-king]
FAILED tests/test_game.py::TestGame::test_get_videos[https://www.moddb.com/games/battle-for-middle-earth-ii-rise-of-the-witch-king]
FAILED tests/test_group.py::TestGroup::test_get_articles[https://www.moddb.com/groups/humour-satire-parody]
FAILED tests/test_hardware.py::TestHardware::test_get_articles[https://www.moddb.com/hardware/htc-vive]
FAILED tests/test_hardware.py::TestHardware::test_get_files[https://www.moddb.com/hardware/htc-vive]
FAILED tests/test_hardware.py::TestHardware::test_get_software[https://www.moddb.com/hardware/htc-vive]
FAILED tests/test_hardware.py::TestHardware::test_get_tutorials[https://www.moddb.com/hardware/htc-vive]
FAILED tests/test_member.py::TestMember::test_get_files[https://www.moddb.com/members/upstart]
FAILED tests/test_mod.py::TestMod::test_get_articles[https://www.moddb.com/mods/edain-mod]
FAILED tests/test_mod.py::TestMod::test_get_reviews[https://www.moddb.com/mods/edain-mod]
FAILED tests/test_parse.py::TestParsers::test_parse_addons[https://www.moddb.com/mods/call-of-chernobyl/addons/arsenal-overhaul]
FAILED tests/test_parse.py::TestParsers::test_parse_addons[https://www.moddb.com/games/cc-generals-zero-hour/addons/usa-challenge]
FAILED tests/test_parse.py::TestParsers::test_parse_addons[https://www.moddb.com/games/axis-allies/addons/simple-vanilla-sai-buff]
FAILED tests/test_parse.py::TestParsers::test_parse_medias[https://www.moddb.com/mods/third-age-total-war/videos/rohan7]
FAILED tests/test_parse.py::TestParsers::test_parse_jobs[https://www.moddb.com/jobs/programmer-enviro-artist-audio-specialist-needed-to-expand-our-launched-game]
FAILED tests/test_parse.py::TestParsers::test_parse_jobs[https://www.moddb.com/jobs/looking-for-3d-modeller-for-fast-past-shooting-game1]
FAILED tests/test_parse.py::TestParsers::test_parse_jobs[https://www.moddb.com/jobs/revshareroyalty-3d-character-artist]
FAILED tests/test_parse.py::TestParsers::test_parse_jobs[https://www.moddb.com/jobs/seeking-2d-uiux-artist-3d-weapon-modler-character-animator]
FAILED tests/test_parse.py::TestParsers::test_parse_reviews - Failed: Timeout...
FAILED tests/test_parse.py::TestParsers::test_parse_blogs - Failed: Timeout >...
FAILED tests/test_platform.py::TestPlatform::test_get_games[https://www.moddb.com/platforms/pc]
FAILED tests/test_platform.py::TestPlatform::test_get_hardware[https://www.moddb.com/platforms/pc]
FAILED tests/test_platform.py::TestPlatform::test_get_mods[https://www.moddb.com/platforms/pc]
FAILED tests/test_platform.py::TestPlatform::test_get_software[https://www.moddb.com/platforms/pc]
FAILED tests/test_team.py::TestTeam::test_get_engines[https://www.moddb.com/company/edainmod-dev-team]
FAILED tests/test_team.py::TestTeam::test_get_files[https://www.moddb.com/company/edainmod-dev-team]
FAILED tests/test_team.py::TestTeam::test_get_games[https://www.moddb.com/company/edainmod-dev-team]
FAILED tests/test_team.py::TestTeam::test_get_hardware[https://www.moddb.com/company/edainmod-dev-team]
FAILED tests/test_team.py::TestTeam::test_get_mods[https://www.moddb.com/company/edainmod-dev-team]
FAILED tests/test_team.py::TestTeam::test_get_software[https://www.moddb.com/company/edainmod-dev-team]
ERROR tests/test_client.py::TestClient::test_get_watched[WatchType.mod] - req...
ERROR tests/test_client.py::TestClient::test_get_watched[WatchType.game] - re...
ERROR tests/test_client.py::TestClient::test_get_watched[WatchType.engine] - ...
ERROR tests/test_client.py::TestClient::test_get_watched[WatchType.group] - r...
ERROR tests/test_client.py::TestClient::test_get_watched[WatchType.member] - ...
ERROR tests/test_client.py::TestClient::test_get_updates - requests.exception...
ERROR tests/test_client.py::TestClient::test_posts[https://www.moddb.com/mods/edain-mod]
ERROR tests/test_client.py::TestClient::test_posts[https://www.moddb.com/mods/the-horse-lords-a-total-modification-for-bfme]
ERROR tests/test_client.py::TestClient::test_posts[https://www.moddb.com/mods/third-age-total-war]
ERROR tests/test_client.py::TestClient::test_posts[https://www.moddb.com/games/battle-for-middle-earth-ii-rise-of-the-witch-king]
ERROR tests/test_client.py::TestClient::test_posts[https://www.moddb.com/games/battle-for-middle-earth-ii]
ERROR tests/test_client.py::TestClient::test_posts[https://www.moddb.com/games/battle-for-middle-earth]
ERROR tests/test_client.py::TestClient::test_posts[https://www.moddb.com/engines/sage-strategy-action-game-engine]
ERROR tests/test_client.py::TestClient::test_posts[https://www.moddb.com/engines/source]
ERROR tests/test_client.py::TestClient::test_posts[https://www.moddb.com/engines/cryengine-3]
ERROR tests/test_client.py::TestClient::test_posts[https://www.moddb.com/groups/humour-satire-parody]
ERROR tests/test_client.py::TestClient::test_posts[https://www.moddb.com/groups/warhammer-40k-fans-group]
ERROR tests/test_client.py::TestClient::test_posts[https://www.moddb.com/members/barneypan]
ERROR tests/test_client.py::TestClient::test_posts[https://www.moddb.com/members/officialnecro]
ERROR tests/test_client.py::TestClient::test_friends - requests.exceptions.Co...
ERROR tests/test_client.py::TestClient::test_messages - requests.exceptions.C...
ERROR tests/test_engine.py::TestEnginePatched::test_get_videos[https://www.moddb.com/engines/sage-strategy-action-game-engine]
ERROR tests/test_engine.py::TestEnginePatched::test_get_watchers[https://www.moddb.com/engines/sage-strategy-action-game-engine]
ERROR tests/test_engine.py::TestEngine::test_get_tags[https://www.moddb.com/engines/sage-strategy-action-game-engine]
ERROR tests/test_game.py::TestGame::test_get_watchers[https://www.moddb.com/games/battle-for-middle-earth-ii-rise-of-the-witch-king]
ERROR tests/test_game.py::TestGame::test_get_tags[https://www.moddb.com/games/battle-for-middle-earth-ii-rise-of-the-witch-king]
ERROR tests/test_group.py::TestGroup::test_get_addons[https://www.moddb.com/groups/humour-satire-parody]
ERROR tests/test_hardware.py::TestHardware::test_get_games[https://www.moddb.com/hardware/htc-vive]
ERROR tests/test_hardware.py::TestHardware::test_get_hardware[https://www.moddb.com/hardware/htc-vive]
ERROR tests/test_hardware.py::TestHardware::test_get_images[https://www.moddb.com/hardware/htc-vive]
ERROR tests/test_hardware.py::TestHardware::test_get_reviews[https://www.moddb.com/hardware/htc-vive]
ERROR tests/test_member.py::TestMember::test_get_friends[https://www.moddb.com/members/upstart]
ERROR tests/test_member.py::TestMember::test_get_games[https://www.moddb.com/members/upstart]
ERROR tests/test_member.py::TestMember::test_get_groups[https://www.moddb.com/members/upstart]
ERROR tests/test_mod.py::TestMod::test_get_tutorials[https://www.moddb.com/mods/edain-mod]
ERROR tests/test_mod.py::TestMod::test_get_videos[https://www.moddb.com/mods/edain-mod]
ERROR tests/test_mod.py::TestMod::test_get_watchers[https://www.moddb.com/mods/edain-mod]
ERROR tests/test_platform.py::TestPlatform::test_get_comments[https://www.moddb.com/platforms/pc]
ERROR tests/test_platform.py::TestPlatform::test_get_engines[https://www.moddb.com/platforms/pc]
ERROR tests/test_software.py::TestSoftware::test_get_videos[https://www.moddb.com/software/project-neptune-vr]
ERROR tests/test_software.py::TestSoftware::test_get_watchers[https://www.moddb.com/software/project-neptune-vr]
ERROR tests/test_software.py::TestSoftware::test_get_tags[https://www.moddb.com/software/project-neptune-vr]
ERROR tests/test_team.py::TestTeam::test_get_addons[https://www.moddb.com/company/edainmod-dev-team]
ERROR tests/test_team.py::TestTeam::test_get_tutorials[https://www.moddb.com/company/edainmod-dev-team]
ERROR tests/test_team.py::TestTeam::test_get_videos[https://www.moddb.com/company/edainmod-dev-team]
ERROR tests/test_team.py::TestTeam::test_get_watchers[https://www.moddb.com/company/edainmod-dev-team]
ERROR tests/test_team.py::TestTeam::test_get_tags[https://www.moddb.com/company/edainmod-dev-team]
==== 38 failed, 167 passed, 721 deselected, 47 errors in 2995.70s (0:49:55) ====

Test log (1.1M): log.txt

ClementJ18 commented 1 year ago

The test_client and TestLoging errors make sense. Those don't currently have cassettes because I still need to fix some stuff in regard to those tests. The rest are odd, I will test them again with this exact parameter.

ClementJ18 commented 7 months ago

The issue was that very long ago I set the tests to have some randomizing elements to the way the tests are ran to allow for an overall larger coverage without increasing the time of individual runs. This was causing the failures. I've removed that bit of code in the next release. All tests that do not require a logged in user should now be able to run without a connection using only the cassettes.