Open adamjstewart opened 1 year ago
Bonus points if anyone can figure out how to jail our runners so that we can actually test this in CI and prevent it from slipping in again. We seem to fix bugs like this with every release.
Note that many of our slow tests (run only on release branches) are actually designed to test these kinds of dataset and weight downloads. We should be able to get 100% coverage without internet access, but can add additional slow tests to ensure that the weights still download correctly.
Related to #194 (fixed in #265) and #495 (fixed in #497)
@calebrob6 I think the detection tests could use the same treatment from #344, that would greatly speed things up.
~Do you want to add that as an issue so we don't lose track of it, or should we just do it now?~ I guess we can just leave this open?
It can be done at the same time, using a fake detection model and avoiding downloads. Both will speed up the test and make it more stable.
FarSeg tests were failing on main because of this. Would love to get these fixed.
Update: slowly whittling these down:
FAILED tests/models/test_farseg.py::TestFarSeg::test_valid_backbone[resnet18-True] - urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
FAILED tests/models/test_farseg.py::TestFarSeg::test_valid_backbone[resnet50-True] - urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
FAILED tests/trainers/test_detection.py::TestObjectDetectionTask::test_pretrained_backbone - urllib.error.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>
Useful discussion (Linux-only):
Description
Several of our tests require internet access. This is problematic for a number of reasons:
We should apply the same magic we used in previous PRs to monkeypatch any downloads.
Steps to reproduce
rm -r $(python -c 'import torch; print(torch.hub.get_dir())')
pytest
Version
0.5.0.dev0 (ce4e7ef2ed3ecff5731c2902d4cd3e8f5fab753d)