mmf pytest failed

[AbSaifi]

Steps :

1. Latest source from git clone https://github.com/facebookresearch/mmf.git --> Success
2. Go to mmf source folder --> Success
3. pip install --editable . --> Success
4. pytest ./tests/ --> Fail

Result --> 8 tests failed, 57 passed, 18 warnings and 2 error. Problem can be seen majorly as AttributeError: module 'torch.distributed' has no attribute 'is_nccl_available' E AssertionError: '0ms' != '000ms' E - 0ms E + 000ms E ? ++

O/p of pytest :

pytest ./tests/ ============================================================================================== test session starts =============================================================================================== platform win32 -- Python 3.7.4, pytest-5.2.1, py-1.8.0, pluggy-0.13.0 rootdir: F:\Python\DD_HM\mmf plugins: arraydiff-0.3, doctestplus-0.4.0, openfiles-0.4.0, remotedata-0.3.2 collected 67 items

tests\common\test_sample.py ... [ 4%] tests\configs\test_configs_for_keys.py .. [ 7%] tests\configs\test_zoo_urls.py . [ 8%] tests\datasets\test_base_dataset.py . [ 10%] tests\datasets\test_processors.py ... [ 14%] tests\models\test_cnn_lstm.py . [ 16%] tests\models\interfaces\test_interfaces.py F [ 17%] tests\modules\test_fusions.py .......... [ 32%] tests\modules\test_layers.py ..... [ 40%] tests\modules\test_losses.py .. [ 43%] tests\modules\test_metrics.py .......... [ 58%] tests\utils\test_checkpoint.py FFFFF.. [ 68%] tests\utils\test_configuration.py . [ 70%] tests\utils\test_distributed.py . [ 71%] tests\utils\test_download.py ... [ 76%] tests\utils\test_file_io.py .... [ 82%] tests\utils\test_general.py .. [ 85%] tests\utils\test_logger.py EE [ 88%] tests\utils\test_text.py ..... [ 95%] tests\utils\test_timer.py F.F [100%]

===================================================================================================== ERRORS ===================================================================================================== __ ERROR at setup of TestLogger.test_log_writer __

cls = <class 'tests.utils.test_logger.TestLogger'>

@classmethod
def setUpClass(cls) -> None:
    cls._tmpdir = tempfile.mkdtemp()
    args = argparse.Namespace()
    args.opts = [f"env.save_dir={cls._tmpdir}", f"model=cnn_lstm", f"dataset=clevr"]
    args.config_override = None
    configuration = Configuration(args)
    configuration.freeze()
    cls.config = configuration.get_config()
    registry.register("config", cls.config)

  cls.writer = Logger(cls.config)

tests\utils\test_logger.py:31:

---

mmf\utils\logger.py:19: in init self._is_master = is_master() mmf\utils\distributed.py:39: in is_master return get_rank() == 0

---

def get_rank():

  if not dist.is_nccl_available():

E AttributeError: module 'torch.distributed' has no attribute 'is_nccl_available'

mmf\utils\distributed.py:31: AttributeError --------------------------------------------------------------------------------------------- Captured stdout setup ---------------------------------------------------------------------------------------------- Overriding option env.save_dir to C:\Users\ABDURR~1\AppData\Local\Temp\tmpjpb0tpp3 Overriding option model to cnnlstm Overriding option datasets to clevr ____ ERROR at setup of TestLogger.test_loggerfiles ____

cls = <class 'tests.utils.test_logger.TestLogger'>

@classmethod
def setUpClass(cls) -> None:
    cls._tmpdir = tempfile.mkdtemp()
    args = argparse.Namespace()
    args.opts = [f"env.save_dir={cls._tmpdir}", f"model=cnn_lstm", f"dataset=clevr"]
    args.config_override = None
    configuration = Configuration(args)
    configuration.freeze()
    cls.config = configuration.get_config()
    registry.register("config", cls.config)

  cls.writer = Logger(cls.config)

tests\utils\test_logger.py:31:

---

mmf\utils\logger.py:19: in init self._is_master = is_master() mmf\utils\distributed.py:39: in is_master return get_rank() == 0

---

def get_rank():

  if not dist.is_nccl_available():

E AttributeError: module 'torch.distributed' has no attribute 'is_nccl_available'

mmf\utils\distributed.py:31: AttributeError ==================================================================================================== FAILURES ==================================================================================================== ___ TestModelInterfaces.test_mmbt_hm_interface ___

self =

@skip_if_no_network
def test_mmbt_hm_interface(self):

  model = MMBT.from_pretrained("mmbt.hateful_memes.images")

tests\models\interfaces\test_interfaces.py:14:

---

mmf\models\mmbt.py:514: in from_pretrained model = super().from_pretrained(model_name, *args, kwargs) mmf\models\base_model.py:196: in from_pretrained output = load_pretrained_model(model_name, *args, *kwargs) mmf\utils\checkpoint.py:40: in load_pretrained_model download_path = download_pretrained_model(model_name_or_path, args, kwargs) mmf\utils\download.py:378: in download_pretrained_model if is_master(): mmf\utils\distributed.py:39: in is_master return get_rank() == 0

---

def get_rank():

  if not dist.is_nccl_available():

E AttributeError: module 'torch.distributed' has no attribute 'is_nccl_available'

mmf\utils\distributed.py:31: AttributeError _ TestUtilsCheckpoint.test_finalize_and_restore_from_it __

self =

def test_finalize_and_restore_from_it(self):
    with mock_env_with_temp():
        checkpoint = Checkpoint(self.trainer)
        self._init_early_stopping(checkpoint)
        original_model = deepcopy(self.trainer.model)
        self._do_a_pass()
        model_1500 = deepcopy(self.trainer.model)

      checkpoint.save(1500)

tests\utils\test_checkpoint.py:218:

---

mmf\utils\checkpoint.py:358: in save if not is_master(): mmf\utils\distributed.py:39: in is_master return get_rank() == 0

---

def get_rank():

  if not dist.is_nccl_available():

E AttributeError: module 'torch.distributed' has no attribute 'is_nccl_available'

mmf\utils\distributed.py:31: AttributeError ___ TestUtilsCheckpoint.test_finalize_and_resume_file ____

self =

def test_finalize_and_resume_file(self):
    with mock_env_with_temp() as d:
        checkpoint = Checkpoint(self.trainer)
        self._init_early_stopping(checkpoint)
        self._do_a_pass()

      checkpoint.finalize()

tests\utils\test_checkpoint.py:273:

---

mmf\utils\checkpoint.py:415: in finalize if is_master(): mmf\utils\distributed.py:39: in is_master return get_rank() == 0

---

def get_rank():

  if not dist.is_nccl_available():

E AttributeError: module 'torch.distributed' has no attribute 'is_nccl_available'

mmf\utils\distributed.py:31: AttributeError ____ TestUtilsCheckpoint.test_pretrained_load ____

self =

def test_pretrained_load(self):
    with mock_env_with_temp() as d:
        checkpoint = Checkpoint(self.trainer)
        self._init_early_stopping(checkpoint)
        self._do_a_pass()
        original_model = deepcopy(self.trainer.model)
        # Test with zoo now
        ret_load_pretrained_zoo = {
            "config": self.config.model_config,
            "checkpoint": deepcopy(self.trainer.model.state_dict()),
            "full_config": self.config,
        }

      checkpoint.save(2000)

tests\utils\test_checkpoint.py:471:

---

mmf\utils\checkpoint.py:358: in save if not is_master(): mmf\utils\distributed.py:39: in is_master return get_rank() == 0

---

def get_rank():

  if not dist.is_nccl_available():

E AttributeError: module 'torch.distributed' has no attribute 'is_nccl_available'

mmf\utils\distributed.py:31: AttributeError ____ TestUtilsCheckpoint.testresets ____

self =

def test_resets(self):
    with mock_env_with_temp():
        checkpoint = Checkpoint(self.trainer)
        self._init_early_stopping(checkpoint)
        self._do_a_pass()

        original_optimizer = deepcopy(self.trainer.optimizer)
        original_model = deepcopy(self.trainer.model)

        self.trainer.current_epoch = 3

      checkpoint.save(2000, update_best=True)

tests\utils\test_checkpoint.py:316:

---

mmf\utils\checkpoint.py:358: in save if not is_master(): mmf\utils\distributed.py:39: in is_master return get_rank() == 0

---

def get_rank():

  if not dist.is_nccl_available():

E AttributeError: module 'torch.distributed' has no attribute 'is_nccl_available'

mmf\utils\distributed.py:31: AttributeError _ TestUtilsCheckpoint.test_save_and_load_state_dict __

self =

def test_save_and_load_state_dict(self):
    with mock_env_with_temp() as d:
        checkpoint = Checkpoint(self.trainer)
        self._init_early_stopping(checkpoint)
        self._do_a_pass()
        # Test normal case

      checkpoint.save(1500)

tests\utils\test_checkpoint.py:105:

---

mmf\utils\checkpoint.py:358: in save if not is_master(): mmf\utils\distributed.py:39: in is_master return get_rank() == 0

---

def get_rank():

  if not dist.is_nccl_available():

E AttributeError: module 'torch.distributed' has no attribute 'is_nccl_available'

mmf\utils\distributed.py:31: AttributeError ____ TestUtilsTimer.test_getcurrent ____

self =

def test_get_current(self):
    timer = Timer()
    expected = "000ms"

  self.assertEqual(timer.get_current(), expected)

E AssertionError: '0ms' != '000ms' E - 0ms E + 000ms E ? ++

tests\utils\test_timer.py:13: AssertionError _ TestUtilsTimer.test_reset __

self =

def test_reset(self):
    timer = Timer()
    time.sleep(2)
    timer.reset()
    expected = "000ms"

  self.assertEqual(timer.get_current(), expected)

E AssertionError: '0ms' != '000ms' E - 0ms E + 000ms E ? ++

tests\utils\test_timer.py:21: AssertionError ================================================================================================ warnings summary ================================================================================================ C:\Users\Abdurrabb\AppData\Local\Continuum\anaconda3\lib\site-packages\win32\lib\pywintypes.py:2 C:\Users\Abdurrabb\AppData\Local\Continuum\anaconda3\lib\site-packages\win32\lib\pywintypes.py:2: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses import imp, sys, os

tests\utils\test_model.py:8 F:\Python\DD_HM\mmf\tests\utils\test_model.py:8: PytestCollectionWarning: cannot collect test class 'TestDecoderModel' because it has a init constructor (from: tests/utils/test_model.py) class TestDecoderModel(nn.Module):

tests\utils\test_model.py:8 F:\Python\DD_HM\mmf\tests\utils\test_model.py:8: PytestCollectionWarning: cannot collect test class 'TestDecoderModel' because it has a init constructor (from: tests/utils/test_text.py) class TestDecoderModel(nn.Module):

tests/configs/test_configs_for_keys.py::TestConfigsForKeys::test_dataset_configs_for_keys F:\Python\DD_HM\mmf\tests\configs\test_configs_for_keys.py:51: UserWarning: Dataset vqa2_ocr has no default configuration defined. Skipping it. Make sure it is intentional ).format(builder_key)

tests/configs/test_configs_for_keys.py::TestConfigsForKeys::test_model_configs_for_keys F:\Python\DD_HM\mmf\tests\configs\test_configs_for_keys.py:26: UserWarning: Model multihead has no default configuration defined. Skipping it. Make sure it is intentional ).format(model_key)

tests/configs/test_configs_for_keys.py::TestConfigsForKeys::test_model_configs_for_keys F:\Python\DD_HM\mmf\tests\configs\test_configs_for_keys.py:26: UserWarning: Model top_down_bottom_up has no default configuration defined. Skipping it. Make sure it is intentional ).format(model_key)

tests/modules/test_metrics.py::TestModuleMetrics::test_caption_bleu4 C:\Users\Abdurrabb\AppData\Local\Continuum\anaconda3\lib\site-packages\nltk\decorators.py:68: DeprecationWarning: formatargspec is deprecated since Python 3.5. Use signature and the Signature object directly regargs, varargs, varkwargs, defaults, formatvalue=lambda value: ""

tests/modules/test_metrics.py::TestModuleMetrics::test_caption_bleu4 C:\Users\Abdurrabb\AppData\Local\Continuum\anaconda3\lib\site-packages\nltk\lm\counter.py:15: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working from collections import Sequence, defaultdict

tests/modules/test_metrics.py::TestModuleMetrics::test_caption_bleu4 C:\Users\Abdurrabb\AppData\Local\Continuum\anaconda3\lib\site-packages\nltk\lm\vocabulary.py:13: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working from collections import Counter, Iterable

tests/modules/test_metrics.py::TestModuleMetrics::test_macro_f1 C:\Users\Abdurrabb\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\metrics\classification.py:1437: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no predicted samples. 'precision', 'predicted', average, warn_for)

tests/modules/test_metrics.py::TestModuleMetrics::test_multilabel_macro_f1 C:\Users\Abdurrabb\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\metrics\classification.py:1439: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 in labels with no true samples. 'recall', 'true', average, warn_for)

tests/utils/test_checkpoint.py::TestUtilsCheckpoint::test_finalize_and_restore_from_it tests/utils/test_checkpoint.py::TestUtilsCheckpoint::test_finalize_and_resume_file tests/utils/test_checkpoint.py::TestUtilsCheckpoint::test_pretrained_load tests/utils/test_checkpoint.py::TestUtilsCheckpoint::test_resets tests/utils/test_checkpoint.py::TestUtilsCheckpoint::test_save_and_load_state_dict tests/utils/test_checkpoint.py::TestUtilsCheckpoint::test_zoo_load F:\Python\DD_HM\mmf\mmf\models\base_model.py:162: UserWarning: 'losses' already present in model output. No calculation will be done in base model. "'losses' already present in model output. "

tests/utils/test_checkpoint.py::TestUtilsCheckpoint::test_zoo_load F:\Python\DD_HM\mmf\mmf\utils\checkpoint.py:224: UserWarning: 'optimizer' key is not present in the checkpoint asked to be loaded. Skipping. "'optimizer' key is not present in the "

-- Docs: https://docs.pytest.org/en/latest/warnings.html ========================================================================= 8 failed, 57 passed, 18 warnings, 2 error in 269.61s (0:04:29) =========================================================================

[apsdehal]

As per https://pytorch.org/docs/stable/distributed.html#backends-that-come-with-pytorch torch.distributed is only supported for linux systems. I would skip the tests and test commands with environment variable CUDA_VISIBLE_DEVICES=0 set. Let us know if that doesn't work. I am closing this issue as it is more related to pytorch rather than mmf.