HireTheHero commented 2 years ago

❓ Questions and Help

Question

To test if I can train UniT with dataset other than 8 datasets specified in tutorial or not, I added hateful_memes following your guidance.

Then tried to run the training script (see code block A ) with small modification over config file (see code block B ), resulted in ConfigKeyError (see code block C ).

Am I missing something for the training? Or the task should be one of 8 tasks in the tutorial?

Code blocks

A: Command for running script

python /home/<username>/<path_to_mmf>/mmf_cli/run.py \
config=/home/<username>/<path_to_one_directory_above_mmf>/mmf_added/shared_dec.yaml \
datasets=hateful_memes \
model=unit run_type=train \
env.save_dir=/home/<username>/data/mmf \
distributed.world_size=2 distributed.port=20000 \
env.data_dir=/home/<username>/data/mmf

B: Config file


# original: 
# https://github.com/facebookresearch/mmf/blob/582c7195cbf1eb948436b66c1e9e4bb2e5652a27/mmf/configs/models/unit/defaults.yaml
includes:

../mmf/projects/unit/configs/vqa2_dataset_cfg.yaml
../mmf/projects/unit/configs/visual_entailment_dataset_cfg.yaml

model_config: unit: base_args: num_queries: detection: detection_coco: 100 detection_visual_genome: 100 hateful_memes: 100 vl: vqa2: 25 hateful_memes: 25

visual_entailment: 25

    glue:
      glue_qnli: 25
      glue_qqp: 25
      glue_sst2: 25
      glue_mnli_mismatched: 25
      hateful_memes: 25
  share_decoders: true
  decoder_hidden_dim: 768
  dilation: true
  use_task_embedding_in_img_encoder: true
  use_task_embedding_in_lang_encoder: true
losses:
- logit_bce
# initialize the ResNet convnet backbone from DETR
base_ckpt_path: https://dl.fbaipublicfiles.com/detr/detr-r50-dc5-f0fb7ef5.pth
base_ckpt_load_backbone_only: true

evaluation: metrics:

type: vqa_accuracy datasets:
- vqa2
type: detection_mean_ap key: detection_mean_ap datasets:
- detection_coco
- detection_visual_genome params: dataset_json_files: detection_coco: val: ${env.data_dir}/datasets/coco/detection_2017/annotations/annotations/instances_val2017.json detection_visual_genome: val: ${env.data_dir}/datasets/visual_genome/detection_split_by_coco_2017/annotations/instances_val_split_by_coco_2017.json
type: accuracy key: accuracy datasets:
- visual_entailment
- glue_qnli
- glue_qqp
- glue_sst2
- glue_mnli_mismatched
- hateful_memes

(rest is same as original yaml file)

#### C: Full Traceback

Traceback (most recent call last): File "/home//anaconda3/envs/mmf/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap fn(i, *args) File "/home///mmf_cli/run.py", line 66, in distributed_main main(configuration, init_distributed=True, predict=predict) File "/home///mmf_cli/run.py", line 52, in main trainer.load() File "/home///mmf/trainers/mmf_trainer.py", line 42, in load super().load() File "/home///mmf/trainers/base_trainer.py", line 33, in load self.load_model() File "/home///mmf/trainers/mmf_trainer.py", line 102, in load_model self.model = build_model(attributes) File "/home///mmf/utils/build.py", line 145, in build_model model.build() File "/home///mmf/models/unit/unit.py", line 91, in build num_cls = self.config.heads["detection"][dataset_name]["num_classes"] File "/home//anaconda3/envs/mmf/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 370, in getitem key=key, value=None, cause=e, type_override=ConfigKeyError File "/home//anaconda3/envs/mmf/lib/python3.7/site-packages/omegaconf/base.py", line 196, in _format_and_raise type_override=type_override, File "/home//anaconda3/envs/mmf/lib/python3.7/site-packages/omegaconf/_utils.py", line 741, in format_and_raise _raise(ex, cause) File "/home//anaconda3/envs/mmf/lib/python3.7/site-packages/omegaconf/_utils.py", line 719, in _raise raise ex.with_traceback(sys.exc_info()[2]) # set end OC_CAUSE=1 for full backtrace File "/home//anaconda3/envs/mmf/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 367, in getitem return self._get_impl(key=key, default_value=_DEFAULTMARKER) File "/home//anaconda3/envs/mmf/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 438, in _get_impl node = self._get_node(key=key, throw_on_missing_key=True) File "/home//anaconda3/envs/mmf/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 465, in _get_node self._validate_get(key) File "/home//anaconda3/envs/mmf/lib/python3.7/site-packages/omegaconf/dictconfig.py", line 167, in _validate_get key=key, value=value, cause=ConfigAttributeError(msg) File "/home//anaconda3/envs/mmf/lib/python3.7/site-packages/omegaconf/base.py", line 196, in _format_and_raise type_override=type_override, File "/home//anaconda3/envs/mmf/lib/python3.7/site-packages/omegaconf/_utils.py", line 821, in format_and_raise _raise(ex, cause) File "/home//anaconda3/envs/mmf/lib/python3.7/site-packages/omegaconf/_utils.py", line 719, in _raise raise ex.with_traceback(sys.exc_info()[2]) # set end OC_CAUSE=1 for full backtrace omegaconf.errors.ConfigKeyError: Key 'hateful_memes' is not in struct full_key: model_config.unit.heads.detection.hateful_memes object_type=dict

ronghanghu commented 2 years ago

Hi, since Hateful Memes is not used for detection tasks, it should not be added to model_config.unit.base_args.num_queries.detection in "B: Config file".

Instead, it should be added only under model_config.unit.base_args.num_queries.vl, as follows:

...
model_config:
  unit:
    base_args:
      num_queries:
        detection:
          detection_coco: 100
          detection_visual_genome: 100
          hateful_memes: 100
        vl:
          ...