search

facebookresearch / mmf

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)
https://mmf.sh/
Other
5.5k stars 939 forks source link

Use a new dataset with same format to hateful memes data on visual bert with coco #584

Closed aggiejiang closed 4 years ago

aggiejiang commented 4 years ago

❓ Questions and Help

I have a new hateful memes dataset based on old hateful memes dataset, both with the same data format and file name.

train.jsonl
dev.jsonl
test.jsonl
img/
....

I put new files to the directory where mmf_convert_hm put files to. When I run the model as the tutorial said, I met problem like

Traceback (most recent call last):
  File "/data/home/acw519/mmfenv/bin/mmf_run", line 8, in <module>
    sys.exit(run())
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/mmf_cli/run.py", line 111, in run
    main(configuration, predict=predict)
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/mmf_cli/run.py", line 44, in main
    trainer.train()
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/mmf/trainers/base_trainer.py", line 245, in train
    for batch in self.train_loader:
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data
    return self._process_data(data)
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
    data.reraise()
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/mmf/datasets/databases/readers/feature_readers.py", line 205, in _load
    img_id_idx = self.image_id_indices[str(image_id).encode()]
KeyError: b'670482'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/torch/utils/data/dataset.py", line 207, in __getitem__
    return self.datasets[dataset_idx][sample_idx]
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/mmf/datasets/builders/hateful_memes/dataset.py", line 47, in __getitem__
    features = self.features_db.get(sample_info)
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/mmf/datasets/databases/features_database.py", line 97, in get
    return self.from_path(feature_path)
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/mmf/datasets/databases/features_database.py", line 105, in from_path
    features, infos = self._get_image_features_and_info(path)
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/mmf/datasets/databases/features_database.py", line 78, in _get_image_features_and_info
    image_feats, infos = self._read_features_and_info(feat_file)
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/mmf/datasets/databases/features_database.py", line 63, in _read_features_and_info
    feature, info = feature_reader.read(feat_file)
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/mmf/datasets/databases/readers/feature_readers.py", line 79, in read
    return self.feat_reader.read(image_feat_path)
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/mmf/datasets/databases/readers/feature_readers.py", line 129, in read
    image_info = self._load(image_feat_path)
  File "/data/home/acw519/mmfenv/lib/python3.8/site-packages/mmf/datasets/databases/readers/feature_readers.py", line 209, in _load
    img_id_idx = self.image_id_indices[image_id]
KeyError: b'670482'

I am not sure what causes this KeyError, the dataset not achieving one-to-one mapping or it requiring to re-generate features. Could you please suggest one solution for me. Thank you!

apsdehal commented 4 years ago

Hi,

Can you provide more details on how you constructed this new HM dataset? I don't see an image with id: 670482 in original images.

aggiejiang commented 4 years ago

the image with id 670482 is generated by the image with id 67048 in original images, by adding something new (like Gaussian noise, etc.

apsdehal commented 4 years ago

Then, you will have to generate a new lmdb which also has features for this image that you generated. Also, since the image has changed, you will have to generate new features for this image using https://github.com/facebookresearch/mmf/blob/master/tools/scripts/features/extract_features_vmb.py. This script will generate npy features for your image. Then you can convert all npy features in a folder into an lmdb using https://github.com/facebookresearch/mmf/blob/master/tools/scripts/features/lmdb_conversion.py

aggiejiang commented 4 years ago

Ok I see. So one more general question -- does this mean every time I need to use this script to generate and convert features if I want to use a new dataset for models?

apsdehal commented 4 years ago

It depends on the model you are using. VisualBERT/ViLBERT work on feature level so you need to generate the features. MMBT works on image level so you can directly use it with images.

aggiejiang commented 4 years ago

ok. Bert-based model is what I want to use. Thanks for your explanation!!

apsdehal commented 4 years ago

Follow these steps:

aggiejiang commented 4 years ago

I follow this step but meet an error KeyError: 'Non-existent config key: MODEL.BACKBONE.OUT_CHANNELS, which seems the same as #113.

For maskrcnn, I install torch 1.4.0 and torchvision 0.5.0 by pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html. cuda version is 10.1

Is anything wrong during this process?

apsdehal commented 4 years ago

Are you using the right maskrcnn_benchmark installation as specified in https://github.com/facebookresearch/mmf/blob/master/tools/scripts/features/extract_features_vmb.py#L3

aggiejiang commented 4 years ago

yes. I follow the install file for maskrcnn. But I install pytorch and torchvision by using 

pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html`

instead of

# follow PyTorch installation in https://pytorch.org/get-started/locally/
# we give the instructions for CUDA 9.0
conda install pytorch-nightly -c pytorch

# install torchvision
cd ~/github
git clone https://github.com/pytorch/vision.git
cd vision
python setup.py install

other steps are all the same with the instruction.

apsdehal commented 4 years ago

I meant you are not using the right maskrcnn_benchmark repo. It should be https://gitlab.com/vedanuj/vqa-maskrcnn-benchmark. The error you are seeing is because you are using actual maskrcnn-benchmark.

apsdehal commented 4 years ago

Also, any particular reason you are using nightly build?

aggiejiang commented 4 years ago

nightly build requires python<3.8 but I use python 3.8.2 before. I change the version back to 3.7 and it seems ok to run. thanks for your tips!

Should I open a new issue for other errors? ('Segmentation fault' but the environment with enough memory)

apsdehal commented 4 years ago

Yes, please open up a new issue. We like to keep issues self-contained.

guanyuhang8866 commented 4 years ago

Follow these steps:

  • Create a new conda environment
  • Install MMF first in this environment
  • Then install maskrcnn, MMF will work with these version of torch and torchvision, but MaskRCNN won't work with higher versions of torch and torchvision.
  • Extract features, move back to your original env, create an LMDB.
  • Update experiment configurations to point to new LMDBs
  • Profit!

here has a queation about how to create lmdb file,would you please answer us?