Inquiry about Dataset

[Qhan-Hu]

Hi Lee, your work on Missing Modality is very fascinating and attractive. I tried to understand your coding logic through debugging and apply it on the private dataset, but failed. I don't quite understand your writing about the dataset and dataloader, so I'm not sure how to integrate my data into the model. Could you provide some examples, such as how hatememes_dataset.py and hatememes_datamodule.py create the modal-missing dataset from the original dataset? Thanks sincerely for your help.

[YiLunLee]

Thank you for your question. The codes related to dataset and dataloader are based on the ViLT and I only revised it by adding missing data construction. In your example (hateful memes), the modal-missing dataset creation function is added in hatememes_dataset.py: it first loads the original data (Line 17-23), and then creates the missing table (Line 26-60) which records the missing type for each data (0 is complete, 1 is missing-text, 2 is missing-image). You should prepare your own missing table json file in ./datasets/missing_tables/ or it will randomly create the table according to the missing config (Line 26-33) and save it in a json file (Line 44-58). During loading data, __getitem__ function will return the modal-missing data pairs according to the missing table.

If you want to use your dataset and apply it to your task, you may need to create or revise the following files:

1. Datasets in vilt/datasets/: create {}_dataset.py and reviseinit.py
2. Dataloaders in vilt/datamodules/: create {}_datamodule.py and revise init.py
3. Objectives: revisevilt/module/objectives.py
4. Task metric records: revise vult/module/vilt_utils.py
5. New metric: revise vilt/gadgets/my_metric.py
6. Config: revise vilt/config.py