search

facebookresearch / mmf

A modular framework for vision & language multimodal research from Facebook AI Research (FAIR)
https://mmf.sh/
Other
5.46k stars 932 forks source link

What is sampled_idx_seq for TextVQA #1222

Closed soonchangAI closed 2 years ago

soonchangAI commented 2 years ago

❓ Questions and Help

Hi, I am trying to reproduce the M4C for TextVQA I noticed there is difference in the batch for different runs.

for batch in train_loader:
    print(batch)
    break

The sample list (same training example but for different runs) are largely similar except for sampled_idx_seq and train_prev_inds fields What causes the difference ? Is there any way to make it constant for different runs?

For example: During first run:

sampled_idx_seq', [(5013, 5021, 5024), (), (6,), (6,), (229, 486), (), (5010, 5012), (5002, 4888), (5001,), (172, 16, 72), (5005,), (18,), (813,), (458, 386, 110, 5012), (22,), (89,), (4,), (338, 265), (), (15,), (79,), (), (5001,), (14, 11, 8, 13, 12, 10, 5039, 5, 9), (5001,), (56,), (18, 133), (), (4,), (1482, 5006), (5005,), (5003, 264, 5008), (3978, 19, 5008), (30,), (518,), (4304,), (14, 11, 8, 13, 12, 10, 7, 5, 9), (156, 5002), (5001,), (5000, 297), (14, 11, 8, 13, 12, 10, 7, 5, 9), (5, 1587, 5004, 446), (286,), (1600,), (2590, 2883, 5004), (5002,), (5001,), (4,), (2662, 42), (97, 5004), (380, 39), (5001,), (5003, 27, 418), (4,), (5000, 599), (), (291,), (5012,), (5000,), (71,), (5001, 5002), (147,), (477,), (5001,)]), 
('train_prev_inds', tensor([[   1, 5013, 5021, 5024,    0,    0,    0,    0,    0,    0,    0,    0]

During second run:

sampled_idx_seq', [(5008, 5023, 5024), (), (6,), (6,), (229, 486), (), (5010, 4023), (5002,), (317, 5000), (5002,), (5005,), (18,), (813,), (5010, 386, 110, 78), (22,), (89,), (4,), (338, 265), (), (15,), (79,), (), (5001,), (377,), (5000,), (56,), (18, 133), (), (15,), (5005, 5006), (1012,), (4,), (3978, 19, 5008), (5001,), (518, 19), (73,), (14, 11, 8, 13, 12, 10, 7, 5, 9), (5003, 5002), (5001,), (5000, 5002), (4,), (5008, 5009), (286,), (22,), (2590, 277, 5004), (5002,), (5001, 66), (4,), (2662, 5014), (97, 5004), (380, 39), (5001,), (5002,), (4,), (107,), (), (291,), (5012,), (5000,), (71,), (2269, 5002), (147, 1727), (477,), (5001,)]), 
('train_prev_inds', tensor([[   1, 5008, 5023, 5024,    0,    0,    0,    0,    0,    0,    0,    0],

Similarly for STVQA During first run:

sampled_idx_seq',[(5013,), (3821, 2667), (), (5002, 5003), (5001,), (301,), (), (712, 181, 659), (5000,), (851, 101, 281), (4377,), (5009, 3337), (1051,), (5001,), (39,), (5000,)]

During second run:

sampled_idx_seq', [(5013,), (5001, 2667), (), (5002, 58), (5001,), (5000,), (), (712, 181, 659), (5000,), (851, 101), (4377,), (5009, 3337), (1051,), (458,), (39,), (5000,)]
soonchangAI commented 2 years ago

sampled_idx_seq is a random sample of one of the many answers for each example for "teacher-forcing". Can make the sampled_idx_seq constant by adding a line np.random.seed(0) in mmf/datasets/processors/processors.py

if len(all_idx_seq_list) > 0:
            # sample a random decoding answer sequence for teacher-forcing
            np.random.seed(0)
            idx_seq = all_idx_seq_list[np.random.choice(len(all_idx_seq_list))]