Overlapping train and test set

zc-alexfan commented 4 years ago

Thanks for sharing the dataset and the code. I am trying to figure out how the train, val and test sets are split. Therefore, I ran the following code and found that some images in train, val and test sets overlaps. I am wondering if I did something wrong? Or it is a mistake in the provided json files.

import pickle as pkl
import json

with open('./data/InterHand2.6M/annotations/all/InterHand2.6M_train_data.json') as f:
    data = json.load(f)

sequences = []
for idx in range(len(data['images'])):
    seq_name = data['images'][idx]['file_name']
    sequences.append(seq_name)
sequences_train = set(sequences)

with open('./data/InterHand2.6M/annotations/machine_annot/InterHand2.6M_val_data.json') as f:
    data = json.load(f)

sequences = []
for idx in range(len(data['images'])):
    seq_name = data['images'][idx]['file_name']
    sequences.append(seq_name)    
sequences_val = set(sequences)

with open('./data/InterHand2.6M/annotations/all/InterHand2.6M_test_data.json') as f:
    data = json.load(f)

sequences = []
for idx in range(len(data['images'])):
    seq_name = data['images'][idx]['file_name']
    sequences.append(seq_name)

sequences_test = set(sequences)

Results

The output for sequences_train.intersection(sequences_val) is set().

The output for len(sequences_test.intersection(sequences_val)) is 13507

The output for list(sequences_test.intersection(sequences_val))[:10] is:

['Capture0/ROM02_Interaction_2_Hand/cam400428/image22223.jpg',
 'Capture0/ROM03_LT_No_Occlusion/cam400364/image16814.jpg',
 'Capture0/ROM03_LT_No_Occlusion/cam400269/image16454.jpg',
 'Capture0/ROM03_LT_No_Occlusion/cam400372/image17444.jpg',
 'Capture0/ROM03_LT_No_Occlusion/cam400299/image16604.jpg',
 'Capture0/ROM03_LT_No_Occlusion/cam400500/image16772.jpg',
 'Capture0/ROM03_LT_No_Occlusion/cam400281/image16550.jpg',
 'Capture0/ROM03_LT_No_Occlusion/cam400375/image17372.jpg',
 'Capture0/ROM03_LT_No_Occlusion/cam400364/image17348.jpg',
 'Capture0/ROM03_LT_No_Occlusion/cam400367/image16472.jpg']

The output for sequences_train.intersection(sequences_test) is:

{'Capture5/0007_thumbup_normal/cam400006/image2564.jpg',
 'Capture5/0007_thumbup_normal/cam400010/image2564.jpg',
 'Capture5/0007_thumbup_normal/cam400042/image2564.jpg',
 'Capture5/0007_thumbup_normal/cam400053/image2564.jpg',
 'Capture5/0007_thumbup_normal/cam400067/image2564.jpg',
 'Capture5/0007_thumbup_normal/cam410003/image2564.jpg',
 'Capture5/0007_thumbup_normal/cam410067/image2564.jpg',
 'Capture5/0007_thumbup_normal/cam410209/image2564.jpg',
 'Capture5/0007_thumbup_normal/cam410210/image2564.jpg',
 'Capture5/0007_thumbup_normal/cam410218/image2564.jpg',
 'Capture5/0007_thumbup_normal/cam410236/image2564.jpg',
 'Capture5/0007_thumbup_normal/cam410238/image2564.jpg',
 'Capture7/0001_neutral_rigid/cam400006/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam400008/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam400012/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam400013/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam400016/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam400017/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam400041/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam400053/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam400059/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam400067/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410001/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410003/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410004/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410007/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410028/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410053/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410062/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410063/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410208/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410209/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410210/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410213/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410218/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410219/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410233/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410236/image0372.jpg',
 'Capture7/0001_neutral_rigid/cam410238/image0372.jpg'}

mks0601 commented 4 years ago

Could you take a look at images of the overlapped file name? They are different images, but just have same file names. Some images from different splits (train/test/val) may have the same filenames, but images from the same split do not share the same file names.

zc-alexfan commented 4 years ago

Aha, my fault. Thanks for the clarification. :)

zc-alexfan commented 4 years ago

Sorry, can I assume that CaptureX in one of train, val, test are different sequences? For example, Capture0 in train and test are not related to each other (even if they have the same action name) except the camera parameters.

mks0601 commented 4 years ago

All data in train, test, val are not related. The camera parameters are also separated into train, test, val.

zc-alexfan commented 4 years ago

Thanks a lot

mks0601 commented 4 years ago

The camera parameter file name is InterHand2.6M_$DB_SPLIT_camera.json

facebookresearch / InterHand2.6M

Overlapping train and test set #11

Results