Establishing training and testing on new animals dataset

[fixer1107]

Package Version

wildlife-datasets Version: 1.0.2 wildlife-tools Version: 0.0.2

Problem Description

Hello, I am a newcomer on Deep Learning and I am doing research on identification for animal individuals. I have gone through your paper, documentations and notebooks then I tried to follow the examples to create my own datasets.

My steps are:

1. take some random screenshots from my video and used the wildlife-dataset package to build a simple metadata with path, two identities and train/test splits with no cropping infos; metadata
2. use a piece of test script to train and extract features from my dataset, then check the similarity and classify out the prediction(s)

cwd = os.getcwd()
print(cwd)

metadata = pd.read_csv(cwd + '/examples/marmoset/metadata.csv')
image_root = './examples/marmoset/images/'

transform = T.Compose([
    T.Resize([224, 224]),
    T.ToTensor(),
    T.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
])
# transform = T.Compose([T.Resize(size=(384, 384)),
#                               T.ToTensor(), 
#                               T.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])]) 

dataset_database = WildlifeDataset(
    metadata = metadata, 
    root = image_root,
    split = SplitMetadata('split', 'train'),
    transform=transform
)

dataset_query = WildlifeDataset(
    metadata = metadata, 
    root = image_root,
    split = SplitMetadata('split', 'test'),
    transform=transform
)

print("the database: ")
print(dataset_database.metadata)
print("the data query: ")
print(dataset_query.metadata)

backbone = timm.create_model('hf-hub:BVRA/MegaDescriptor-S-224', num_classes=0, pretrained=True)
extractor = DeepFeatures(backbone)
query, database = extractor(dataset_query), extractor(dataset_database)

print(f'Query features shape: {query.shape}, Database features shape: {database.shape}')

# Cosine similarity between deep features
similarity = CosineSimilarity()
sim = similarity(query, database)['cosine']
print("Similarity matrix: \n", sim.shape)

# Nearest neigbour classifier using the similarity
classifier = KnnClassifier(k=1, database_labels=dataset_database.labels_map)
preds = classifier(sim)
print(type(preds))
print("Prediction \t", preds)
print("Ground truth \t", dataset_query.labels_string)

acc = sum(preds == dataset_query.labels_string) / len(dataset_query.labels_string)
print('\n Accuracy: ', acc)

The model is going on well with 3-5 images. The issue appears when the samples come to 6 or more, the predictions may fall beyond the identities set(database_labels), like belows:

the database: image_id identity path split 0 image_000 a marmoset_a/image_000.jpg train 1 image_001 a marmoset_a/image_001.jpg train 2 image_003 b marmoset_b/image_003.jpg train 3 image_005 b marmoset_b/image_005.jpg train the data query: image_id identity path split 0 image_002 a marmoset_a/image_002.jpg test 1 image_006 b marmoset_b/image_006.jpg test 100%|█████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 2.47it/s] 100%|█████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 1.64it/s] Query features shape: (2, 768), Database features shape: (4, 768) Similarity matrix: (2, 4) The databse labels are: ['a' 'b'] The prediction(s) is: [0 3] Traceback (most recent call last): File "/mnt/d/Programs/SIAT/codes/wildlife-tools/test.py", line 64, in preds = classifier(sim) ^^^^^^^^^^^^^^^ File "/mnt/d/Programs/SIAT/codes/wildlife-tools/wildlife_tools/inference/classifier.py", line 41, in call pred = self.database_labels[pred]

IndexError: index 3 is out of bounds for axis 0 with size 2

the database: 
    image_id identity                      path  split
0  image_000        a  marmoset_a/image_000.jpg  train
1  image_001        a  marmoset_a/image_001.jpg  train
2  image_003        a  marmoset_a/image_003.jpg  train
3  image_005        a  marmoset_a/image_005.jpg  train
4  image_006        a  marmoset_a/image_006.jpg  train
5  image_007        b  marmoset_b/image_007.jpg  train
6  image_008        b  marmoset_b/image_008.jpg  train
7  image_010        b  marmoset_b/image_010.jpg  train
8  image_011        b  marmoset_b/image_011.jpg  train
9  image_012        b  marmoset_b/image_012.jpg  train
the data query: 
    image_id identity                      path split
0  image_002        a  marmoset_a/image_002.jpg  test
1  image_009        b  marmoset_b/image_009.jpg  test
100%|█████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.17it/s]
100%|█████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.17s/it]
Query features shape: (2, 768), Database features shape: (10, 768)
Similarity matrix: 
 (2, 10)
The databse labels are: 
['a' 'b']
The prediction(s) is: 
[9 6]
Traceback (most recent call last):
  File "/mnt/d/Programs/SIAT/codes/wildlife-tools/test.py", line 64, in <module>
    preds = classifier(sim)
            ^^^^^^^^^^^^^^^
  File "/mnt/d/Programs/SIAT/codes/wildlife-tools/wildlife_tools/inference/classifier.py", line 41, in __call__
    pred = self.database_labels[pred]
           ~~~~~~~~~~~~~~~~~~~~^^^^^^
IndexError: index 9 is out of bounds for axis 0 with size 2

So I guess the I might have done wrong in some initial steps like data preprocessing or the training parts? So would you please kindly have a check on my problems and provide some suggestions? Thank you!

[wisestump]

# Nearest neigbour classifier using the similarity
classifier = KnnClassifier(k=1, database_labels=dataset_database.labels_map)

Here it should be database_labels=dataset_database.labels_string, as in other calls, not database_labels=dataset_database.labels_map

[fixer1107]

# Nearest neigbour classifier using the similarity
classifier = KnnClassifier(k=1, database_labels=dataset_database.labels_map)

Here it should be database_labels=dataset_database.labels_string, as in other calls, not database_labels=dataset_database.labels_map

Thanks so much for your help! I think it is working properly now.