External exemplars

[GioFic95]

Hi @Verg-Avesta, is it possible to make inference providing exemplars which are not part of the input image? That is, I would like to select k exemplars from the first of n images composing my test dataset, and then use the same k crops from the first image for all the n queries, given that the objects are of the same class across the whole dataset.

Is this possibility already available and implemented somewhere? Thank you in advance.

[Verg-Avesta]

It is possible to make inference with other exemplars, while I didn't implement such function. You can try to replace the exemplar boxes with fixed image crops.

[GioFic95]

I'll try, thank you very much.

[GioFic95]

Hi again, do you have any advice on how to select the exemplars? In particular, have you ever tried with very crowded images, where the objects are mostly superimposed (i.e., it is hard to select a box with only one object and no parts of other objects)?

[Verg-Avesta]

I think choosing an exemplar that is as clean as possible can make the effect better. But when the images are too crowded, the vision transformer architecture may struggle on such images. So the choice on the exemplars probably won't affect the results too much.

[ketul93]

@GioFic95 Did you try adding external exemplar (from other input image) ? if yes can help to share the code snippet ? thanks. Also how many exemplars are recommended to chose ?

Also can help to share how to add multiple category external exemplars ?

[GioFic95]

@ketul93 Yes, I managed to use external exemplars, you can find the details in this PR. So pull the updated code and run the inference with the --external parameter.

Based on my observations, increasing the number of exemplars per image decreases the performance, so I'd stay around 3-5 exemplars (you can pick them from one or more images in the dataset, indicating the boxes' coordinates in the annotation json file). However, this may be related to my specific task and dataset.

I'm not sure about your last question. The model does not distinguish specific categories, it just counts objects, so you can add exemplars of multiple categories from different images, internally (different exemplars for each image) or externally (the same exemplars from a few images for the whole test set). But currently you can't specify a different set of external exemplars for a different subset of the dataset.

I'm available for further doubts.

[israel-cj]

@ketul93 Yes, I managed to use external exemplars, you can find the details in this PR. So pull the updated code and run the inference with the --external parameter.

Based on my observations, increasing the number of exemplars per image decreases the performance, so I'd stay around 3-5 exemplars (you can pick them from one or more images in the dataset, indicating the boxes' coordinates in the annotation json file). However, this may be related to my specific task and dataset.

I'm not sure about your last question. The model does not distinguish specific categories, it just counts objects, so you can add exemplars of multiple categories from different images, internally (different exemplars for each image) or externally (the same exemplars from a few images for the whole test set). But currently you can't specify a different set of external exemplars for a different subset of the dataset.

I'm available for further doubts.

Dear, thank you for your great work in creating the external function I wonder if you have a minimal example of how the data should be organized to run --external

I mean, currently I have a folder called 'external data' with the next tree: └───_annotationexternal.json └───_testexternal.json └───images └──────41.jpg └──────42.jpg └──────52.jpg └──────56.jpg └──────60.jpg └──────69.jpg

My _annotationexternal.json has the next info:

{
    "41.jpg": {
        "H": 490,
        "W": 800,
        "box_examples_coordinates": [
            [
                [
                    419,
                    158
                ],
                [
                    419,
                    229
                ],
                [
                    496,
                    229
                ],
                [
                    496,
                    158
                ]
            ],
            [
                [
                    257,
                    64
                ],
                [
                    257,
                    117
                ],
                [
                    298,
                    117
                ],
                [
                    298,
                    64
                ]
            ],
            [
                [
                    354,
                    107
                ],
                [
                    354,
                    172
                ],
                [
                    406,
                    172
                ],
                [
                    406,
                    107
                ]
            ]
        ]
    }
}

My _testexternal.json contains:

{
    "test": [
        "41.jpg",
        "42.jpg",
        "52.jpg",
        "56.jpg",
        "60.jpg",
        "69.jpg"  
    ]
}

Then I run the next:

CUDA_VISIBLE_DEVICES=6 python -u 'FSC_test_cross(few-shot).py' \
    --external True \
    --data_path ./data/external/ \
    --anno_file annotation_external.json \
    --data_split_file test_external.json \
    --im_dir images \
    --resume '/home/israel/israel/snails/CounTR/data/out/fim6_dir/checkpoint__finetuning_999.pth'

But I get the error: _anno = annotations[imid] KeyError: '52.jpg'

Could you please help me to know what I am doing wrong in order to fix the boxes only with the first image and use them with the rest of the set. Thank you in advance

[GioFic95]

Hi @israel-cj, I just made a new pull request #39 with a new file with examples of the main commands to train and test CounTR (https://github.com/GioFic95/CounTR/blob/main/run_minimal.MD).

Moreover, I uploaded here input files with annotations I used for minimal examples of training and testing, including few-shot test with external exemplars: https://drive.google.com/file/d/134ac1x381PsaC3Y6RFXQWzpmlSzaoc3X/view?usp=sharing.

You can refer the file and the Google Drive folder for better understanding how to use the --external parameter.

Note that in my pull request I changed it from

parser.add_argument('--external', default=False, help='True if using external exemplars')

to

parser.add_argument('--external', action='store_true', help='Set this param for using external exemplars')

so you will have to write --external instead of --external True if you decide to use it, but I didn't change the way it works.

 

By the way, I think your problem is due to the fact that you didn't provide any information about the images without exemplars in the annotation_external.json file, while you need to have all the test images in that file, even if with empty lists. For example:

{
    "41.jpg": {...},
    "42.jpg": {
        "box_examples_coordinates": [],
        "points": []
    },
    "52.jpg": {...},
    ...
}

[israel-cj]

@GioFic95 you are a lifesaver Thank you for the clear minimal example I just ran it yesterday and today and it is working perfectly! :D :) I am using this to tackle a problem of sustainability Indeed, my problem was that I was not providing any information about the images without examplars, I needed to include empty lists I appreciate you included even Fine-tuning with data augmentation and your weights

[GioFic95]

@israel-cj you're welcome!

The weights are the same provided by the authors (@Verg-Avesta). I Included that file in my zip just to provide the same folder structure I used, so that if you run the examples in run_minimal.MD everything works.

By the way, right at this very moment I am introducing an update for the "external exemplars" approach, so that different exemplars can be used for each subset of the dataset (e.g. train, test, val splits, or different test sets) based on the same json file already used to define the dataset splits for standard training and testing.

[israel-cj]

@GioFic95 thanks again, you are right regarding the weights. The update you just mentioned is something that definitely is worthy to look into, thanks