Open sandylaker opened 2 years ago
Thanks for this. Can you explain why we are using a forward batch here? eg: for deletion, the aim is to remove perturb-step-size pixels from the image (sorted by saliency map importance) until all pixels are perturbed and calculate scores at each stage of perturbation, why do we need to process it in batches of size 'forward batch size'?
For a single image, we progressively replace the pixels with values from another pre-defined tensor. At each step, we place perturb_step_size
e.g. 10 pixels. If an image with spatial size 224 by 224 is given, then it takes over 4000 steps. If we forward perturbed tensors individually, then it is too slow. Instead, we would batch these perturbed tensors and forward them batch by batch to fully utilize the computation power of the GPU.
Thanks, it's clear now :).
What is the use of the method update in InsertionDeletion class? Is it different from add_single_result method in InsertionDeletionResult class?
@rjagtani It is basically calling self._result.add_single_result
inside the function. Note that self._result
is private. So we cannot call add_single_result
outside the InsertionDeletion
class. Therefore, we wrap it a little bit. In the end it is like this:
ins_del = InsertionDeletion(...)
single_result = ins_del.evaluate(...)
# maybe some post-processing of the single_result, e.g. add img_path to the dict
...
ins_del.update(single_result)
I'm writing tests for Insertion Deletion Metric and one of the problems I'm facing is that I don't have a constant output to test my code against. Here is the test - assert expected_output == ins_del.evaluate(...) but I'm getting different prediction scores for the same image (because the weights are initialized randomly and pretrained is set to False). Any ideas would be appreciated. I'd also like to discuss some changes to the code and tests that I have added. I have published the code on my github - the name of the branch is 'id3'.
I'm writing tests for Insertion Deletion Metric and one of the problems I'm facing is that I don't have a constant output to test my code against. Here is the test - assert expected_output == ins_del.evaluate(...) but I'm getting different prediction scores for the same image (because the weights are initialized randomly and pretrained is set to False). Any ideas would be appreciated. I'd also like to discuss some changes to the code and tests that I have added. I have published the code on my github - the name of the branch is 'id3'.
Please do not use any pre-trained/ randomly initialized torchvision or timm models in your tests. They are too heavy. Instead, create a dummy and shallow CNN on your own, and initialize the weights to certain constants.
In addition, it is not necessary to test the numeric equality in all cases. Sometimes testing the object types or array shapes are sufficient.
Regarding the updated code, please send a Draft PR so that I can comment and send suggestions.
I'm writing tests for Insertion Deletion Metric and one of the problems I'm facing is that I don't have a constant output to test my code against. Here is the test - assert expected_output == ins_del.evaluate(...) but I'm getting different prediction scores for the same image (because the weights are initialized randomly and pretrained is set to False). Any ideas would be appreciated. I'd also like to discuss some changes to the code and tests that I have added. I have published the code on my github - the name of the branch is 'id3'.
Please do not use any pre-trained/ randomly initialized torchvision or timm models in your tests. They are too heavy. Instead, create a dummy and shallow CNN on your own, and initialize the weights to certain constants.
In addition, it is not necessary to test the numeric equality in all cases. Sometimes testing the object types or array shapes are sufficient.
Regarding the updated code, please send a Draft PR so that I can comment and send suggestions.
Thanks, I'll make these changes and send a Draft PR
I get this error when I run 'mypy saliency_metrics' - Signature of "evaluate" incompatible with supertype "ReInferenceMetric". Seems like the issue has been discussed here and would require changes to ReInferenceMetric https://stackoverflow.com/questions/51003146/python-3-6-signature-of-method-incompatible-with-super-type-class I've created a pull request nevertheless.
@rjagtani The mypy issue is because you modify the method signature to def evaluate(self, img: Tensor, smap: Tensor, img_path: str = None) -> Dict
. Please use def evaluate(self, img: Tensor, smap: Tensor, target: int, **kwargs: Any) -> Dict
. The img_path
can be retrieved from the kwargs as img_path: str = kwargs["img_path"]
.
Implementation of Insertion-Deletion
As indicated by the name, Insertion-Deletion contains two experiments:
ProgressivePerturbation
To implement these two experiments, we first need to implement
ProgressivePerturbation
class which looks like this:input_tensor
is the tensor to be modified, and the replacement values come fromreplace_tensor
.sorted_inds
is a tuple of(row_inds, col_inds)
, which indicate the row and column indices of the sorted saliency map pixels.perturb
method returns an iterator of Tensors. Each tensor should containsforward_batch_size
perturbed tensors, except for the last one, which contains(num_pixels // perturb_step_size + 1) % forward_batch_size
perturbed tensors. At each step, the method perturbsperturb_step_size
pixels from the current tensor. After generatingforward_batch_size
perturbed tensors, it then stacked the perturbed tensors together and yield the batched tensor.InsertionDeletionResult
The
single_result
is a dictionary containing following fields:"del_scores"
: anp.ndarray
that stores the deletion scores at each perturbation step."ins_scores": a
np.ndarray` that stores the insertion scores at each perturbation step."img_path"
: the path to the image."ins_auc"
: insertion AUC."del_auc"
: deletion AUC.The flag
summarized
specifies whether to dump all the raw single results into one JSON file, or compute the mean and std of theins_auc
anddel_auc
and dump only the statistics. Ifsummarized = True
, a dict containing following fields should be dumped to a JSON file:"mean_ins_auc"
: a float."std_inds_auc"
: a float."mean_del_auc"
: a float."std_del_auc"
: a float."num_samples"
: an int.I.e. a JSON file like this:
Otherwise, the JSON file is like this:
InsertionDeletion
eval
mode.scipy
provides functions for computing the AUC by integrating theins_scores
ordel_scores
.