Questions about the calculation of evaluation indices(IOU)

[wujiang0156]

Thank you very much, your work is very meaningful, but I have some questions. Thank you. In the table above, is the IoU of the foreground （building）change class? But in your code, IoU is calculated like this It doesn't seem to be the IoU of the foreground change class (building), Here it might be minus self.confusion_matrix[1, 1]) instead of adding self.confusion_matrix[1, 1]).

[ChenHongruixuan]

Thank you for your question. Please review the definition of IoU. It is [IoU = TP / (TP + FP + FN)].

I don't understand what do you mean about > This is a serious error, please provide a solution?

[ChenHongruixuan]

About the calculation of mIoU, it is also right. You can review the definition of mIoU and calculate it yourself manually, in comparison with using our code.

[wujiang0156]

About the calculation of mIoU, it is also right. You can review the definition of mIoU and calculate it yourself manually, in comparison with using our code. Thank you for your response. The calculation method of mIoU is correct What I mean is that when calculating IoU, it should be subtraction instead of addition here.

[wujiang0156]

Thank you for your question. Please review the definition of IoU. It is [IoU = TP / (TP + FP + FN)].

I don't understand what do you mean about > This is a serious error, please provide a solution?

Thank you for your response.

The formula is correct, but there is a problem with the code.

What I mean is that when calculating IoU, it should be subtraction instead of addition here.

Additionally, the IoU of the foreground (building) change class in LEVIR+CD, WHU, and SYSU?

[wujiang0156]

About the calculation of mIoU, it is also right. You can review the definition of mIoU and calculate it yourself manually, in comparison with using our code. self.confusion_matrix[1, 1])

Thank you for your response. The calculation methods of mIoU and IoU are similar; both are obtained through the confusion matrix. The difference is that mIoU is cumulative. Therefore, in the code, when calculating IoU, it should subtract self.confusion_matrix[1, 1] instead of adding self.confusion_matrix[1, 1]. IoU should be calculated as follows:

I look forward to your answer. thank you very much.

[ChenHongruixuan]

I think you didn't figure out the concept of IoU here. When calcualting IoU for binary change detection, the num_class is 2. The size of confusion matrix is 2x2, where self.confusion_matrix[1,1] is TP, the remaning two terms are FP and FN. Therefore, according to the definition of IoU=TP / (TP + FN + FP), the code should be self.confusion_matrix[1,1] / (self.confusion_matrix[1,0] + self.confusion_matrix[1,0] + self.confusion_matrix[1,1]). The accumulation does not exist here.

BTW, I think the more appropriate way about raising issue is that you should try to calculate them yourself and then use the wording like [ciritial error].

[wujiang0156]

I think you didn't figure out the concept of IoU here. When calcualting IoU for binary change detection, the num_class is 2. The size of confusion matrix is 2x2, where self.confusion_matrix[1,1] is TP, the remaning two terms are FP and FN. Therefore, according to the definition of IoU=TP / (TP + FN + FP), the code should be self.confusion_matrix[1,1] / (self.confusion_matrix[1,0] + self.confusion_matrix[1,0] + self.confusion_matrix[1,1]). The accumulation does not exist here.

BTW, I think the more appropriate way about raising issue is that you should try to calculate them yourself and then use the wording like [ciritial error].

Thank you for your response. So you mean that when calculating mIoU, addition should also be used, but the code here uses subtraction. How can this be explained? np.diag(self.confusion_matrix) is TP

[wujiang0156]

I think you didn't figure out the concept of IoU here. When calcualting IoU for binary change detection, the num_class is 2. The size of confusion matrix is 2x2, where self.confusion_matrix[1,1] is TP, the remaning two terms are FP and FN. Therefore, according to the definition of IoU=TP / (TP + FN + FP), the code should be self.confusion_matrix[1,1] / (self.confusion_matrix[1,0] + self.confusion_matrix[1,0] + self.confusion_matrix[1,1]). The accumulation does not exist here.

BTW, I think the more appropriate way about raising issue is that you should try to calculate them yourself and then use the wording like

Thank you for your response.

yes , you are right.
FP = np.sum(self.confusion_matrix, axis=0) - TP FN = np.sum(self.confusion_matrix, axis=1) - TP
so , When calculating mIoU, it should be +TP.

Thank you for your patient explanations. Your work has been very inspiring to me. Thanks again for your efforts.

[JTRNEO]

For mIoU, when summing over rows and columns to get the total predicted and actual areas for each class, True Positive gets counted twice (once in the row sum and once in the column sum). Therefore, we subtract True Positive once to correct this over-count. Please note that the code for calculating mIoU and IoU based on the confusion matrix is not exactly the same. You may try using the calculation methods from official libraries or your own methods to verify consistency with our evaluation code.

[wujiang0156]

For mIoU, when summing over rows and columns to get the total predicted and actual areas for each class, True Positive gets counted twice (once in the row sum and once in the column sum). Therefore, we subtract True Positive once to correct this over-count. Please note that the code for calculating mIoU and IoU based on the confusion matrix is not exactly the same. You may try using the calculation methods from official libraries or your own methods to verify consistency with our evaluation code.

It's like this, you are too kind, thank you very much! I reproduced your paper, it's fantastic!

[ChenHongruixuan]

It's glad to hear that your question has been solved! Thank you again for your interest to our work :)