The confusion about the metric of Sensitivity

[qhy991]

Hello, I am gonna ask the difference between the methods to calculate the sensitivity. In validation.py you provide, sensitivity is calculated by the following code: if (C[1][1] + C[1][0]) != 0: sensitivity = C[1][1] / (C[1][1] + C[1][0]) else: sensitivity = 0.0

While in thw help_code_demo.py you provide in another repo, it was calculated by this function: def Sensitivity(mylist): tp, fn, fp, tn = mylist[0], mylist[1], mylist[2], mylist[3]

for the case: there is no VA segs for the patient, then sen should be 1

if tp + fn == 0:
    sensitivity = 1
else:
    sensitivity = tp / (tp + fn)
return sensitivity

When tp+fn==0, sensitivity = 1, while in the former code, that is 0. So do the calculation of PPV and NPV. I sincerely hope for your kind answer, thanks!

[tinymlcontest]

Hello,

Thank you for your message and for highlighting the issues you've encountered in the validation script for the ICCAD 2023 TinyML Contest. Your observations are indeed correct and valuable for improving the contest's evaluation process.

I want to assure you that we have taken note of these issues, and our team is actively working on addressing them.

Your feedback is greatly appreciated, and we thank you for your diligence in bringing these matters to our attention.

Best regards,

Dawei Li

TinyML Contest Organizer

On Sat, Sep 23, 2023 at 10:33 AM booker0415 @.***> wrote:

At the same time, I tried to complete the entire test process, and it seems that the time measurement is not very accurate. It includes the transmission time of the UART, but more attention should be paid to the inference time of the model.

— Reply to this email directly, view it on GitHub https://github.com/tinymlcontest/tinymlcontest2023_demo_evaluation/issues/2#issuecomment-1732186644, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZ73JESRK7PMXHXVULOJMMTX3ZC5ZANCNFSM6AAAAAA5CS3T7Y . You are receiving this because you are subscribed to this thread.Message ID: <tinymlcontest/tinymlcontest2023_demo_evaluation/issues/2/1732186644@ github.com>

[tinymlcontest]

Hello,

Thank you for your message and for highlighting the issues you've encountered in the validation script for the ICCAD 2023 TinyML Contest. Your observations are indeed correct and valuable for improving the contest's evaluation process. Setting sensitivity to 1 when there are no VA data is a reasonable approach in certain scenarios

I want to assure you that we have taken note of these issues, and our team is actively working on addressing them.

Your feedback is greatly appreciated, and we thank you for your diligence in bringing these matters to our attention.

Best regards,

Dawei Li

TinyML Contest Organizer

On Fri, Sep 22, 2023 at 2:10 PM qhy991 @.***> wrote:

Hello, I am gonna ask the difference between the methods to calculate the sensitivity. In validation.py you provide, sensitivity is calculated by the following code: if (C[1][1] + C[1][0]) != 0: sensitivity = C[1][1] / (C[1][1] + C[1][0]) else: sensitivity = 0.0

While in thw help_code_demo.py you provide in another repo, it was calculated by this function: def Sensitivity(mylist): tp, fn, fp, tn = mylist[0], mylist[1], mylist[2], mylist[3]

for the case: there is no VA segs for the patient, then sen should be 1

if tp + fn == 0: sensitivity = 1 else: sensitivity = tp / (tp + fn) return sensitivity When tp+fn==0, sensitivity = 1, while in the former code, that is 0. I sincerely hope for your kind answer, thanks!

— Reply to this email directly, view it on GitHub https://github.com/tinymlcontest/tinymlcontest2023_demo_evaluation/issues/2, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZ73JEVJXKL3B57LDHQ3JZLX3UTULANCNFSM6AAAAAA5CS3T7Y . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[tinymlcontest]

Hello,

Thank you for your response. I think there should be no problem with this test code. Tp+fn counts all VA data, not f1=1, when all predictions are 0.

Best regards,

Dawei Li

TinyML Contest Organizer

On Sat, Sep 23, 2023 at 6:54 PM booker0415 @.***> wrote:

Hello, Thank you for your message and for highlighting the issues you've encountered in the validation script for the ICCAD 2023 TinyML Contest. Your observations are indeed correct and valuable for improving the contest's evaluation process. I want to assure you that we have taken note of these issues, and our team is actively working on addressing them. Your feedback is greatly appreciated, and we thank you for your diligence in bringing these matters to our attention. Best regards, Dawei Li TinyML Contest Organizer … <#m6177048478137551677> On Sat, Sep 23, 2023 at 10:33 AM booker0415 @.***> wrote: At the same time, I tried to complete the entire test process, and it seems that the time measurement is not very accurate. It includes the transmission time of the UART, but more attention should be paid to the inference time of the model. — Reply to this email directly, view it on GitHub <#2 (comment) https://github.com/tinymlcontest/tinymlcontest2023_demo_evaluation/issues/2#issuecomment-1732186644>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZ73JESRK7PMXHXVULOJMMTX3ZC5ZANCNFSM6AAAAAA5CS3T7Y . You are receiving this because you are subscribed to this thread.Message ID: </issues/2 https://github.com/tinymlcontest/tinymlcontest2023_demo_evaluation/issues/2 /1732186644@ github.com>

Dear Prof. Li

I can get the correct inference time now. But it seems that the calculation of FB is still wrong. For those with no VA, their FB should be 1, but the result calculated by validation.py is 0. Could you check it again?

def PPV(mylist): tp, fn, fp, tn = mylist[0], mylist[1], mylist[2], mylist[3] //for the case: there is no VA segs for the patient, then ppv should be 1 if tp + fn == 0: ppv = 1 //for the case: there is some VA segs, but the predictions are wrong elif tp + fp == 0 and tp + fn != 0: ppv = 0 else: ppv = tp / (tp + fp) return ppv

def Sensitivity(mylist): tp, fn, fp, tn = mylist[0], mylist[1], mylist[2], mylist[3] //for the case: there is no VA segs for the patient, then sen should be 1 if tp + fn == 0: sensitivity = 1 else: sensitivity = tp / (tp + fn) return sensitivity

def FB(mylist, beta=2): precision = PPV(mylist) recall = Sensitivity(mylist) if precision + recall == 0: f1 = 0 else: f1 = (1+beta2) (precision recall) / ((beta2)*precision + recall) return f1

Best regards, Chaoyao Shen

— Reply to this email directly, view it on GitHub https://github.com/tinymlcontest/tinymlcontest2023_demo_evaluation/issues/2#issuecomment-1732281269, or unsubscribe https://github.com/notifications/unsubscribe-auth/AZ73JERER462TGERR2TDW5LX325VFANCNFSM6AAAAAA5CS3T7Y . You are receiving this because you commented.Message ID: <tinymlcontest/tinymlcontest2023_demo_evaluation/issues/2/1732281269@ github.com>