Question about the metric on DAD datasets

[muse1998]

Hello, recently I am also engaged in research related to driver behavior recognition. I want to confirm whether the metric you are using on the DAD benchmark is the top1 accuracy or the average top1 accuracy of all classes. Due to the unbalanced problem in the DAD dataset, there is a large gap in the values ​​of these two metrics. Based on the code you provided, I'm having trouble achieving 65% average top1 accuracy on the DAD dataset

[muse1998]

Besides, according to the figures you provided in the paper, you use the RGB video to conduct all the experiments on DAD datasets, however the other results for comparision are obtained from the NIR video, so it may not be a fair comparision.

[Dichao-Liu]

@muse1998 Hi, thank you very much for the questions.

Hello, recently I am also engaged in research related to driver behavior recognition. I want to confirm whether the metric you are using on the DAD benchmark is the top1 accuracy or the average top1 accuracy of all classes. Due to the unbalanced problem in the DAD dataset, there is a large gap in the values ​​of these two metrics. Based on the code you provided, I'm having trouble achieving 65% average top1 accuracy on the DAD dataset

We used the video recognition framework provided by Hara et al. for training, which seems to use top-1 accuracy. Sorry for the misunderstanding. We will remark on this in the GitHub repository to avoid misunderstanding. Again, thanks for the question. Also, we recommend you to use the framework provided by Hara et al. for training, which performed the best in our experiments.

Besides, according to the figures you provided in the paper, you use the RGB video to conduct all the experiments on DAD datasets, however the other results for comparision are obtained from the NIR video, so it may not be a fair comparision.

DAD is a multi-modal video dataset, including NIR, Depth, and Color data. And the methods based on a different modal for the same task are also compared together by the dataset authors.

If you have any other questions, please feel free to ask me here.

[muse1998]

Thanks for the reply, that's very helpful! Since the baseline provided in DAD is calculated by the mean accuracy of all the classes, could you share the mean accuracy results of your method so that I can cite your paper and make a fair comparision in my own paper?

[muse1998]

[Dichao-Liu]

Thanks for the reply, that's very helpful! Since the baseline provided in DAD is calculated by the mean accuracy of all the classes, could you share the mean accuracy results of your method so that I can cite your paper and make a fair comparision in my own paper?

@muse1998 Hi, sincerely thank you very much for the information. But I'm sorry, now I don't have the average accuracy on hand because I've graduated from university for a long time, and the pre-processed video frames and pre-trained weights are not saved. I may reimplement the experiments when I would be free from the work of my company, which would take a huge time for me. My current suggestion is, in addition to the table of average accuracy, could you please add a small section in your paper to compare ordinary accuracy with ours to show the superiority of your method?

For example, in the Tables 3~5 of Qin et al.'s paper and the Table 8 of our paper, we compared the methods on SFD3 dataset. Since the previous methods on this dataset have different evaluation settings, we also showed the performance of our methods with different settings.

[zmttttttt]

Hello, recently I am also engaged in research related to driver behavior recognition. I want to confirm whether the metric you are using on the DAD benchmark is the top1 accuracy or the average top1 accuracy of all classes. Due to the unbalanced problem in the DAD dataset, there is a large gap in the values ​​of these two metrics. Based on the code you provided, I'm having trouble achieving 65% average top1 accuracy on the DAD dataset Hello, may I ask if the AUC data results of this paper have been reproduced? My results differ from those of the authors, and I cannot achieve an accuracy of over 90%. If it's convenient, can you provide a way to contact you? My contact information is qq:1845212417. Thank you.

[Dichao-Liu]

@zmttttttt Hi, the AUCD2 has two versions. Please make sure to use AUCD2 V1 rather than V2, the former of which has been used in most current studies, including our work and other researchers' work as we cited. Achieving an accuracy of over 90% on AUCD2 V1 should be easy.

Hello, may I ask if the AUC data results of this paper have been reproduced? My results differ from those of the authors, and I cannot achieve an accuracy of over 90%. If it's convenient, can you provide a way to contact you? My contact information is qq:1845212417. Thank you.

[zmttttttt]

Dear Dichao ,

Thank you very much for your helpful suggestions. I really appreciate your guidance . Your insights have been very valuable to me and have helped me improve my work.

Once again, thank you for taking the time to provide feedback and advice. I am grateful for your help and wish you all the best.

Best regards, mengting

vivid @.***

 

------------------ 原始邮件 ------------------ 发件人: "Dichao-Liu/Lightweight_Distracted_Driver_Recognition_with_Distillation-Based_NAS_and_Knowledge_Transfer" @.>; 发送时间: 2023年4月17日(星期一) 中午11:36 @.>; @.**@.>; 主题: Re: [Dichao-Liu/Lightweight_Distracted_Driver_Recognition_with_Distillation-Based_NAS_and_Knowledge_Transfer] Question about the metric on DAD datasets (Issue #1)

@zmttttttt Hi, the AUCD2 has two versions. Please make sure to use AUCD2 V1 rather than V2, the former of which has been used in most current studies, including our work and other researchers' work as we cited. Achieving an accuracy of over 90% on AUCD2 V1 should be easy.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>