How to evaluate on Caltech?

[zhangjiangqige]

Hi, I saw that there are some evaluation scripts under examples/kitti_result, but is seems that these are for kitti? So if I want to evaluate the results on caltech, am I supposed to use run_mscnn_detection.m under examples/caltech to generate the detection results then run the evaluation scripts provided by caltech, i.e. the "Matlab evaluation/labeling code" on http://www.vision.caltech.edu/Image_Datasets/CaltechPedestrians ?

Thanks!

[zhangjiangqige]

I finally managed to work this out... the devkit is so user-unfriendly steps (the working dir of all the following steps is the folder of the devkit):

0. I used the python script to generate detection results in #4 . The provided run_mscnn_detection.m was unreasonably slow for me
1. create a folder "data-USA", and put the "annotations" folder of caltech in it (copy/soft-link/whatever)
2. create a folder "data-USA/res, and place the unzipped results from other algorithms here (http://www.vision.caltech.edu/Image_Datasets/CaltechPedestrians/datasets/USA/res/)
3. put your own results under res following the format of the official results from other algorithms
4. run dbEval, and a folder named "results" will be created to store the generated graphs

FYI: the devkit evaluates 1 based 30,60,90... frames, so in python they are 29, 59, 89... and it is already well handled

[skyforwork]

@neodooth hello I have some questions want to ask you about this project. Could you give me the way to contact with you.Here is my mailbox:972466873@qq.com

[zhangjiangqige]

@skyforwork maybe we'd better discuss here so people with the same questions will be able to read our discussion in the future，哪怕是用中文

[zhaoweicai]

A quick start for caltech pedestrian dataset can be found in Piotr's Toolbox. I used the same format for the detection results, as Piotr's toolbox. For evaluation, you just need to change the directory of the detection results.

[wanghao14]

@neodooth Hello, I have folllowed your guidence as putting my own results under res following the format of the official results from other algorithms. But when I run dbEval, an error occurred:

**Loading detections: C:\Users\whscu\Desktop\WH\SCUT\caltech/results/UsaTest Algorithm #1: Faster-RCNN 索引超出矩阵维度。

出错 dbEval>loadDt (line 484) for f=frames, bb=bbs(bbs(:,1)==f+1,2:6);

出错 dbEval (line 178) dts = loadDt( algs, plotName, aspectRatio );** Have you ever met a similar bug?

[zhangjiangqige]

@JisuiWang Sorry I didn't meet this error, but I think maybe you can inspect the variable bbs, I guess it is empty, probably related to the different path separators in Windows \ and Linux /

[wulixunhua]

@neodooth 你好，我下载了(http://www.vision.caltech.edu/Image_Datasets/CaltechPedestrians/datasets/USA/res/) 提供的结果，打开看到每行的第一个参数为30,30,30,30.....60,60,60,60.....90,90,90,90,90.......。但是我生成的caltech测试集的图片名字是：set06_V001_I00029_usatest.jpg, set06_V001_I00059_usatest.jpg, set06_V001_I00119_usatest.jpg 。是不是写图片名字要写成30,60,90这样子才行？还有个问题就是，测试集的有些图片里面是没有人的，都要送入去测试吗？

[zhangjiangqige]

@wulixunhua 那些要么是坐标，要么是图片id，记不太清了，不用改图片名字。测试集全都要测

[wulixunhua]

@neodooth 刚才正在修改上面的问题，我运行了dbEval.m，在标准HOG和RPN+BF结果下，生成了15个miss rate图，有没有手册介绍这15个结果都对应的哪一种呢？

[zhangjiangqige]

@wulixunhua 我记得只会往固定的一个路径生成图片，那一个图片里会有多个代表不同方法的曲线，没遇到过生成15个图的情况

[wulixunhua]

@neodooth 不是15个，是18个，打开dbEval.m 最上面找到了这些标签，如下：

---

exps = { 'Reasonable', [50 inf], [.65 inf], 0, .5, 1.25 'All', [20 inf], [.2 inf], 0, .5, 1.25 'Scale=large', [100 inf], [inf inf], 0, .5, 1.25 'Scale=near', [80 inf], [inf inf], 0, .5, 1.25 'Scale=medium', [30 80], [inf inf], 0, .5, 1.25 'Scale=far', [20 30], [inf inf], 0, .5, 1.25 'Occ=none', [50 inf], [inf inf], 0, .5, 1.25 'Occ=partial', [50 inf], [.65 1], 0, .5, 1.25 'Occ=heavy', [50 inf], [.2 .65], 0, .5, 1.25 'Ar=all', [50 inf], [inf inf], 0, .5, 1.25 'Ar=typical', [50 inf], [inf inf], .1, .5, 1.25 'Ar=atypical', [50 inf], [inf inf], -.1, .5, 1.25 'Overlap=25', [50 inf], [.65 inf], 0, .25, 1.25 'Overlap=50', [50 inf], [.65 inf], 0, .50, 1.25 'Overlap=75', [50 inf], [.65 inf], 0, .75, 1.25 'Expand=100', [50 inf], [.65 inf], 0, .5, 1.00 'Expand=125', [50 inf], [.65 inf], 0, .5, 1.25 'Expand=150', [50 inf], [.65 inf], 0, .5, 1.50 };

每一个标签对应生成了一张miss rate图，其中这些： “All” ，3个 “Occ” 的参数，三个"Ar"的参数，三个“Expand”的参数，我没明白代表什么意思，可以讲一下吗？谢谢 @neodooth

[lijinxi1314]

dbEval Loading detections: D:\code3.2.1 (1)/results/UsaTest Loading ground truth: D:\code3.2.1 (1)/results/UsaTest Experiment #1: Reasonable Evaluating: D:\code3.2.1 (1)/results/UsaTest The index exceeds the dimension of the matrix.

error: dbEval>plotExps (line 246) stra={res(1,:).stra}; stre={res(:,1).stre}; scores1=round(scores*100);

error: dbEval (line 181) plotExps( res, plotRoc, plotAlg, plotNum, plotName, ... 我在运行dbEval时一直都在出现这个错误，当我把D:\code3.2.1 (1)/results/UsaTest改成D:\code3.2.1 (1)\results\UsaTest时还是相同的错误，您能告诉我怎么解决比较好吗？

[jiulongwong]

Maybe because the result files of the algorithms in /res folder don't match with the labels of the algorithms in dbeval.m. Or you don't even unzip the algorithms result files which file format is zip in /res folder.

[lijinxi1314]

when I run dbEval, an error occurred: error: dbEval>plotExps (line 287) savefig(fName1,1,'pdf','-r300','-fonts'); close(1); f1=[fName1 '.pdf'];

error: dbEval (line 181) plotExps( res, plotRoc, plotAlg, plotNum, plotName, ...

I can't get my UsaTestRocReasonable.pdf, so how can I handle this problem? thank you

[LewX]

@neodooth Hello, I have folllowed your guidence as putting my own results under res following the format of the official results from other algorithms. But when I run dbEval, an error occurred:

**Loading detections: C:\Users\whscu\Desktop\WH\SCUT\caltech/results/UsaTest Algorithm #1: Faster-RCNN 索引超出矩阵维度。

出错 dbEval>loadDt (line 484) for f=frames, bb=bbs(bbs(:,1)==f+1,2:6);

出错 dbEval (line 178) dts = loadDt( algs, plotName, aspectRatio );** Have you ever met a similar bug?

请问这个错误已经解决了吗？

[wanghao14]

@neodooth Hello, I have folllowed your guidence as putting my own results under res following the format of the official results from other algorithms. But when I run dbEval, an error occurred: Loading detections: C:\Users\whscu\Desktop\WH\SCUT\caltech/results/UsaTest Algorithm #1: Faster-RCNN 索引超出矩阵维度。 出错 dbEval>loadDt (line 484) for f=frames, bb=bbs(bbs(:,1)==f+1,2:6); 出错 dbEval (line 178) dts = loadDt( algs, plotName, aspectRatio ); Have you ever met a similar bug?

请问这个错误已经解决了吗？

这是好久之前做的内容了，我记得错误的原因是在于如果有一个文件夹的图都没有被检测到人的话，那么matlab对应的格子里面的内容就是空的，但这样会导致这个错误的出现。所以你可以考虑往那一组都为空的格子里填上点东西，“装作”检测算法检测到了一些人。

[thhung]

@neodooth Could you help me a description for the output format in step3: "put your own results under res following the format of the official results from other algorithms"? Thanks.

[hdjsjyl]

@LewX @neodooth 请问你知道怎样用新的annotation来做evaluation吗

[Egrass]

@neodooth 你好，我下载了(http://www.vision.caltech.edu/Image_Datasets/CaltechPedestrians/datasets/USA/res/) 提供的结果，打开看到每行的第一个参数为30,30,30,30.....60,60,60,60.....90,90,90,90,90.......。但是我生成的caltech测试集的图片名字是：set06_V001_I00029_usatest.jpg, set06_V001_I00059_usatest.jpg, set06_V001_I00119_usatest.jpg 。是不是写图片名字要写成30,60,90这样子才行？还有个问题就是，测试集的有些图片里面是没有人的，都要送入去测试吗

你好！请问你下载的关于Caltech的对比结果还有吗？我根据你的连接现在下载不了，可以提供一份给我吗？