Closed zchrissirhcz closed 6 years ago
@zchrissirhcz I guess that you have used the wrong wrappers. You should use the function of boundaryBench in the BSDS500 benchmark to evaluate the produced edges. Moreover, the edge thinning has been performed in the function of _evaluation_bdryimage using bwmorph function in matlab.
Thanks for your reply. Now I can get correct evaluation result. I'll put my steps here.
Now I use the downloaded rcf_pretrained_bsds.caffemodel
to generated multi-scaled results, saved as *_fuse.png
.
I use your provides edge_nms.m
script, and find it cannot use edgesNms
funtion, thus I move it to Piotr Dollar's edges
folder, i.e. /opt/work/piotr/edges
, and change this line:
edge = edgesNms(edge, O, 2, 5, 1.01, 8);
to
edge = edgesNmsMex(edge, O, 2, 5, 1.01, 8);
Then run it to generate nms-ed results.
Note, I've also downloaded Piotr Dollar's toolbox
and both these two folders(and children folders) are added to path.
I downloaded http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/BSR/BSR_full.tgz
, and by editing bench_bsds500.m
to these lines:
addpath benchmarks
clear all;close all;clc;
imgDir = '../BSDS500/data/images/test';
gtDir = '../BSDS500/data/groundTruth/test';
%inDir = '../BSDS500/ucm2/test';
%outDir = '../BSDS500/ucm2/test_eval';
inDir = '/opt/data/HED-BSDS/test-fcn-nms';
outDir = '/opt/data/HED-BSDS/RCF-eval';
%mkdir(outDir);
% running all the benchmarks can take several hours.
tic;
%allBench(imgDir, gtDir, inDir, outDir)
boundaryBench(imgDir, gtDir, inDir, outDir);
toc;
plot_eval(outDir);
Then I run that script, and after a long time(several hours) waiting, some .txt
files are generated, including eval_bdry.txt
and eval_bdry_thr.txt
.
I thought eval_bdry.txt
and eval_bdry_thr.txt
can be put under your provided plotting code folder, and I run https://github.com/yun-liu/plot-edge-pr-curves/blob/master/plot_bsds.m.
However, errors just occurs. Notice that there are exactly 7 numbers (7 columns) in *_bdry.txt
in plot-edge-pr-curves/edge_eval_bsds
, and in BSR/bench/benchmarks/collect_eval_bdry.m
, there is:
fname = fullfile(pbDir,'eval_bdry.txt');
fid = fopen(fname,'w');
if fid==-1,
error('Could not open file %s for writing.',fname);
end
fprintf(fid,'%10g %10g %10g %10g %10g %10g %10g %10g\n',bestT,bestR,bestP,bestF,R_max,P_max,F_max,Area_PR);
fclose(fid);
i.e. *The BSDS boundary benchmark will generate eval_bdry.txt
with exactly 8 numbers, instead of 7.
Similarly, my generated eval_bdry_thrs.txt
have 4 columns, and your provides `_bdry_thr.txt` have 3 columns.**
eval_bdry.txt
and eval_bdry_thr.txt
This is claimed in https://github.com/yun-liu/plot-edge-pr-curves/blob/master/README.md
modify and run plot-edge-pr-curves/plot_bdsd.m
Now I get a nearly exactly same result:
ODS=0.806 OIS=0.823 AP=0.839 R50=0.914 - RCF (2017)
ODS=0.811 OIS=0.830 AP=0.846 R50=0.913 - RCF-MS (2017)
ODS=0.811 OIS=0.831 AP=0.847 R50=0.916 - MyRCF (2018)
MyRCF_bdry.txt:
0.796388 0.826376 0.811105 0.823153 0.838022 0.830521 0.846884
MyRCF_bdry_thr.txt:
0.916885 0.481511 0.631423
0.913225 0.533613 0.673619
0.911576 0.552105 0.687699
0.909399 0.571562 0.701947
0.908081 0.581843 0.709245
0.906144 0.59488 0.718239
0.904932 0.602573 0.72343
0.903104 0.612941 0.730254
0.901829 0.619326 0.734345
0.89963 0.628464 0.739987
0.897387 0.637358 0.745345
0.89596 0.642711 0.748495
0.893796 0.6503 0.752849
0.89235 0.654891 0.755399
0.890059 0.661843 0.759171
0.888551 0.666303 0.761543
0.886357 0.672549 0.764791
0.884839 0.676687 0.76689
0.882561 0.682873 0.769981
0.881107 0.686943 0.772004
0.878687 0.693304 0.775065
0.876151 0.699512 0.777931
0.874373 0.703607 0.77975
0.871617 0.709732 0.782388
0.869685 0.713748 0.784038
0.866786 0.719888 0.786536
0.864841 0.723922 0.788132
0.8619 0.729702 0.790311
0.859778 0.73352 0.791646
0.856537 0.739282 0.793602
0.853173 0.744997 0.795424
0.851031 0.74876 0.796626
0.847647 0.754523 0.798379
0.845409 0.758379 0.799533
0.842058 0.763776 0.801009
0.839689 0.767495 0.801971
0.836156 0.773108 0.803397
0.833805 0.776892 0.804343
0.830269 0.782603 0.805732
0.827745 0.786396 0.806541
0.823914 0.792099 0.807694
0.819892 0.797895 0.808744
0.817159 0.801629 0.80932
0.812873 0.807348 0.810101
0.809865 0.811157 0.81051
0.80516 0.816768 0.810922
0.801654 0.820633 0.811032
0.796388 0.826376 0.811105
0.792854 0.830137 0.811067
0.787342 0.835859 0.810875
0.781505 0.841581 0.810432
0.777526 0.845342 0.810017
0.771536 0.850996 0.809321
0.767256 0.854771 0.808653
0.760605 0.860382 0.807422
0.755853 0.864141 0.806378
0.748499 0.86977 0.80459
0.743674 0.873558 0.8034
0.736299 0.879468 0.801541
0.731124 0.883238 0.800014
0.722958 0.888869 0.797375
0.714331 0.894332 0.794261
0.70829 0.897902 0.791904
0.698607 0.902929 0.787735
0.692028 0.906222 0.784772
0.681991 0.911481 0.78021
0.674913 0.914805 0.776759
0.66391 0.91985 0.7712
0.656116 0.923474 0.767169
0.643958 0.928576 0.76051
0.630972 0.933331 0.75293
0.621734 0.936501 0.747326
0.607383 0.941441 0.738386
0.597245 0.944526 0.731773
0.581938 0.948922 0.721443
0.570807 0.951668 0.713599
0.553205 0.955427 0.700697
0.540565 0.958061 0.691159
0.51997 0.961771 0.675006
0.505487 0.964148 0.663245
0.482445 0.967519 0.643843
0.458149 0.970794 0.622514
0.441003 0.972765 0.606878
0.413966 0.976042 0.581361
0.394876 0.977813 0.562568
0.364598 0.981287 0.531658
0.34286 0.983228 0.508428
0.30815 0.986119 0.469567
0.283816 0.988241 0.440984
0.244985 0.991295 0.392876
0.20395 0.993577 0.338431
0.176363 0.994982 0.299618
0.134409 0.996868 0.236879
0.106832 0.998216 0.193007
0.0671202 0.999695 0.125794
0.0440549 0.999961 0.0843918
0.0173671 1 0.0341413
0.00601356 1 0.0119552
0.000150339 1 0.000300633
First I would appreciate the released repos of RCF and related ploting codes.
I've trained the RCF edge detector on BSDS500 dataset (yes, originally, instead of my own dataset, and with default training hyper-parameters). Now evaluating on BSDS500 benchmark, have some problems.
As metioned in
readme.md
, non-maximum suppression and edge thinning, are two acquired steps before evaluation. I installed Piotr Dollar's toolbox and edge detection code, and successfully get the results of nms-ed edges, saved in png images.However, whether doing edge thinning or not, the BSDS500 benchmark's code, always give complains that the indexes are 0, which should be greater than or equal to 1. I notice that this is caused by, the automatically generated thresholding values are from 0.01 to higher ones, and the pixel values after binarization (via
bwlabel
in matlab) contains 0 values.Wondering if you have encounterd same problem, if so, would you provide some suggestions? Thank you.