How to use eval_tool?

[ftlong6666]

Hello @fyangneil could you explain the step how to do evaluation?,

1. should we convert all the predicted .png file into .mat?
2. How to setup your toolbox? I always got this error.

Error in crack_nms (line 11)
    load(fullfile(path_to_input, iids(i).name));

3. What should we do first? I tried to compile the toolbox successfully and modify the code into this.

   mkdir(path_to_output);
   iids = dir(fullfile(path_to_input, '*.png'));
   for i = 1:length(iids)
   %edge = imread(fullfile(path_to_input, iids(i).name));\
   %load(fullfile(path_to_input, iids(i).name));
   %predmap=predmap;
   %disp(fullfile(path_to_input, iids(i).name))
   filename = fullfile(path_to_input, iids(i).name);
   predmap = imread(filename);
   predmap = rgb2gray(predmap);
   [Ox, Oy] = gradient2(convTri(predmap, 4));
   [Oxx, ~] = gradient2(Ox);
   [Oxy, Oyy] = gradient2(Oy);
   O = mod(atan(Oyy .* sign(-Oxy) ./ (Oxx + 1e-5)), pi);
   predmap = edgesNmsMex(predmap, O, 2, 5, 1.01, 8);
   imwrite(predmap, fullfile(path_to_output, [iids(i).name(1:end-4) '.png']));
   save(fullfile(path_to_output, [iids(i).name(1:end-4) '.mat']),'predmap');
   end

However I got this error. Suppose that I have the predicted crack in .png file in the input_folder and I want to get the evaluation result only.

Error using convConst
A must be a 4x4 or bigger 2D or 3D float array.

Error in convTri (line 87)
    J = convConst('convTri',I,r,s);

Error in crack_nms (line 17)
    [Ox, Oy] = gradient2(convTri(predmap, 4));

[fyangneil]

@ftlong6666

1. you should convert to 'mat' file and refer to the results we uploaded to Google Drive or Baidu Yunpan.
2. You can use our dataset and provided results to test if the tool can work or not. If everything goes well, you should get the results reported in our paper.
3. Before using the tool, please make sure that in predicted crack map the bright regions are crack, background is black. To get ODS and OIS, run 'crack_nms.m' first then 'crack_eval.m'. To get AIU, run 'crack_AIU.m'..

[ftlong6666]

Hi I have tried it and I still got this error.

Error using convConst
A must be a 4x4 or bigger 2D or 3D float array.

Error in convTri (line 87)
    J = convConst('convTri',I,r,s);

Error in crack_nms (line 14)
    [Ox, Oy] = gradient2(convTri(predmap, 4));

Also how to change the path of Ground truth?, wherepath_to_input is the path of result.

[fyangneil]

@ftlong6666 Could you provide more details of how you use the tool. path_to_input is the directory to the crack prediction results (mat files).

[ftlong6666]

@fyangneil After I check I realized that the testdata is not in mat file. Here is the step to produce the error.

1. Extract your test data and fphb result
2. Change the path_to_input to the fphb location (the result in mat)
3. run the crack_nms. However here is the things: I don't know how to point the program to ground-truh location, and your testing groundtruth has no .mat file, so I need to convert to mat first. Do you have a code to convert all of the .png files into mat? Thank you

[ftlong6666]

Also I confuse with your code in file crack_nms.m, in line 9 predmap=predmap; where the predmap is not assign first.

[fyangneil]

@to run crack_nms, you do not need the ground truth files, only crack prediction files are needed.

[fyangneil]

@ftlong6666 the predmap is from line 8 "load(fullfile(path_to_input, iids(i).name))"

[ftlong6666]

@fyangneil Thank you for quick reply it helps a lot. I have changed it to predmap = load(fullfile(path_to_input, iids(i).name)); but I got this error.

Error using double
Conversion to double from struct is not possible.

Error in convn (line 27)
    A = double(A);

Error in convTri (line 93)
  J = convn(convn(J,f,'valid'),f','valid');

Error in crack_nms (line 15)
    [Ox, Oy] = gradient2(convTri(predmap, 4));

[fyangneil]

@ftlong6666 you do not need to change the line 9, it should work.

[ftlong6666]

@fyangneil , I used exactly the same code here is the full code.

clear; clc;
path_to_input='../fphb_testresult/hed_fuse_fpn_ada_python_v1_iter_12000';
path_to_output='../fphb_testresult/hed_fuse_fpn_ada_python_v1_iter_12000/nms';
mkdir(path_to_output);
iids = dir(fullfile(path_to_input, '*.mat'));
for i = 1:length(iids)
    %edge = imread(fullfile(path_to_input, iids(i).name));\
    load(fullfile(path_to_input, iids(i).name));
    predmap=predmap;
    [Ox, Oy] = gradient2(convTri(predmap, 4));
    [Oxx, ~] = gradient2(Ox);
    [Oxy, Oyy] = gradient2(Oy);
    O = mod(atan(Oyy .* sign(-Oxy) ./ (Oxx + 1e-5)), pi);
    predmap = edgesNmsMex(predmap, O, 2, 5, 1.01, 8);
    imwrite(predmap, fullfile(path_to_output, [iids(i).name(1:end-4) '.png']));
    save(fullfile(path_to_output, [iids(i).name(1:end-4) '.mat']),'predmap');
end

But I still got this error message.

Error using convConst
A must be a 4x4 or bigger 2D or 3D float array.

Error in convTri (line 87)
    J = convConst('convTri',I,r,s);

Error in crack_nms (line 10)
    [Ox, Oy] = gradient2(convTri(predmap, 4));

/fphb_testresult/hed_fuse_fpn_ada_python_v1_iter_12000 is the folder which contains your result.

[ftlong6666]

Here is the workspace.

[fyangneil]

@ftlong6666 what is the version of your matlab

[ftlong6666]

Matlab R2019b Windows 10

[fyangneil]

@ftlong6666 could you show the predmap?

[ftlong6666]

@fyangneil Here is the predmap variables.

[fyangneil]

@ftlong6666 looks good, it should be fine.Could you set a breakpoint in " J = convConst('convTri',I,r,s);"

[ftlong6666]

Sure here is the result. Also the content of variables. Here is the l.

[fyangneil]

@ftlong6666 have compiled the edges toolbox?

[ftlong6666]

Sure I have compiled it before.

[ftlong6666]

I do this one to compile.

Simply unzip, then add all directories to the Matlab path:
  >> addpath(genpath('/edges/toolbox/')); savepath;

If needed, run the compile script for the mex files:
  >> toolboxCompile;

[fyangneil]

Looks good. I have no idea

On Thu, Mar 5, 2020 at 12:53 PM ftlong6666 notifications@github.com wrote:

I do this one to compile.

Simply unzip, then add all directories to the Matlab path:

addpath(genpath('/edges/toolbox/')); savepath;

If needed, run the compile script for the mex files:

toolboxCompile;

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fyangneil/pavement-crack-detection/issues/10?email_source=notifications&email_token=ALAUWTTZOUA2TZ2WHSRODPDRF7RJHA5CNFSM4LA43X62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN6IBHI#issuecomment-595361949, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALAUWTXXSV6M3LPOVJ7RW7LRF7RJHANCNFSM4LA43X6Q .

[ftlong6666]

1. What is your Matlab version?
2. Do you have a code to convert .png result into .mat?
3. Why we don't need the ground truth?, we should compare the result against the ground truth right?

[fyangneil]

For nms, you need groundtruth I do not have code to convert png to mat, you can do it using matlab. My matlab version is 2017b. But I think it has no business with the matlab version

On Thu, Mar 5, 2020 at 1:01 PM ftlong6666 notifications@github.com wrote:

1. What is your Matlab version?
2. Do you have a code to convert .png result into .mat?
3. Why we don't need the ground truth?, we should compare the result against the ground truth right?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fyangneil/pavement-crack-detection/issues/10?email_source=notifications&email_token=ALAUWTTYMESA6JS4ITTH5GTRF7SJDA5CNFSM4LA43X62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEN6I7YQ#issuecomment-595365858, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALAUWTRIAEB722NWWUVJKLLRF7SJDANCNFSM4LA43X6Q .

[ftlong6666]

@fyangneil You mentioned that to run nms_crack.m we don't need ground-truth. So if we need it, in which part should I change the path? Also what is the dimension of the mat file?, is it in HxWx3 or HxWx1 or HxW?

[fyangneil]

@ftlong6666 for the deimension of mat file, you can check our provided results. When you run crack_eval.m and crack_AIU.m, you need ground truth files.

[ftlong6666]

@fyangneil If you don't have the code to convert .png to .mat, do you have the .mat of the groundtruth?, as long as you have the result, you should have the .mat of the groundtruth right?, in case you have, could you also upload it? Thank you

[ftlong6666]

Hello @fyangneil its just working, I don't know why. Anyway after I run the nms_crack.m , I try to run crack_eval.m . But I got this error, do you know why?

Index exceeds the number of array elements (0).

Error in edgesEvalDir_crack (line 68)
for i=1:n, id=ids{i}(1:end-4);idRes=idsRes{i}(1:end-4);

Error in try_old2 (line 5)
edgesEvalDir_crack('resDir',resDir,'gtDir',gtDir, 'thin', 1, 'pDistr',{{'type','parfor'}},'maxDist',0.0075);

Also in your code I see that you point it to directory testcrop folder, but I also see testdata folder, so which one is used? Also I realized that both of them don't have .mat files.

[fyangneil]

@ftlong6666 regarding crack_eval, only png image are needed. After nms, you can get png files, which used for evaluation. The ground truth file used for evaluation is also png files. The images in testcrop folder are the cropped chips from images in testdata. For details, you are referred to our paper (FPHBN).

[ftlong6666]

@fyangneil yes got it, but the problem is your code still looking for ".mat" files. First I put this line in edgesEvalImg.m in line 54 so the full code will be like this.

% load edges (E) and ground truth (G)
disp(E);
if(all(ischar(E))), E=double(imread(E))/255; end
G=load(G); G=G.groundTruth; n=length(G);
for g=1:n, G{g}=double(G{g}.Boundaries); end

Then after I run it, the result is still an error message like this.

FPHB
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_081839_1281_1.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_114759_1281_1081.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_114759_1_361.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_163940_1_1081.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_164000_1281_1081.mat
Error using imread>get_format_info (line 541)
Unable to determine the file format.

Error in imread (line 389)
        fmt_s = get_format_info(fullname);

Error in edgesEvalImg (line 55)
if(all(ischar(E))), E=double(imread(E))/255; end

Error in fevalDistr>fedParfor (line 99)
parfor i=1:nJob, r=feval(funNm,jobs{i}{:});

Error in fevalDistr (line 80)
  case 'parfor',    [out,res]=fedParfor(funNm,jobs,store);

Error in edgesEvalDir_crack (line 74)
fevalDistr('edgesEvalImg',jobs(do),p.pDistr{:});

Error in try_old2 (line 5)
edgesEvalDir_crack('resDir',resDir,'gtDir',gtDir, 'thin', 1, 'pDistr',{{'type','parfor'}},'maxDist',0.0075);

Thus the error said that your code crack_eval.m is still looking for mat file. I am using your result that you provided in Google drive. Do you know what I am missing in here?, from the error I can see that your code try to load in image, but the path is still .mat .

[ftlong6666]

@fyangneil Are you sure use this code to provide the result?, because I just realize that the path of the result needs .png but the groundtruth should be in .mat. Let me show you. In the code edgesEvalDir.m line 70 you type this one.

im1=fullfile(resDir,[idRes '.mat']); gt1=fullfile(gtDir,[id '.png']);

but in edgesEvalImg.m line 53 if we print out the path, the E is the mat and G is the .png.

% load edges (E) and ground truth (G)
disp(E);
disp(G);
if(all(ischar(E))), E=double(imread(E))/255; end
G=load(G); G=G.groundTruth; n=length(G);
for g=1:n, G{g}=double(G{g}.Boundaries); end

so here is the result.

FPHB
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_081839_1281_1.mat
..\testcrop2\20160222_081839_1281_1.png
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_114759_1281_1081.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_114759_1_361.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_115305_1921_361.mat
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_163940_1_1081.mat
..\testcrop2\20160222_163940_1_1081.png
..\fphb_testresult\hed_fuse_fpn_ada_python_v1_iter_12000\nms1\20160222_164000_641_1.mat
Error using imread>get_format_info (line 541)
Unable to determine the file format.

Error in imread (line 389)
        fmt_s = get_format_info(fullname);

Error in edgesEvalImg (line 56)
if(all(ischar(E))), E=double(imread(E))/255; end

Error in fevalDistr>fedParfor (line 99)
parfor i=1:nJob, r=feval(funNm,jobs{i}{:});

Error in fevalDistr (line 80)
  case 'parfor',    [out,res]=fedParfor(funNm,jobs,store);

Error in edgesEvalDir_crack (line 75)
fevalDistr('edgesEvalImg',jobs(do),p.pDistr{:});

Error in try_old2 (line 6)
edgesEvalDir_crack('resDir',resDir,'gtDir',gtDir, 'thin', 1, 'pDistr',{{'type','parfor'}},'maxDist',0.0075);

From here we can see that you try to read .mat files with imread, and read groundtruth .png with load which is wrong. If I change all of the GT to .mat files, there is no fields groundtruthand Boundaries. Yes this make sense that this code originally used to evaluate the edge detection from BSDS but since you have ground truth by your self, you should format it in the same field as in BSDS. Please could you help me how to do the evaluation?, I critically found that this is very crucial part, because its related with the comparison with another method. Thank you

[fyangneil]

@ftlong6666 my code does not use 'edgesEvalImg.m', it uses edgesEvalDir_crack.m, the line 70 is 'im1=fullfile(resDir,[idRes '.mat']); gt1=fullfile(gtDir,[id '.png']);' . So my code use mat file for prediction results and png file for ground truth. '

[ftlong6666]

@fyangneil Yes I also use it but the edgesEvalDir_crack.m also point to edgesEvalImg as stated in line 72 in edgesEvalImg(jobs{i}{:}); (edgesEvalDir_Crack.m line 72). should I change it to another function?

for i=1:n, id=ids{i}(1:end-4);idRes=idsRes{i}(1:end-4);
  res{i}=fullfile(evalDir,[id '_ev1.txt']); do(i)=~exist(res{i},'file');
  im1=fullfile(resDir,[idRes '.mat']); gt1=fullfile(gtDir,[id '.png']);
  jobs{i}={im1,gt1,'out',res{i},'thrs',p.thrs,'maxDist',p.maxDist,...
    'thin',p.thin}; if(0), edgesEvalImg(jobs{i}{:}); end
end

[fyangneil]

@ftlong6666 in my code, the line 56 in edgesEvalImg is load(E).

[ftlong6666]

@fyangneil Hi finally I resolved all the issues. Actually I run your code but because I have tested the edge detection before and my matlab point to my previous experiment, so it point to the original edge detection metric code because the function name is simillar. After I used the tescrop data I got this result.

Starting parallel pool (parpool) using the 'local' profile ...
Connected to the parallel pool (number of workers: 8).
ODS=0.604 OIS=0.635 AP=0.363 R50=0.677 - FPHB
AIU 0.4299
IdleTimeout has been reached.
Parallel pool using the 'local' profile is shutting down.

From here I got the same OIS and ODS in the paper. However in here my AIU is 0.4299 which is lower than the paper which is 0.489. I got your pre-computed result that you shared in Gdrive. BTW is there any mistake that I have done in here?, Also how to the plot of the AIU, Precission, and Recall? Thank you so much for your help this 2 days, it help a lot :) .

[fyangneil]

@ftlong6666 To compute AIU, you do not need nms, just use the provided prediction results. To display the plot of curve, you can use edgesEvalPlot.

[ftlong6666]

@fyangneil Hi it works, thank you so much for your help.

[ftlong6666]

@fyangneil Sorry last question. Do you have others method results that you presented in your paper? such as HED,RCF,FCN, and CrackForest? I only found your method results in your drive. Thanks a lot for your help.

[fyangneil]

@ftlong6666 regarding the results of other method, i just upload the CrackForest results on google drive. For the other method's results, you may need to run their codes. Since it is a long time ago that i conducted the experiments, the results cannot be found anymore.

[ftlong6666]

Hello @fyangneil, thank you for your support, now I have 3 questions for the evaluation.

1. What is the input image size for FPHB network?, since its fully convolutional it should be not fixed right. However if we use original size it will be time consuming. In your code at test_fphb_crack.prototxt I found this line.

   
   name: "fphb"

input: "data" input_dim: 1 input_dim: 3 input_dim: 500 input_dim: 500

did you resize the image to 500x500 and do inference later up-sampled again to original size?

2) I always see `N/A` for `AIU` measurement in crackforest dataset, what is it means?

3) You also measure the time of FPHB. Related with question 1, if your input is 500x500 then why different dataset has different running time?

[fyangneil]

1. input size in prototxt is not the real input size. real input size should be input image size
2. N/A is because the results from crackforest is binary map.

On Thu, Apr 23, 2020 at 7:00 PM ftlong6666 notifications@github.com wrote:

Hello @fyangneil https://github.com/fyangneil, thank you for your support, now I have 3 questions for the evaluation.

1. What is the input image size for FPHB network?, since its fully convolutional it should be not fixed right. However if we use original size it will be time consuming. In your code at test_fphb_crack.prototxt I found this line.

name: "fphb"

input: "data" input_dim: 1 input_dim: 3 input_dim: 500 input_dim: 500

did you resize the image to 500x500 and do inference later up-sampled again to original size?

1.

I always see N/A for AIU measurement in crackforest dataset, what is it means? 2.

You also measure the time of FPHB. Related with question 1, if your input is 500x500 then why different dataset has different running time?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/fyangneil/pavement-crack-detection/issues/10#issuecomment-618714056, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALAUWTUKS2EL2N77NV2ZOCDRODCBPANCNFSM4LA43X6Q .

[ftlong6666]

Hello @fyangneil thank you for your reply. As you mentioned before the result of Crackforest is binary so the AIU is N/A then how can we evaluate the ODS and OIS? since there is no threshold is used. You also produce same OIS and ODS but different for each dataset. How to get this number?

[fyangneil]

if I remember correctly, when calculating ODS and OIS, I did not use the threshold, since the result of crackforest is binary. I directly use the binary mask to compute ODS and OIS, thus in each dataset the ODS and OIS of crackforest are same.

---

From: ftlong6666 notifications@github.com Sent: Saturday, December 12, 2020 6:22 AM To: fyangneil/pavement-crack-detection pavement-crack-detection@noreply.github.com Cc: Fan Yang fyang@temple.edu; Mention mention@noreply.github.com Subject: [External] Re: [fyangneil/pavement-crack-detection] How to use eval_tool? (#10)

Hello @fyangneilhttps://github.com/fyangneil thank you for your reply. As you mentioned before the result of Crackforest is binary so the AIU is N/A then how can we evaluate the ODS and OIS? since there is no threshold is used. You also produce same OIS and ODS but different for each dataset. How to get this number?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/fyangneil/pavement-crack-detection/issues/10#issuecomment-743741659, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ALAUWTVGMQD5SXCSAO26TCTSUNHAXANCNFSM4LA43X6Q.

[lian666-ch]

我直接给matlab添加地址后直接就能用了