porting to the newest caffe

[saedrna]

The HED uses an old caffe, which is not compatible with the latest CUDNN v5 (2x faster reported by NVIDIA).

If only for testing, after some investigation, it seems that HED uses no specific layers, so it will run the edge detection work and it is.

Except that I have found the boundaries are shift to the right bottom part, as shown below. After I change the pad value of the first layer to 1 from 35, the shift is corrected, but the boarder of the images is also detected as edges.

I have compare the cpp files with caffe repo, I have found several difference parts, e.g. something related to image label map data, data transform. But I don't see any critical modifications with the conv layers. So would you give me some hints of how to solve this problem?

Image

with pad 35

with pad 1

[zeakey]

May the different implementation in crop layer cause this issue.

[fxw6000]

I have the same problem.

[juntingzh]

Actually I don't know why a customized Caffe is used? Even for training, any special layers are designed and used? No clue is found by looking at the prototxt only. Anyone has any idea?

[s9xie]

Hi all,

First I want to really apologize for the slow response in the repo issues.

@saedrna @zeakey @fxw6000 First that's right you need to have the padding in the first layer to avoid the shifting problem. And it is the correct behavior that you are detecting the image borders as boundaries. To solve this problem, what I was doing is a simple trick:

During testing stage, I "augment" the testing image by padding it with mirror reflections (both vertical and horizontal) of itself. Then after you get the edge map, you can just crop the center image. I'll put a note in Readme.

I'm currently a little bit swamped with some other projects but I'll keep this open and let you know when I migrate this to the new caffe.

[saedrna]

@s9xie Thanks, I think for the time being, I can also use hed by the same trick. That is set pad the image manually with mirror reflection (using opencv) and then set pad=1 for the first layer rather than 35.

Looking forward to your porting to the caffe master repo!

[stevendbrown]

@saedrna, can you make your port available?

[saedrna]

@stevendbrown , the porting is simple, because recent caffe support the all the layers in testing HED, except for a simple bug (the effects and solution are described above). So I will only enlarge the image outside caffe using copyMakeBorder and after hed, subseting the corresponding region.

And caffe is also ported to windows by in a separate branch.

[stevendbrown]

@saedrna if you can help me get on the right track i'd be grateful. i started by attempting to apply the changes in caffe PR #4159 but there seem to be some asynchronies between the caffe branch that PR applied to and the caffe source included with hed which i haven't resolved yet. my alternative strategy was going to be to port the changes to caffe included with hed into the current caffe master branch.

maybe i'm misunderstanding you - do you mean that the current caffe master branch is a drop-in replacement for the caffe version included with hed (with the exception of the border issue described above)?

[saedrna]

@stevendbrown , that's right, I just used the model, prototext and changed the pad to 1, the results are reasonable, except for the boarder problem.

On windows, the windows branch of caffe is used.

[stevendbrown]

@saedrna i'm replicating your result with the padding. thanks for the pointer!

[stevendbrown]

it looks like the way layers are defined has changed with the current master branch of caffe. have any of you ported ImageLabelmapData to the current caffe?

i've moved ImageLabelmapData into the current caffe master branch and it seems to be working. if there is no code available for other people's ports, i can make a fork/PR/something with the ported code.

[stevendbrown]

my hack port of the ImageLabelmapData to the current version of caffe is here:

https://github.com/stevendbrown/caffe/tree/HED

i set the padding in train_val.prototxt to 1 and get excellent results with de novo training and prediction, albeit on a relatively simple dataset. my non-quantified impression is that with the current caffe branch i can increase the training batch size without the performance hit i noticed if i increased the batch size using the HED version of caffe.

i'm not a computer scientist, so if anyone can comment on the chop-shop job i've done here, i'd be grateful.

[BingTSC]

@saedrna I build the caffe on windows, then how to run HED edge detection on images? I don't know what kind of modification should apply to top of Caffe Windows branch. Could you please help me? thanks.

[saedrna]

@BingTSC ,just use the deploy prototxt, and change pad = 35 to 1, then you will reproduce the results as above.

[zeakey]

Hi @s9xie @saedrna @fxw6000 @BingTSC @juntingzh @stevendbrown

Actually the shift at predicted edge map are due to the different implementation of CropLayer between HED and official branch of caffe.

In HED version of caffe which I think is based on Longjon's FCN, the crop offset is computed automatically inside the network, that is, in crop layer you will not need to specify the offset. But this implementation is very complicated and has a lot of modifications on almost all layers cpp code.

In the newly official branch of caffe, the crop layer has a parameter named "offset", this allow you to specify the crop offset manually, sounds not good because you have to compute offset by yourself, but caffe now provide NetSpec which you can generate net prototxt file by a python script, in NetSpec it will compute the crop layer offset automatically, I think this is a good solution which avoid modifying the C++ layer implementations and provide automatically offset computation.

Berkeley.FCN is a example of NetSpec script that generate net prototxt file which contain a crop layer, and here is my reimplementation of HED network in NetSpec, it is totally compatitable with the newest version of caffe.

[stevendbrown]

adding crop_params { axis: 2 offset: N offset: N } (where N = padding) to the Crop layers also seems to work well.

[ps48]

@stevendbrown @s9xie The new ported caffe works and the build is successful 👍. But the edges are not as clear as they should be like the original HED caffe. Is there anyway to make them more accurate?

[hustzxl]

@stevendbrown hai ,I clone your git code,but when I run ./solver , I get Nan . it's ok when I run s9xie's do you konw why . I set the pad = 1 . and the offset 1 in cov2 , 2 in cov3 , 4 in cov4 , 8 in cov5

[madnavs]

@stevendbrown Hi I used your caffe to train hed. However, the output looks all dark. I changed pad from 35 to 1 in train_val and deploy. Did I miss anything? Can you share your train_val.prototxt?

[luckyboysmith]

@zeakey
thanks for your contribution,then after using hed-caffe-netspec.py to produce three .pt file,what should i do next step?

[luckyboysmith]

@zeakey
can u share your trainval.prototxt? after i modify my trainval.prototxt accoring to hed-train.pt the loss is zero thanks!

[luckyboysmith]

@zeakey according to hed-train.pt
data layer type is Input, but where should i input the data route？ hope to get your respond

[luckyboysmith]

@zeakey thanks for your code hed-caffe-netspec.py, after i use hed-caffe-netspec.py to produce hed-train.pt, and modify my trainval.prototxt accoring to hed-train.pt, throung i set new_width=500 and new_height=500 in ImageData layer, it seems there still exist problem in crop layer, can u share your tranval.prototxt and solve.py? i will be grateful thx!!!!!!

[luckyboysmith]

@hustzxl hi i meet the same problem can u share your method to solve this problem? thx!!!!!!!

[zeakey]

Hi all, I've reimplemented hed based on new caffe, check out it here https://github.com/zeakey/hed!

[luckyboysmith]

@zeakey thanks a lot!!!!!