if RPCA can remove highlight well,why we need JSHDR?

[zhuguangqiang]

in JSHDR paper, you use RPCA to generate the highlight-free image from highlight image as the removal ground truth, if the RPCA can remove highlight well,why we need a new mothod of JSHDR?

[fu123456]

RPCA needs multiple input images of the same scene but with different illumination conditions, while our JSHDR only needs to a single RGB image.

[zhuguangqiang]

in your paper: "Finally, (d), (h), (f) and (i) in Figure 2 (i.e., input and its corresponding highlight-free,highlight as well as highlight mask images) are an example quadruples in our dataset."
is there mistake in above description? I think the quadruples should be h(highlight image)/e(hightlight-free image)/g(highlight intensity map)/i(hightlight mask), am I right? ![Uploading 1.jpg…]()

[zhuguangqiang]

attach the paper snapshot here:

[zhuguangqiang]

and I think also some wrong in the Figure 5 aboout D branch, the introduction of D branch is "(3) D is estimated by feeding the concatenation of [F, M, S, I−MS] into a convolutional block consisting of three 3×3 convolutions." and the Figure 5 is below,I think the D concat should as the red arrow before the Convs:

[fu123456]

Hi, for your first question, actually "highlight image" refers to "highlight intensity image in our paper, and "input image" in our paper refers to "highlight image" as your said. I checked our paper again, and found that four components in a quadruples may be not unified for their names. There is such a name of "highlight image" which refers to "highlight component (intensity) image" in existing papers. So what you said is not wrong. Thanks.

[fu123456]

For your second question about D branch, I think that the original structure in our main paper is not wrong, and what you said should also work actually (produce reasonable results). Please note that in Figure 5, "[F, M, S]" should refer to "[F, M, S, I-MS]". Why we write like this? The reason is due to space limit. Therefore, "[F, M, S]" can be considered as the shorthand of "[F, M, S, I-MS]". After all, there are the first three variables (F,M,S) to derive the fourth variable I-MS, where I is a known variable. Finally, In the processing of estimating D, we use the idea of residual learning which has been widely used in deraining, denoising and so on.

[zhuguangqiang]

so about the D branch, out output of D is: output_D = I - Convs(Concat[F,M,S,I-MS]), am I right?

[zhuguangqiang]

and finally for getting the predicted hightlight-free image,we have 2 choices: 1、just use the D branch output 2、use the M branch and S branch and doing " I - M*S ". which choice we should take?

[fu123456]

Your first question: yes. The second question: the purpose is that we want to design a multi-task network for joint specular highlight detection and removal, and argue that the accurate detection result is able to benefit to the specular highlight removal. Our multi-task network can simultaneously constrain M, S, and D, and thus estimate them.

[wcyjerry]

@zhuguangqiang hey, I want to know how u preprocess the data, like how to normilize it, because if scale I to between (-1,1), and S to (0,1), then in the end, output_D = I - Convs(Concat[F, M, S, I-MS]) , I - MS will be like in (-2, 1), it seems doesnt make any sense. Should I concern about it?