metrics used in the paper

[zengyh1900]

Hi @JiahuiYu

I saw the numeric results report in your paper includes mean L1 loss, mean L2 loss and mean TV loss. I wonder why they all have percentage ? for example, I used to know L1_loss=abs(real-fake), does the percentage comes from dividing 255 like L1_loss=abs(real-fake)/255?

[JiahuiYu]

For errors we used relative L1 error and squared relative L2 error as this: metric(x/256, y/256)

If you just want to compare your results with ours, I suggest you to use our model and compute with your own formulations. As I mentioned in paper, these numbers are just reported as a reference. Since for inpainting, perceptual results are much more important.

[an1018]

@JiahuiYu (2)Is it right？ (2)And in your formula y represent original image?

[JiahuiYu]

@an1018 Sorry for confusion. It should be metric(x/256, y/256), assuming x is prediction, y is ground truth.

[an1018]

mean l1 loss in your article, l1 norm in your formula,,and Could you see which one is right? OR

[JiahuiYu]

@an1018 The bottom one is correct. I just checked my previous code, my code is:

np.sum(np.abs(x/256-y/256))/mask_height/mask_width/3.

It makes slight difference compared with yours. Your code averages over entire image while mine averages over masked region.

Again, if you just want to compare your results with ours in your paper, I suggest you to use our model and compute with your own formulations/implementation.

[an1018]

Sorry, I use deepfillv2, I don't think free-form masks have mask_height and mask_width

[JiahuiYu]

@an1018 Ok, do whatever you like then, as long as they are fair comparison. :)

[awsssix]

@JiahuiYu Q1: 3 represent channel? Q2: x/y represent hole image ？ Q3：mask_height/mask_width/3 ：represent only masked region？ Q4：psnr metric is correct? Looking forward your replay!

[JiahuiYu]

@awsssix 1: yes. 2: x is image after inpainting, y is ground truth. 3: Normalize over masked region. 4: Different than ours. We use

-10.0 * math.log10(np.sum(np.square(x-y))/mask_height/mask_width/3.)

[awsssix]

@JiahuiYu Thanks for your kindness. But I still have a question,as you said yours is averages over masked region, so x/y is hole image or only the extracted mask part,like below:

[JiahuiYu]

@awsssix The pixels in x and y are different only in the masked region. In fact I cannot understand your question after reading more than 10 times... Can you make it more clear what is your question?

[awsssix]

I have understood.Thanks! Why divide the number of channels ?

[wangping1408]

hi, @JiahuiYu I want to know what the batch_data here is.Whether it refers to the complete picture in the original data set？and Why is this calculated?batch_pos=batch_data/127.5-1 I hope you can answer my doubts.

[Yukariin]

@wangping1408 that's data normalization, from [0, 255] to [-1, 1].

[JiahuiYu]

@Yukariin Thanks.

[neouyghur]

@JiahuiYu I can't find the code for TV loss evaluation. Could you help with this? Thanks.

Hi this is my code, could you check it ?

        img = np.multiply(img, mask)
        pixel_dif1 = img[1:, :, :] - img[:-1, :, :]
        pixel_dif2 = img[:, 1:, :] - img[:, :-1, :]
        return (np.sum(np.abs(pixel_dif1)) + np.sum(np.abs(pixel_dif2))) / np.sum(mask)

[JiahuiYu]

@neouyghur The code should be right - except we normalize image to 0-1 before computing tv loss.

[neouyghur]

Thanks for your quick reply. The input is normalized before calculating.

On Sat, Jun 15, 2019 at 2:22 PM JiahuiYu notifications@github.com wrote:

@neouyghur https://github.com/neouyghur The code should be right - except we normalize image to 0-1 before computing tv loss.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JiahuiYu/generative_inpainting/issues/146?email_source=notifications&email_token=ABQRQ3EQDP2HUAPL6332ECTP2RVB7A5CNFSM4FX275VKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXYP6TI#issuecomment-502333261, or mute the thread https://github.com/notifications/unsubscribe-auth/ABQRQ3CW7O7ZCKKIEBO3Q3LP2RVB7ANCNFSM4FX275VA .