I'm sorry to bother you,but I have some questions...

[erhuodaosi]

@msakai @delta2323 @iwiwi @okapies @mitmul Hi,

Awesome Work! I have some question while training the model,an error occurred as follow: My chainer version is 5.3.0

In addition,I wonder if we could use the irregular mask put forward by partial convolution as input rather than use the free form mask.How could we solve it?

Last but not the least,I have noticed that the EDGE_FLIST is none,I wonder you say there are no influences whether EDGE_FLIST exist or not.How to get the EDGE_FLIST?

Looking forwarding to receiving your reply!I would very appreciate if you could help me!

Thanks,

[nogu-atsu]

Thank you for the questions.

1. chainer v6 installation

Chainer and cupy version 6.0.0 can be installed with pip install cupy --pre . Otherwise, you can specify the versions to install directly; pip install chainer==6.0.0rc1 .

2. Irregular mask

The irregular mask is not supported. You can use it if you implement the function and specify it in the _create_mask function.

3. EDGE_FLIST

Edge input is optional, and If EDGE_FLIST is None, edge input is not used for training.

[erhuodaosi]

@nogu-atsu

Hi,

Thanks for your timely reply!I have installed chaine V6.0.0rc1 successfully.but the same error occurred again while training.

As for the second question,what is the meaning of the function of free_formmask ?what is the difference between free form mask and irregular mask provided by partical convolution and the rectangle mask (below 128 * 128) generated by bbox2mask ? 

When it talks to the third question,if we put the picture without edge as input,could we get a worse result?

I would very appreciate if you could help me!Thank you!

Thanks,

[nogu-atsu]

1. Original implementation would work.

   from chainer.link_hooks.spectral_normalization import SpectralNormalization

2. Free_form_masks are created online during training, and irregular_masks are collected before training. Irregular mask datasets are available in https://nv-adlr.github.io/publication/partialconv-inpainting. The rectangle mask is used in DeepFillv1 paper, and free_form_mask is used in DeepFillv2 paper.

3. Yes. The performance would be worse without edge input because the task would be difficult.

[erhuodaosi]

@nogu-atsu
Hi,

Thank you very much!

1.It's my fault to mistake the function of chainer.link_hooks.spectral_normalization.I'm so sorry to modify the original implementation.

2.what is the shape of the free form mask generated?random,circle,trangle,square,line,or rectangle?After training the model eventually,what is the performance on irregular mask provided by Pconv?To be honest,I am weak in programming,especially I am a green hand in chainer.

3.How to get the edge sketch?

I would very appreciate if you could help me!Thank you!

Thanks,

[nogu-atsu]

2. The shape of the mask is (1, 256, 256), and the mask is an image like this.

3. You can use any kind of edge detector to save edge images. For example, Canny edge detector with opencv (https://docs.opencv.org/3.4.3/da/d22/tutorial_py_canny.html) or HED (https://github.com/s9xie/hed).

[erhuodaosi]

@nogu-atsu

Hi,

Thank you for your patient!Thank you very much!I would have a try to get the edge sketch in your way.

Thanks,

[erhuodaosi]

@nogu-atsu Hi,

An error occurred while training,it is the second time to report the error,how to solve it?

I would very appreciate if you could help me!Thank you!

Thanks,

[nogu-atsu]

Thank you for the report.

Could you test this modification?

batch_data = self.xp.array(batch_data)
mask = self.xp.array(mask_data)

->

batch_data = self.xp.array(list(batch_data))
mask = self.xp.array(list(mask_data))

at https://github.com/pfnet-research/Chainer-DeepFill/blob/master/src/inpaint_model.py#L388

(sorry, this may not solve the problem.)

[erhuodaosi]

@nogu-atsu

Hi,

Thank you for your timely reply!Thank you for your advice!I have followed your way to modify the code.I wolud retrain the gate conv and see if therer is any wrong while training.

Thanks,

[nogu-atsu]

Can you print the content of batch_data before https://github.com/pfnet-research/Chainer-DeepFill/blob/master/src/inpaint_model.py#L388 if the error still occurs?

[erhuodaosi]

@nogu-atsu

Hi,

I am not sure whether what I have done is true or false,but there show nothing.

batch_data, mask_data = zip(*batch_and_mask)

before

        print(batch_data)
        print(mask_data)
        print(type(batch_data))
        print(type(mask_data))
        batch_data = self.xp.array(list(batch_data))
        mask = self.xp.array(list(mask_data))
        #after
        print(batch_data)
        print(mask_data)
        print(type(batch_data))
        print(type(mask_data))

Thanks,

[nogu-atsu]

1. What is the value of VAL_PSTEPS at https://github.com/pfnet-research/Chainer-DeepFill/blob/master/src/gated_convolution.yml#L13 now?

2. Is there any output image in the directory specified at https://github.com/pfnet-research/Chainer-DeepFill/blob/master/src/gated_convolution.yml#L3?

[erhuodaosi]

@nogu-atsu

Hi,

Thank you for your timely reply!

1.The value of VAL_PSTEPS is set by default value 100. 2.there are output image(100 per each) and npz model(1000 per each) in the directory results_gc,a log file as well.

Thanks,

[nogu-atsu]

I do not know why nothing is printed, but considering validation results are output at the beginning of the training, it is possibly related to some shape mismatch problem.

For example, some of your test images might be smaller than 256 x 256.

It is a similar problem as ...

import numpy as np
import cupy as cp

cp.array([np.arange(5), np.arange(4)])

-> ValueError: Unsupported dtype object

To solve this, all images should be resized to sizes larger than 256 before training.

I will support inputting smaller images than 256 later. Thank you for the information.

[erhuodaosi]

@nogu-atsu

Hi,

Thank you for your information!Thank you very much!

My dataset is based on 256 256 resolution places2.All of my train,test and validation images are 256 256 size.The same question occurred again,for the first time,it stops at 20500 iterations,for the second time,it ends at 2100 iterations,for the third time,it ends at 3300 iterations.

Thank you!

Thanks,

[nogu-atsu]

Are there grayscale images in the training or validation dataset?

[erhuodaosi]

@nogu-atsu

Hi,

No,they are colorful images,which have 3 channels.

Thanks,

[nogu-atsu]

I supported small images and gray images. Could you pull the latest code and run the training?

[erhuodaosi]

@nogu-atsu

Hi,

Thank you for your patient and modify the code timely.I would use the latest code and have a try to retrain the model.

Thank you very much!

Thanks,

[erhuodaosi]

@nogu-atsu

Hi,

I found that there is a color difference between the inpainting image and the original image.How to solve it?

I would very appreciate if you could help me!Thank you!

Thanks,

[nogu-atsu]

What do you mean by "color difference"? Inpainted images always become different from the original images because they are predicted only from masked images. Does this answer your question?

By the way, does the error above still occur?

[erhuodaosi]

@nogu-atsu

Hi,

Thank you for your timely reply!Thank you for your modification!the error does not occur due to your help!Thank you!As for the color difference,I mean that the area outside the holes is different with the area outside the holes between inpainting image and original image.Maybe I think the output should be the form of I_comp = F.where(M_b,I_gt,I_out) rather than the form of I_out = self.model(I_gt,M). As is shown in iter_43400 and iter_43400,the color of the original image is deeper,and the inpainting results is shallow than the original image,are the first line called original image,and the third line called inapinting results? I would very appreciate if you could help me!Thank you!

Thanks,

[nogu-atsu]

Sorry, what do you mean? This is the visualization of the color differences between the first and third line of your result, but there is no difference between the colors of un-masked regions. (Black color shows there is no difference.)

[erhuodaosi]

@nogu-atsu

Hi,

Thank you for your timely reply! First of all,I apologize for my poor English expression.I am so sorry for my expression that bother you. I mean that the un-masked regions is different between the original image and the inpainting results, such as the color information etc..there are color differences subjectively.Is the third line called inpainting results or model results?,The color of the original image is deeper,but the color of the inpainting results is shallower than the original image. Thank you for your visualization!

Thanks,

[nogu-atsu]

No problem.

Do you mean that you feel there are color differences in the first and third lines of the results subjectively (for example in https://user-images.githubusercontent.com/45588148/56782097-376e7380-6818-11e9-9226-d080ed7a5c88.jpg)?

The third line is a combination of "un-masked region from the original images" and "masked region from the model output".

[erhuodaosi]

@nogu-atsu

Hi,

Yes. I mean that I feel there are color differences in the first and third lines of the results subjectively.That lead me to think the third line seems to a combination of "un-masked region from the model output" and "masked region from the model output"(a wrong way) rather than "un-masked region from the original images" and "masked region from the model output"(a right way).Maybe I see the results too subjectively.

Thank you very much!

Thanks,

[erhuodaosi]

@nogu-atsu

Hi, I'm sorry to bother you again ,but I have some questions in the code.

1.In inpaint_model.py,the code is listed as follows: Should it be modified as below?

attention branch

    x = self.pmconv1(xnow)
    x = self.pmconv2_downsample(x)
    x = self.pmconv3(x)
    x = self.pmconv4_downsample(x)
    x = self.pmconv5(x)
    x = self.pmconv6(x)

    x, offset_flow = contextual_attention(x, x, mask_s, 3, 1, rate=2, return_flow=return_offset)
    x = self.pmconv9(x)
    x = self.pmconv10(x)

2.The result is selected from the test dataset randomly .How to get the Mean L1 loss? 3.I have followed your way to use HED and canny,but I stll do not know how to use it and could not get the edge flist from the whole dataset in order.Could I get the edge flist without training?

I would very appreciate if you could help me!Thank you very much!

Thanks,

[nogu-atsu]

Thank you for the report.

1. Yes, you are right. I will fix it.
2. Do you mean L1 loss across the entire batch? How about calculating L1 loss between batch_pos and batch_complete in test.py?
3. This is the sample code for the canny detector.

   import cv2
   img = cv2.imread("path to your image")
   imgRGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
   imgYUV = cv2.cvtColor(img, cv2.COLOR_BGR2YUV)
   imgY = imgYUV[:,:,0]
   result = cv2.Canny(imgY, 100, 300)
   cv2.imwrite("filename", result)

   Please do the same things for all training and validation images and save the flist of them.

[erhuodaosi]

@nogu-atsu

Hi,

Thank you for your timely reply!Thank you for your sample code!

1.When I modify the code as presente,I get an error while training.

2.Yes!I mean how to caculate L1 loss across the entire batch.

In addition,could I use Vgg16 as pretrained model based on deepfill V2?I mean that I add vgg16 layers based on deepfill V2.Could I improve the quality of image inpainting by using this way?

I would very appreciate if you could help me!Thank you!

Thanks,

[nogu-atsu]

I have pushed the modification. Please pull the master branch and try training again.

I have no plans to support perceptual loss at present. In DeepFill v1 paper, it is reported that perceptual loss doesn't improve the quality of the outputs.

[erhuodaosi]

@nogu-atsu

Hi,

Thank you for your timely reply!I'm so sorry that I can not see the modification.Could you point it out?I am getting the edge flist from the whole places2 dataset in your way successfully.Thank you!

Thank you very much!

Thanks,

[nogu-atsu]

Sorry, I have just pushed the modification.

[erhuodaosi]

@nogu-atsu

Hi,

I would have a try to use the latest version. Thank you very much!

Thanks,

[erhuodaosi]

@nogu-atsu

Hi,

I'm sorry to bother you again ,but I still have some questions about the code.

How to do image inpainting work on a real broken picture?If given a broken picture rather than a combination of ground truth and the mask generated by free-form mask. That's to say,how to save mask,I want to get the mask dataset generated by free-form mask,and put mask as input.

I would very appreciate if you could help me!Thank you! Thank you very much!

Thanks,