A bug that edge image size will change after `EdgeModel`

[youyuge34]

In the testing phase, I found a new cryptic bug that the edge output size of EdgeModel will dismatch with the input image size.
For instance, sizes of input images and masks are both [256,265], but handled by the EdgeModel, the edges output size will turn into [256,264], which I guess the odd number 265 is the key of problem. Maybe we can add a resize() in the test() function.

• Below is some debug prints:

  
  images size is torch.Size([1, 3, 256, 265]),
  edges size is torch.Size([1, 1, 256, 265]),
  masks size is torch.Size([1, 1, 256, 265])

images_masked size torch.Size([1, 3, 256, 265]) edges size after EdgeModel is torch.Size([1, 1, 256, 264])

[knazeri]

@youyuge34 This is because of kernel_size=4 in the strided convolution layers: (265+2-4)/2 + 1 = 132.5 which then gets rounded to 132, and by upsampling becomes 132 x 2 = 264 We made sure the kernel_size is divisible by the stride for downsampling/upsampling following pix2pix model and to eliminate the checkerboard artifact! However, it doesn't solve the problem of inputs with odd width/height sizes!

Let's keep this issue open; @CVDLBOT, @eshn any thoughts on this?

[CVDLBOT]

@knazeri Yes, you are right.

[knazeri]

@CVDLBOT I was thinking maybe we could force the input dimensions to be even by adding ReplicationPad2d to the input with odd dimensions and remove it as post-processing! This could fix both training and test phases!

[yuxiaoleipeng]

In the testing phase, I found a new cryptic bug that the edge output size of EdgeModel will dismatch with the input image size. For instance, sizes of input images and masks are both [256,265], but handled by the EdgeModel, the edges output size will turn into [256,264], which I guess the odd number 265 is the key of problem. Maybe we can add a resize() in the test() function.

• Below is some debug prints:

images size is torch.Size([1, 3, 256, 265]),
 edges size is torch.Size([1, 1, 256, 265]),
 masks size is torch.Size([1, 1, 256, 265])

images_masked size torch.Size([1, 3, 256, 265])
 edges size after `EdgeModel` is torch.Size([1, 1, 256, 264])

can you run it? i have a problem, may i have your qq or wechart?

[youyuge34]

@yuxiaoleipeng I have solved it temporarily using ReplicationPad2d in my modified demo application, which involves the problem of pixel alignment. You can test images in my project using commands as before.

P.S. This solvement is not totally perfect so I didn't merge it into this project.
Maybe only the re-design of original networks can solve it perfectly as Mask R-CNN did.

[yuxiaoleipeng]

@yuxiaoleipeng I have solved it temporarily using ReplicationPad2d in my modified demo application, which involves the problem of pixel alignment. You can test images in my project using commands as before.

P.S. This solvement is not totally perfect so I didn't merge it into this project. Maybe only the re-design of original networks can solve it perfectly as Mask R-CNN did.

i must download the 105g dataset? i only download the maskdata before..... i hvae a mistake?

[youyuge34]

@yuxiaoleipeng If u just want to test, just download the pre-trained weights files and optional maskdata. I will update a detailed Chinese using manual in these days in my demo application. If u are the new comer into Pytorch, I recommend u to wait for my Chinese manual update.

[knazeri]

@yuxiaoleipeng It's a really cool application you created 👍

If you are using ReplicationPad2d, you have to make sure the input dimensions are divisible by 4 (because of the two strided convolution in the model), but as you said this does not completely solve the problem, but it's a workaround!

[youyuge34]

@knazeri I added this function in util.py which is called on EdgeModel output when testing :

def output_align(input, output):
    """
    In testing, sometimes output is several pixels less than irregular-size input,
    here is to fill them
    """
    if output.size() != input.size():
        diff_width = input.size(-1) - output.size(-1)
        diff_height = input.size(-2) - output.size(-2)
        m = nn.ReplicationPad2d((0, diff_width, 0, diff_height))
        output = m(output)

    return output

[knazeri]

@youyuge34 I see, only the output of the model is going to be off by several pixels which is ok in most cases.

[yuxiaoleipeng]

@yuxiaoleipeng If u just want to test, just download the pre-trained weights files and optional maskdata. I will update a detailed Chinese using manual in these days in my demo application. If u are the new comer into Pytorch, I recommend u to wait for my Chinese manual update.

thank you, i want to train the module, look forward to your Chinese manual

[youyuge34]

@yuxiaoleipeng I have completed a detailed training manual, here. Make sure u have firstly read my README.md.

[knazeri]

@youyuge34 It's a very cool interactive tool you have created. I was going to create a web-based one but is ok with you if we include your work here?

[youyuge34]

@knazeri Sure，include my work to your README.md which is up to you~

And at first I was going to create a web-based one too. But pytorch build-in webpage maybe not that easy as KerasJS.

[knazeri]

Thanks, @youyuge34
For a web-based tool, I created a simple python web server and used javascript to post an image to the server, inpaint it and push the result back to the browser. The UI part in javascript still needs work though!

About the interactive tool, the tool_patch.py script in your repo, can it run standalone? Do you think you could open a pull request?

[youyuge34]

@knazeri I am afraid it can't work standalone. I changed util.py, main.py and edge_connect.py as well. And the way of my changes may be not that consistent with your work, which would confuse the ones a bit who want to read the algorithm only.

[knazeri]

@youyuge34 No worries, I'll read it through later to see how compatible it is!

[anshen666]

你好，我最近也在跑这个代码。可以加你交流一下吗？我的微信：loveanshen 我的QQ：519838354 我的邮箱：519838354@qq.com 非常期待你百忙中的回复