irfanICMLL
commented
7 years ago BMVC2017 The British Machine Vision Conference September 4-7, 2017, London, UK
Reviews For Paper Paper ID 947 Title Auto-painter: Cartoon Image Generation from Sketch by Using Conditional Generative Adversarial Networks
Masked Reviewer ID: Assigned_Reviewer_37
Review:
Question
Paper Summary. Please summarize the paper in your own words. This paper proposes auto-painter, an end-to-end deep learning system, for automatic colorization. The system arguments the previous ‘pix2pix’ model [10] by introducing additional two terms (1) a TV loss to avoid the color mutation, and (2) a feature loss to measure the difference in VGG feature space. This evaluation experiments show that the proposed system outperforms pix2pix model. The paper also includes a comprehensive ablation study, indicating the advantage of the full method, compared to pix2pix+TV, and pix2pix+feature loss.
Paper Strengths. Describe positive aspects of the paper. What is of interest to the community? Consider criteria such as novelty, clarity, technical achievements (theoretical or practical), and thoroughness of validation. If accepted, what aspects would be cite-worthy? - The paper is well-written and easy to follow.
- The system produces impressive results on automatic cartoon colorization. It seems to be a useful application.
- The proposed system outperforms the previous ‘pix2pix’ model.
- comprehensive ablation study for understanding the importance of feature loss, and TV loss.
- I think the color control could be useful in practice. Paper Weaknesses. Discuss negative aspects such as lack of novelty or clarity, technical errors, missing validation or proofs, etc.. Please justify your comments with details. If you think the paper is not novel, explain why and provide specific references. Be kind. - The paper missed a few relevant reference on sketch2photo and manga colorization. I recommend the paper discuss the connection and difference between the proposed method and these prior work.
Scribble-based cartoon colorization: Qu, Yingge, Tien-Tsin Wong, and Pheng-Ann Heng. "Manga colorization." ACM Transactions on Graphics (TOG). Vol. 25. No. 3. ACM, 2006.
Non-parametric sketch2photo: Chen, Tao, et al. "Sketch2photo: Internet image montage." ACM Transactions on Graphics (TOG) 28.5 (2009): 124.
GAN-based sketch2photo: Zhu, Jun-Yan, et al. "Generative visual manipulation on the natural image manifold." European Conference on Computer Vision. Springer International Publishing, 2016.
-
If I understand it correctly, the current Table 1 is only for Minions. It would be great if the paper can also report the “like vs dislike’ for Japanese anime datasets.
-
In Sec 3.1 “Network Structure” and Sec 5, the paper should credit ‘pix2pix’ [10] for introducing U-Net architecture for the image-to-image translation. This article should not describe it as a contribution. Overall Rating. Try to avoid choosing "Borderline." Borderline Overall Evaluation. Please convey to the authors, the Area Chair, and to fellow reviewers, how you decided your Overall Rating. Good ideas need a reviewer to champion their paper, and bad papers need constructive criticism. What key things would you like the authors to include in their rebuttal? Overall the paper proposes an excellent automatic cartoon colorization system with useful user color control. I would like to see its acceptance if my above concerns are addressed.
Masked Reviewer ID: Assigned_Reviewer_38 Review:
Question
Paper Summary. Please summarize the paper in your own words. The paper presents sketch-to-image conditional-GAN based model auto-painter that extends prior pix2pix work to generate better colorization for the given sketch of cartoon images. The paper is very clean and easy to read. The methods are valid and the experimental results show better visual performance than the prior work. The author put more constraints on the prior pix2pix network to get better results. They conduct user study to compare the results generated by different methods. Also, they demonstrate that they can generate images according to different user inputs of the colors. Besides, the image outputs are of quite high resolution, 128x128. However, the technical contribution is marginal. The novelty is not so strong. It is unclear if the proposed method can work in other image colorization or generation settings, such as the applications mentioned in pix2pix paper (e.g. changing day to night, etc.). Paper Strengths. Describe positive aspects of the paper. What is of interest to the community? Consider criteria such as novelty, clarity, technical achievements (theoretical or practical), and thoroughness of validation. If accepted, what aspects would be cite-worthy? The paper consider the task of cartoon colorization given some inputs black-and-white sketch images. In order to improve the performance than the prior pix2pix work, they put more constraints when designing the loss functions. The experiments show quite high-resolution better performance than the pix2pix work. Also, the network allows users to indice the color style as initial point and can control the gan generated images accordingly. They conduct user study and it seems valid. Paper Weaknesses. Discuss negative aspects such as lack of novelty or clarity, technical errors, missing validation or proofs, etc.. Please justify your comments with details. If you think the paper is not novel, explain why and provide specific references. Be kind. The idea in the paper is not novel. And, they are targeting at cartoon image generating setting, which is narrower than the prior work pix2pix. It is unclear if their methods can improve the performance in other settings, such as changing day images to night ones, etc. The technical contribution is weak since both U-net structure and GAN training loss are extensively used in the prior works. It is unclear where is the exact technical contribution in this paper. Overall Rating. Try to avoid choosing "Borderline." Borderline Overall Evaluation. Please convey to the authors, the Area Chair, and to fellow reviewers, how you decided your Overall Rating. Good ideas need a reviewer to champion their paper, and bad papers need constructive criticism. What key things would you like the authors to include in their rebuttal? The paper is using some existing technique (U-net, pix2pix) with some improvement to tackle a narrower task, cartoon image generation. It is good since the methods and the experiments are valid. But the weak points are: 1) It is unclear if the methods can improve the image generation in broader applications. 2) the technical contribution is not much. It is very helpful if the authors can apply their modifed methods on other applications, such as changing day images into night ones, etc.
Masked Reviewer ID: Assigned_Reviewer_39 Review:
Question
Paper Summary. Please summarize the paper in your own words. This paper proposes an extension to the pix2pix model for the application of cartoon character generation. The main change is adding two new terms in the loss function. The feature loss, tries to minimize the difference in feature space of a certain layer of VGG between the generated and the ground truth image. Further, a total variation loss is used to regularize the output. Results are demonstrated on two types of image sets. Paper Strengths. Describe positive aspects of the paper. What is of interest to the community? Consider criteria such as novelty, clarity, technical achievements (theoretical or practical), and thoroughness of validation. If accepted, what aspects would be cite-worthy? It is reasonable to assume that colorizing cartoons is different to natural photographs, since the images possess quite different characteristics in terms of textures, shading, and colors.
A user-study has been performed to evaluate the algorithm in comparison the three baselines. Further, many different examples of generated images are shown in the paper.
Controlling the color with color block is useful for a more manual colorization by an artist or for recoloring existing cartoons. Paper Weaknesses. Discuss negative aspects such as lack of novelty or clarity, technical errors, missing validation or proofs, etc.. Please justify your comments with details. If you think the paper is not novel, explain why and provide specific references. Be kind. I am not sure about the choice of loss functions here. The data-term L_p is using an L1 loss while the TV loss is using an L2 loss. I would find the opposite choice more natural. A L1 TV regularizer would allow for color discontinuities, which are actually desired in a cartoon setting, instead of the smoothness of L2. The opposite effect applies for L_p.
The number of images in the user study seems to be 40. How where these images chosen? In my opinion, they should have been selected randomly from the test set, to allow a fair comparison between the different algorithms. I cannot find any information about this in the paper. Also the qualitative samples in the figures should be randomly selected and not cherry picked to allow the reader to get a better understanding of the image quality since no objective quantitative evaluation is possible.
In terms of contribution, this paper combines pix2pix [10] with two losses described in [11]. There is no comparison to other work except the baseline pix2pix although there are several (colorization) techniques already out there. For example [9,10,21].
Why was the depth of the features in L_f chosen to be 4. It would be interesting to see the results for different j. Also, I am not sure why here the metric is L2 and for L_p the metric is L1. If j was 0 then the images are compared directly. In this case it would be natural to assume L_p = L_f which is not the case.
Minor comments:
- The TV loss L_tv is defined on the ground truth y and not the prediction G(x,z), I suppose this is a mistake and the regularization is carried out on the prediction.
- Typos: 262 epoches, 321 a specific color schemes, 355 should not across different color regions
- Fig. 4. Vertical alignment problem in TV_loss column Overall Rating. Try to avoid choosing "Borderline." Weak Reject Overall Evaluation. Please convey to the authors, the Area Chair, and to fellow reviewers, how you decided your Overall Rating. Good ideas need a reviewer to champion their paper, and bad papers need constructive criticism. What key things would you like the authors to include in their rebuttal? GANs, style transfer and colorization are difficult to evaluate since they depend on user preference and subjective perception which is hard to measure. Thus, I find it very important to show a random selection of images in the paper and also to understand how the samples for the user study were selected. Further, comparing to other existing approaches [9,10,21] would be desired, especially since the method uses a non-public, non-benchmark dataset. Many arbitrary choices were made L2 vs. L1, j = 4, w_f = 0.01, w_tv = 0.0001 which are not evaluated or motivated. Given all above strengths, weaknesses and comments, my current rating is “weak reject”.
Reviewer 1#: Q1:The paper missed a few relevant reference on sketch2photo and manga colorization. I recommend the paper discuss the connection and difference between the proposed method and these prior works. A1: Thanks a lot the reviewer’s recommendation of adding reference paper. I would like to add these discussions in my new vision paper in the section “related works”. "Manga colorization" is based on the texture-based segmentation. It can only filled the region with user’s input color. Meanwhile ‘Auto-painter’ is a generative model, which can learn the color scheme itself. It can both consider the user’s input and the original sketch. It can also paint the sketch without the user input. "Sketch2photo: Internet image montage" is different from our work mainly in the application field and method. It focus on the natural photos synthesis based on the image retrieval and segmentation while we can generate virtual cartoon character with vivid color based on the user’s arbitrary creation. "Generative visual manipulation on the natural image manifold" learns the manifold of natural images consider the user’s input instead of generating images. They already have a natural image as input while we just have a sketch.
Q2: If I understand it correctly, the current Table 1 is only for Minions. It would be great if the paper can also report the “like vs dislike’ for Japanese anime datasets. A2: To be honest, I have already finished the test on seven persons, but limited by the page length, I choose to show the evaluation results on Minions. I would like to add the “like vs dislike” test (auto-painter vs pix2pix) for Japanese anime datasets and employ more volunteers to do the test and show the results in the revised version. The elementary results of the seven person on the Japanese anime datasets are as follows: Like 1 2 3 4 5 6 7 Total Pix2pix 16 13 25 12 29 17 26 138 Auto-painter 40 43 31 44 27 39 30 254
Volunteers were asked to choose which one they prefer between two pictures (pix2pix vs auto-painter).
Q3: In Sec 3.1 “Network Structure” and Sec 5, the paper should credit ‘pix2pix’ [10] for introducing U-Net architecture for the image-to-image translation. This article should not describe it as a contribution. A3: I will reorganize my contribution in the revised version. The ambiguous expression in section 5 caused the reviewer’s misunderstanding. The main contribution of this work is that we focus on interesting applications and show high resolution (512*512) results. The model can also accept the user input, which is a kind of potential digit entertainment. Reviewer 2# A1: The paper is using some existing technique (U-net, pix2pix) with some improvement to tackle a narrower task, cartoon image generation. It is good since the methods and the experiments are valid. Q1: Thanks for the reviewer’s positive feedback and comments. But the weak points are: 1) It is unclear if the methods can improve the image generation in broader applications. 2) the technical contribution is not much. It is very helpful if the authors can apply their modified methods on other applications, such as changing day images into night ones, etc.