Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN

[reyllama]

Key Idea

• Propose feed-forward network that applies the style of a painting to a sketch
• Explain why U-net training might fail, and propose a remedy using two guide decoders
• Suggest AC-GAN based discriminator to deal with paintings of different styles

Previous Approaches

Neural Style Transfer

• Applies style of one image to content of the other
• Concept of "style" here is very broad, so it doesn't necessarily perform colorization

Pix2Pix

• cGAN family perform transformation between paired images
• The quality spoils if the information quantity difference between input and output is large
  • This task = SKETCH (very little information) + PAINTING (huge information)
• Suitable for simple transformation (ex) edge2cat, bag2edge

Paintchainer

• Abandoned cGAN and moved to unconditional discriminator
• Best quality so far

Formulation

• Overall Generative Adversarial Network Structure

• VGG 16 / 19 fc_layer_1 output provides global style hint which is added at the nadir of U-net

• AC-GAN discriminator reveals whether real or fake + classification of style

• Generator Architecture

• Sketch input → Feature Map → Global Style Hint added → Upsampled → Output Image

  • 2 Guide Decoders right before & after Global Style Hint
    • U-net can be lazy, not incorporating noisy-like information of style hint and skip ahead
    • Force the network to make operations at the bottleneck by imposing two more losses
    • Reconstructed image from feature map right before is compared with gray-scale
    • Reconstructed image from feature map right after is compared to target image
  • VGG fc_layer_1 output consisting of 2048 or 4096 dimension vector

Lazy U-net

• If U-net finds it sufficient to handle task with low-level layers, higher level layers are simply skipped.
• Above shows simple copying task - No gradient propagated to higher level layers
• In the model, sudden injection of global style hint can be overwhelming to untrained network
  • The model might view this as random noise and skip the connections - No Gradient!

• This is the ideal architecture to fully utilize layers of U-Net.
• That is, force layers at each level to learn by imposing separate losses.
• In the model, use guide decoders to reconstruct image from feature maps right before & after the global style hint and compute losses.
  • Higher layers must produce meaningful feature maps (activated) to avoid great loss.

Loss

Generator

• x = sketch, y = paint, V(x) = VGG output, f=full, $g_1$=guide decoder 1, $g_2$=decoder 2
• I think $V(x)$ should be $V(y)$ since it performs feature extraction from paint.
• $T(y)$ maps paint image to gray scale, since $g_1$ is yet a weak generator.

GAN

• Traditional GAN loss. Discriminator maximizes, and generator minimizes it.

Full Objective

Experiments

• No quantitative evaluation, just some (cherry picked?) sample outputs.
• It incorporates colors of style image well to given sketch.

Comments

• Interesting topic and very pragmatic applications can be made.
• I liked how they explained U-net can be lazy, that some loss enforcement can be helpful.
• Still not clear what advantage there is in using AC-GAN discriminator.
• The overall architecture is quite huge, and it relies on pre-trained VGG, which clearly is a limitation.
• More room for manual interpolation would be nice. It roughly follows the color palette of given style paint image, but illustrator/creator might want to make more specific directions.

[rapidrabbit76]

what is the value of λ that adjusts L_{l_1} used for adversarial loss in G ? similar to pix2pix, is the coefficient value used as 100?

[reyllama]

That I'm not sure about. We should probably look at the official codes and its configurations, if available.