"Burn marks" created by Neural-Style?

[ProGamerGov]

Is it caused by the model I am using? Can I change settings to get rid of these "burn marks"? They expand with each iteration like burning paper with a match.

Gfycat: https://gfycat.com/UniformWhiteClumber

This was the style image: https://i.imgur.com/Ddv78Io.jpg

[htoyryla]

Which model you are using? Which layers? Models that are not well-trained cause all kinds of effects like this.

[ProGamerGov]

The original image that Neural-Style output was only (512x288), but I was able to get up to (8192x4608) by using Waifu2x. The secret which let me do this was using the illustration2vec model.

Model files:

http://illustration2vec.net/models/illust2vec.prototxt

http://illustration2vec.net/models/illust2vec_ver200.caffemodel

Model Files are from this github page:

https://github.com/rezoo/illustration2vec

Re-sizing album: https://imgur.com/a/HsTZl

The reason for Waifu2x's effectiveness is because the model was trained on anime images, and so was Waifu2x. The only issue is the model seems to produce a decent amount of "burn marks" so finding a good image to re-size is hard. As for layers, I was just using a basic command:

th neural_style.lua -tv_weight 0.0001 -style_image trianglestyle.jpg -content_image detailed_animel.jpg -output_image out.png -save_iter 50 -num_iterations 1500 -model_file models/illust2vec_ver200.caffemodel -proto_file models/illust2vec.prototxt -backend cudnn -cudnn_autotune -optimizer adam

[htoyryla]

"The reason for Waifu2x's effectiveness is because the model was trained on anime images, and so was Waifu2x. The only issue is the model seems to produce a decent amount of "burn marks""

I have made several trials training a model with geometrical shapes of single color, and found that in neural-style they tend to produce colored blobs here and there in the image. I wonder if there is a connection... having large areas of single color tend to lead having blobs of color in the output.

[htoyryla]

Tried it with convis:

th convis.lua -image Ddv78Io.jpg -model models/illust2vec_ver200.caffemodel -proto models/illust2vec.prototxt -layer conv3_2 -output_dir convisimages

There is something strange going on... convis only finds three featuremaps per layer and the activations appear rather weak. I don't know if this could be a problem in convis... so far it worked fine with VGG19, VGG16 and NIN.

[ProGamerGov]

@htoyryla

When running Neural-Style with the model, there are warning: warning: module 'data [type Data]' not found, warning: module 'attr [type Data]' not found, and warning: module 'loss [type SigmoidCrossEntropyLoss]' not found

ubuntu@ip-Address:~/neural-style$ th neural_style.lua -tv_weight 0.0001 -style_image vgsn_larger.jpg -content_image andromeda.jpg -output_image out.png -save_iter 50 -num_iterations 1500 -model_file models/illust2vec_ver200.caffemodel -proto_file models/illust2vec.prototxt -backend cudnn -cudnn_autotune -optimizer adam

[libprotobuf WARNING google/protobuf/io/coded_stream.cc:505] Reading dangerously large protocol message.  If the message turns out to be larger than 1073741824 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 978590066
Successfully loaded models/illust2vec_ver200.caffemodel
warning: module 'data [type Data]' not found
warning: module 'attr [type Data]' not found
warning: module 'loss [type SigmoidCrossEntropyLoss]' not found
conv1_1: 64 3 3 3
conv2_1: 128 64 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv6_1: 1024 512 3 3
conv6_2: 1024 1024 3 3
conv6_3: 1024 1024 3 3
encode1: 1 1 50176 4096
encode2: 1 1 4096 1539
Setting up style layer          2       :       relu1_1
Setting up style layer          5       :       relu2_1
Setting up style layer          8       :       relu3_1
Setting up style layer          13      :       relu4_1
Setting up content layer        15      :       relu4_2
Setting up style layer          18      :       relu5_1
WARNING: Skipping content loss
Running optimization with ADAM
Iteration 50 / 1500
  Content 1 loss: 270.592461
  Style 1 loss: 993.490505
  Style 2 loss: 840.814304
  Style 3 loss: 194.942021
  Style 4 loss: 13.041255
  Style 5 loss: 0.462199
  Total loss: 2313.342746

Not sure if those warning contribute to or are the cause of the issue, or are the result of something else entirely.

[ProGamerGov]

I tested it at 2500 iterations and I found that the "burn spots" started to fade away. The image details under the "burn spots" were still as detailed as before. I wonder if the "burn spots" disappear completely after a certain number of iterations, if their size stays constant, or they begin growing again?

[htoyryla]

Those warnings relate to layers which are used only during training and thus are completely harmless. What I noticed looks like that either each layer has far fewer feature maps than what is typical, but it could as well be that somehow the software does not get to all of them. Now the question is does neural-style get all the feature maps. I could try it tomorrow. (Tried already... looks like neural-style gets 64 to 512 featuremaps per layer so I have to check why convis only processes three).

As to the spots disappearing after a while... perhaps one can accelerate this, either by increasing both content and style weights, or with adam, increasing learning rate. Slow learning might be connected to the model having weak activations. Just my feelings, though.

[htoyryla]

I made some tests with this model and the style image from your original post.

Using adam and the defaults weights, after 500 iterations, my face is faintly visible and there are indeed some spots.

Increasing weights (cw=1e4 and sw=1e5) brings about this, at 300 iter. It is not only that the spots disappear fast, they do not even appear at all.

I would like to test also default weights, adam and higher learning rate, as well as defaults but increasing content weight. It is only that without a GPU it takes time... on the other hand one perhaps learns to look at the loss values and learn from how they behave. Or so I like to think :)

PS. I still tested the first case but increased content_weight to 200. No spots any more, even at early iterations. So I would suggest experimenting with the content and style weights.

[htoyryla]

Tried this model again with convis, it now worked perfectly. Can't see what was wrong earlier. I can get all the feature maps there should be. And they appear to detect the features quite well. The only thing is the lack of contrast, the highlights are where you would expect but the rest is visible too, not black as with VGG19, for instance. I think that using larger content/style weights can compensate for this.