Output Stylized Image looks different on torch and coreml

[engahmed1190]

I have used same image on both models , Torch model present a good output while the Coreml model changes the main picture color .

i used this image for testing

the output from torch model is

Coreml model provides this

the model trained options

percep_loss_weight: 1   
padding_type: reflect-start 
batch_size: 4   
arch: c9s1-32,d64,d128,R128,R128,R128,R128,R128,u64,u32,c9s1-3  
resume_from_checkpoint:     
style_image: images/styles/home.jpeg    
max_train: -1   
style_image_size: 700   
tv_strength: 0.001  
lr_decay_factor: 0.5    
checkpoint_name: checkpoint 
loss_network: models/vgg16.t7   
gpu: 0  
content_layers: 16  
task: style 
use_instance_norm: 1    
tanh_constant: 150  
preprocessing: vgg  
style_weights: 5,5,5,5  
checkpoint_every: 1000  
num_val_batches: 10 
num_iterations: 10000   
use_cudnn: 1    
pixel_loss_weight: 0    
content_weights: 1  
style_target_type: gram 
weight_decay: 0 
pixel_loss_type: L2 
lr_decay_every: -1  
learning_rate: 0.001    
style_layers: 4,9,16,23 
backend: cuda   
upsample_factor: 4  
h5_file: /home/Desktop/coreml/models/file.h5    

[engahmed1190]

my first thoughts that it's related to this part of the code

if is_grayscale:
        gray_bias = deprocessing_args.get('gray_bias', 0.0)
        W = np.array([image_scale])
        b = np.array([gray_bias])
    else:
        W = np.array([image_scale, image_scale, image_scale])

        red_bias = deprocessing_args.get('red_bias', 0.0)
        green_bias = deprocessing_args.get('green_bias', 0.0)
blue_bias = deprocessing_args.get('blue_bias', 0.0) 

[opedge]

@engahmed1190 I can try to investigate and debug this problem if you could attach your torch7 model.

[engahmed1190]

here is the torch model Model Link

[opedge]

@engahmed1190 I tried to run the model which you provided with latest torch7 installation (without CUDA on macOS) using this command: th fast_neural_style.lua -model home.jpeg_700.t7 -input_image test.jpg -output_image test_result.jpg

I got this output:

So it looks like your output from CoreML. May be something inconsistent with torch7 installation or with fast-neural-style repo revision that you use.

[engahmed1190]

i have tried the same model but different result can you please try the same model but with batch normalization , it works with coreml in batch normalization Batch Model

[engahmed1190]

can you suggest a solution for this case @opedge seems to be working with batch norm and instance norm getting reddish

[gbildson]

I've noticed that all new instance norm models don't work when converted to coreml. Also, all historical models in fast-neural-style don't work in torch. Have yet to figure out the connection.

[gbildson]

I found an older machine with an older torch setup (~ 6 months) and the server images for old models generate fine. I then stepped through the updates that I had applied to the newer machine.

luarocks install torch luarocks install nn

The server started screwing up image generation for old models after applying the "luarocks install nn" step.

This makes sense I believe because the pytorch convert step always screws up on new models and I suspect it has something to do with the use of "from torch.nn import InstanceNorm3d".

[AndreJFBico]

Ive also been having issues with instance normalization models that i have trained in torch and converted to coreml.

Even training with the exact same parameters as the johnson pretrained models it causes issues.

Im trying now with batch normalization.

[gbildson]

Yeah, batch works but lower quality and/or bigger model.

[opedge]

For appropriate conversion may be you need to implement InstanceNorm layer by yourself like I did in sample converter but using another algorithms than InstanceNorm3d layer from pytorch.

[AndreJFBico]

If your still interested about instance normalization i ended up using this pytorch https://github.com/abhiskk/fast-neural-style implementation, and its instance normalization works straight off the bat.

I adjusted the current torch2coreml implementation and adjusted it to convert the pretrained models present in the link above, directly from pytorch instead of legacy torch.

This is their instance normalization code.

class InstanceNormalization(torch.nn.Module):
    """InstanceNormalization
    Improves convergence of neural-style.
    ref: https://arxiv.org/pdf/1607.08022.pdf
    """

    def __init__(self, dim, eps=1e-9):
        super(InstanceNormalization, self).__init__()
        self.scale = nn.Parameter(torch.FloatTensor(dim))
        self.shift = nn.Parameter(torch.FloatTensor(dim))
        self.eps = eps
        self._reset_parameters()

    def _reset_parameters(self):
        self.scale.data.uniform_()
        self.shift.data.zero_()

    def forward(self, x):
        n = x.size(2) * x.size(3)
        t = x.view(x.size(0), x.size(1), n)
        mean = torch.mean(t, 2).unsqueeze(2).unsqueeze(3).expand_as(x)
        # Calculate the biased var. torch.var returns unbiased var
        var = torch.var(t, 2).unsqueeze(2).unsqueeze(3).expand_as(x) * ((n - 1) / float(n))
        scale_broadcast = self.scale.unsqueeze(1).unsqueeze(1).unsqueeze(0)
        scale_broadcast = scale_broadcast.expand_as(x)
        shift_broadcast = self.shift.unsqueeze(1).unsqueeze(1).unsqueeze(0)
        shift_broadcast = shift_broadcast.expand_as(x)
        out = (x - mean) / torch.sqrt(var + self.eps)
        out = out * scale_broadcast + shift_broadcast
        return out

[opedge]

@AndreJFBico sounds great! May you'd like to make a PR for your pytorch models support and instance_norm layer?

[RahulBhalley]

@AndreJFBico can you please provide the modified torch2coreml code for direct conversion from PyTorch pre-trained model to CoreML?