Issue getting simultaneous Deepdream working alongside Neural-Style

[ProGamerGov]

I am trying to get Fast-Neural-Style's DeepDreamLoss feature working in Neural-Style.

I currently have this modified version of Neural-Style setup: https://gist.github.com/ProGamerGov/9e7b109458828a1f9e7cf995a4d2fd4b, but the Deepdream loss remains 0 no matter what I do. Here is the command line output when using the Deepdream related parameters: https://gist.github.com/ProGamerGov/fa20f22cdd5a89c2ef998b2f6b716dd1

Currently in my modified version of neural_style.lua, there are two parameters: -deepdream_layers, and -deepdream_weights.

I tried to replicate the style/content loss functions with the Deepdream loss function as well as remaining faithful to the Fast-Neural-Style code. But I am not sure where exactly the Deepdream code should connect with the loss functions as the Fast-Neural-Style code is setup differently than Neural-Style's code.

[jcjohnson]

@ProGamerGov In DeepDreamLoss:updateOutput you are not updating self.loss, so it will always be 0. Computing the loss doesn't actually matter for creating images; it's just a convenient way for you to track progress. What actually matters is that you compute the gradient with respect to the loss, which your code seems to do correctly.

Therefore even though the loss remains 0, I think that your code should be able to produce DeepDream images if you get the hyperparameters right.

[ProGamerGov]

@jcjohnson Thanks for the reply. It turns out that I misspelled the -deepdream_weights parameter I was passing to nn.DeepDreamLoss, and that's why the code was not working. After some experimentation, I believe it now works like slow_neural_style.lua's DeepDream, but with all the features of the regular neural_style.lua.

I cleaned up the script by removing the unneeded parts that I originally added for DeepDream: https://gist.github.com/ProGamerGov/ff2715386d738e8245335b1a95617a1b

In DeepDreamLoss:updateOutput you are not updating self.loss, so it will always be 0. Computing the loss doesn't actually matter for creating images; it's just a convenient way for you to track progress.

The style and content loss functions update self.loss with the criterion. Is it possible to update the Deepdream self.loss without the criterion, and is the information obtained from doing so as useful as it is for the content and style loss functions?

[ProGamerGov]

I am attempting to get octaves working, but it looks like image.scale does not appear to work inside DeepDreamLoss:updateGradInput using the code below.

     local octaves = {}
     octaves[self.octave_n] = self.gradInput:add(self.clipped, -self.max_grad):div(self.max_grad) 
     for i=self.octave_n-1,1,-1 do
        local _,h,w = unpack(self.base_img:size():totable())
        octaves[i] = image.scale(octaves[i+1], math.ceil((1/self.octave_scale)*w), math.ceil((1/self.octave_scale)*h),'simple')
     end

From DeepDreamLoss:__init:

  self.octave_n = octave_n or 4
  self.octave_scale = octave_scale or 1.4

  self.base_img = base_img

base_img is the resized input image:

-- DeepDream contrast fix 
  if params.deepdream_contrast ~= '' then
    for i = 1, #deepdream_losses do
      deepdream_losses[i].mode = 'contrast'

      local content_image_size = image.load(params.content_image, 3)
      content_image_size = image.scale(content_image_size, params.image_size, 'bilinear')
      deepdream_losses[i].base_img = content_image_size
    end
  end

The full modified neural_style.lua can be found here: https://gist.github.com/ProGamerGov/ad143adf1b7d4eb2bd2d177153abc00e

Edit:

Looking at how the total variance code works, would something like this solve the problem?

self.gradInput:resize(2, octave_h, 3, octave_w)

Second Edit:

This code here:

if self.mode == 'contrast' then
     self.gradInput = self.gradInput:mul(1/self.gradInput:max())
     local octaves = {}
     octaves[self.octave_n] = self.gradInput:add(self.clipped, -self.max_grad):div(self.max_grad) 
     for i=self.octave_n-1,1,-1 do
        local _,h,w = unpack(self.base_img:size():totable())
        local octave_w, octave_h = octaves[i+1], math.ceil((1/self.octave_scale)*w), math.ceil((1/self.octave_scale)*h)
        --self.gradInput:resize(_, h, w)
        self.gradInput = self.gradInput:view(_, h, w)
        self.gradInput:resize(2, octave_h, 3, octave_w)
     end
  end

Results in this error:

/home/ubuntu/torch/install/share/lua/5.1/torch/Tensor.lua:462: Wrong size for view. Input size: 256x132x250. Output size: 3x526x1000

Full error message: https://gist.github.com/ProGamerGov/28d48c2a164199610576c31aa54e4033

Third Edit:

This seems to work, but the output image now has a weird blurring effect:

if self.mode == 'contrast' then
     --self.gradInput = self.gradInput:mul(1/self.gradInput:max())
     local octaves = {}
     octaves[self.octave_n] = self.gradInput:add(self.clipped, -self.max_grad):div(self.max_grad) 
     for i=self.octave_n-1,1,-1 do
        --octaves[i+1]
        local c,h,w = unpack(self.base_img:size():totable())
        local octave_w = math.ceil((1/self.octave_scale)*w) 
        local octave_h = math.ceil((1/self.octave_scale)*h)
        self.gradInput:resize(1, c, 2, octave_h, 3, octave_w)
     end
  end

Fourth Edit:

I have gotten the code to run without an error, but I don't think that the octaves are having any effect:

function DeepDreamLoss:updateGradInput(input, gradOutput)
  if self.mode == 'contrast' then
     local octaves = {}
     octaves[self.octave_n] = self.gradInput:add(self.clipped, -self.max_grad):div(self.max_grad) 
     for i=self.octave_n-1,1,-1 do
        octaves[i] = octaves[i+1]
        local c,h,w = unpack(self.base_img:size():totable())
        local octave_w = math.ceil((1/self.octave_scale)*w), octaves[i+1] 
        local octave_h = math.ceil((1/self.octave_scale)*h)
        self.gradInput = self.gradInput:resize(1, c, 2, octave_h, 3, octave_w)
     end
  end
  self.gradInput = self.gradInput:mul(1/self.gradInput:max())
  self.gradInput:resizeAs(gradOutput):copy(gradOutput)
  self.clipped:resizeAs(input):clamp(input, -self.max_grad, self.max_grad)
  self.gradInput:add(-self.strength, self.clipped)
  return self.gradInput
end

This is the latest version: https://gist.github.com/ProGamerGov/c3eab9584d4d00cfb60d35953894788d

[htoyryla]

@ProGamerGov at this line you effectively make gradInput a copy of gradOutput, which is probably why you avoid the errors, but you also lose everything you have done with gradInput so far.

 self.gradInput:resizeAs(gradOutput):copy(gradOutput)

Furthermore when looking at your code, you initialize several variables in your class but you don't pass the values as parameters, so they will use the defaults (40, 5, 'none'). Note in particular that your base_img is not an image, it is a text string 'none', self.mode is also always 'none' so the part of code using octaves is never run.

self.mode = mode or 'none' self.octave_n = octave_n or 40 self.octave_scale = octave_scale or 5 self.base_img = base_img or 'none'

As to the octaves part... I don't understand what it is trying to achieve, so I will only say that one needs to understand quite well what one is doing when writing an updateGradInput function.