search

NervanaSystems / neon

Intel® Nervana™ reference deep learning framework committed to best performance on all hardware
http://neon.nervanasys.com/docs/latest
Apache License 2.0
3.87k stars 811 forks source link

Instance normalization for fast style transfer #431

Closed sangameshnr closed 6 years ago

sangameshnr commented 6 years ago

Hi, I am trying to implement a custom layer in neon for Instance normalization as given in this paper, https://arxiv.org/abs/1607.08022. It is different from BatchNormalization in a way that for an image tensor (C,H,W,N), the mean and variance are calculated only for HxW dimensions. I created an op-tree and the autodiff generates an error which is quite obscure. Please see the code below,

be = gen_backend('mkl') Im = be.array(np.random.randn(3,32,32,8)) #(C=3,H=32,W=32,N=8) x = Im.reshape(3,32*32,8) mean = be.mean(x,axis=1) var = be.var(x,axis=1) F_op_tree = (x - mean) / be.sqrt(var + eps ) # eps=1e-4 ad = Autodiff(op_tree=mean,be=be,next_error=None) g = ad.get_grad_asnumpyarray([x]) print(g)

This code gives me the following error: File "/usr/local/lib/python2.7/dist-packages/nervananeon-2.4.0-py2.7.egg/neon/backends/autodiff.py", line 94, in _unbroadcast out_shape = grad_op_tree.shape AttributeError: 'NotImplementedType' object has no attribute 'shape'

I have noticed that the Autodiff gives this error for a 3-d tensor input and works fine for 2-d tensor. Could anyone please help me understand what's happening here?

baojun-nervana commented 6 years ago

@sangameshnr It seems you encountered a scenario which is currently not supported. The batchnorm reduced 1 dimension and created a shape mismatch. Would you be able to work around in a 2d domain?

sangameshnr commented 6 years ago

Hi, Thanks for the reply. I was able to do that in 2d after a couple of transposes and reshapes.

Following up, I have another question. I need to tap outputs from multiple convnet layers of my VGG pretrained network. Is there a simple way to do that? model.get_ouputs() gives final layer outputs but I need outputs from some middle layers also in the VGG-16 network that has "pretrained weights" loaded into it. I am modifying the sequential container "fprop" to get that done. Is there a simple way to do that?

sangameshnr commented 6 years ago

I figured out that I need to use model.layers.layers[i].outputs after running model.get_outputs() to get outputs from different layers.

Thanks.