Keras Lambda Layer Warping Image Input

[crvogt]

I'm using a network composed of two separate CNNs; the first outputs a depth map. Before the second, I have a lambda layer that takes the depth map and a regular 3 channel image as input and warps it, returning the warped image. The warping function uses RectBivariateSpline.

My trouble lies in trying to pass channels to RectBivariateSpline, I'm getting the sense that the operation is illegal, ie, RectBivariateSpline won't accept a tensor. So either I'm doing the lambda layer wrong, or I need to split the model into two submodels? Some direction would be very appreciated.

def warp2DImage(args):
    depth_map, img2D = args
    z = interp2WithMap(depth_map, img2D)
    return z

def interp2WithMap(depth_map, img2D):
    img2DShape = img2D.get_shape().as_list()
    batch = img2DShape[0]
    height = img2DShape[1]
    width = img2DShape[2]
    channels = img2DShape[3]
    x = np.arange(0, width, 1)
    y = np.arange(0, height, 1)
    Z = np.zeros(img2DShape[1:4])

for it in range(1, channels):
    Z = img2D[:, :, it]
    interped = RectBivariateSpline(y, x, img2D[:, :, it], kx=1, ky=1)
    numElem = depth_map.shape[1]*depth_map.shape[2]
    depth_map = np.reshape(depth_map, (1, numElem))
    x2 = x + depth_map
    y2 = y + depth_map
    Z[:, :, it] = interped(y2, x2)
    Z[:, :, it] = Z[:, :, it].astype(np.uint8)

    return Z

# Passing values to depth net
depthMap = depth_net(dFeatures)

# Lambda function layer taking depthMap to warp img2D to the refPos 
position to be compared with refImg
cConvInput = Lambda(warp2DImage, output_shape=(cHEIGHT, cWIDTH, cCHANNELS))([depthMap, 
img2D])

finalImg = color_net(cConvInput)

[farizrahman4u]

No, you cant use numpy methods in a Lambda layer.

[crvogt]

This makes sense, I assume it would disconnect the graph. Would the solution be to create two separate models? I guess it depends on if I can use numpy methods on an input. Ideally, I'd be able to take the output from one model, run some operation on it not native to keras, and set that as the input to the next model, and both models would be trained.

[farizrahman4u]

That would break the backprop. Soultion would be to rewrite the function using Keras/TF Ops.

[crvogt]

Yes, coming from matconvnet I'd been hoping there was a way around the backprop issue. But now that you mention it, quick search yields tf.contrib.image.dense_image_warp which should do what I want. Thanks for taking the time :)