Visualizing Custom Models

[suddhasourav]

Dear @pokman , many thanks for this wonderful package!

I was wondering if it would be possible to visualize custom models, e.g. a simple sequential keras model. Currently I get the following error while trying to render a channel objective:

ValueError: Input 0 of layer "model_4" is incompatible with the layer: expected shape=(None, 200, 200, 3), found shape=(1, 227, 227, 3)

The weird thing is that the error messages seem to follow this format:

ValueError: Input 0 of layer "model_<n>" is incompatible with the layer: expected shape=(None, 200, 200, 3), found shape=(1, <x>, <x>, 3)

At each iteration of the code block (objective+render), n increases by 1, and x seems to oscillate.

The model I am trying to visualize is a simple cnn:

model = keras.Sequential([
    keras.layers.Conv2D(16, (3, 3), activation='relu', input_shape=(HEIGHT, WIDTH, 3)),
    keras.layers.MaxPooling2D((2, 2)),
    keras.layers.Conv2D(32, (3, 3), activation='relu'),
    keras.layers.MaxPooling2D((2, 2)),
    keras.layers.Conv2D(64, (3, 3), activation='relu'),
    keras.layers.MaxPooling2D((2, 2)),
    keras.layers.Flatten(),
    keras.layers.Dense(512, activation='relu'),
    keras.layers.Dense(1, activation='sigmoid')
])

When I load the InceptionV3 model with include_top=True, I get similar errors. Inspired by this I tried popping the dense layers in the simple keras model, but it did not work. I would be very grateful for your suggestions!

[pokman]

Hi @suddhasourav,

Thanks for your interest in this package, and for raising this issue. I just had a look into why that happens. Let me describe the cause of the issue and then a temporary fix, before implementing a more permanent solution later.

Cause: In the main rendering function, by default, a sequence of perturbations are applied to the image at each optimization step, and every time these perturbations change the size of the image randomly. In particular, when the objective function is evaluated, the size of the image is in fact different from the specified value. This leads to an error if the objective function comes from a CNN with specified input height and width, like your custom model or InceptionV3 with the top. (In contrast, InceptionV3 without the top does not have a specified input height and width, and so it can take images of different sizes.)

inception_v3 = tf.keras.applications.InceptionV3(include_top=True) 
inception_v3.input.shape   # TensorShape([None, 299, 299, 3])

inception_v3 = tf.keras.applications.InceptionV3(include_top=False) 
inception_v3.input.shape   # TensorShape([None, None, None, 3])

Temporary fix: When working with a CNN with specified input height and width, you can override the default perturbation sequence with a dummy one that does nothing to the image (while supplying the right image size), i.e.,

render.render_vis(objective, img_size, transforms=[lambda x: x])

Note that transforms=[] would be interpreted as using the default, size-changing perturbation sequence.

I will try to find a better way to resolve this issue. In the meantime, I hope this helps and please let me know if you have any other questions.

Best, Pokman

[pokman]

I have now updated the repo to address the issue.

Update: In the main rendering function, after the specified or default sequence of perturbations are applied to the image, the image is then rescaled to the specified size before evaluation of the objective function.