how did you create the mobilenet visualizations?

[JaspervDalen]

in this nvidia article (linked below) about your work you show layer visualizations of mobilenet. I tried to do that in a project of mine but it failed with multiple libraries (e.g. lucid : https://github.com/tensorflow/lucid/issues/68). So I was wondering how you did it.

article: https://devblogs.nvidia.com/deep-learning-hotel-aesthetics-photos/?nvid=em-ded-63644&mkt_tok=eyJpIjoiT1dGaE5ERm1NbVV5TmpreSIsInQiOiJOK0pzQnBxclNwUWNKNmNoNUNYbCtDVUlibzAzM21uYnNsRTBvZFhnWlk0RnJwZmltZHdRTFZTXC9Idk5vQXg2K3hjWE94cE9UU2NnV0xWVk1tWnhHSGhLUVZMbW84R1Z0QUN3RVNVaWdqS05yR1ZaR0FEbmFCMGlmNTZ5bndMNGcifQ%3D%3D

[clennan]

Hi, we used Lucid as well (we mentioned and linked to Lucid in the article). We haven't open sourced the code for the filter visualizations yet, will try to do that over the Christmas period

[clennan]

Here is a minimal example to visualize the first filter in layer 23 in MobileNet, it should produce

Hope it helps..

import os
import tensorflow as tf

import lucid.optvis.param as param
import lucid.optvis.render as render
import lucid.optvis.transform as transform
from lucid.modelzoo.vision_base import Model

from keras import backend as K
from keras.applications.mobilenet import MobileNet
import matplotlib.pyplot as plt

WORK_DIR = os.getcwd()
CONV_LAYER = 23
CONV_FILTER = 0

# save MobileNet in TF pb format
tf.reset_default_graph()
sess = tf.Session()
K.set_session(sess)

model_keras = MobileNet(include_top=False)

frozen_graph_def = tf.graph_util.convert_variables_to_constants(
        sess,
        sess.graph_def,
        [model_keras.outputs[-1].op.name],
        )

with open(os.path.join(WORK_DIR, 'model_tf.pb'), 'wb') as f:
    f.write(frozen_graph_def.SerializeToString())

# convert model for lucid
class Mobilenet_lucid(Model):
    def __init__(self, model_path, labels_path, input_name, image_shape=None, image_value_range=None):
        self.model_path = model_path
        self.labels_path = labels_path
        self.input_name = input_name
        self.image_shape = image_shape
        self.image_value_range = image_value_range
        super().__init__()

# locations
image_shape = [224, 224, 3]
image_value_range = [-1, 1]

param_f = lambda: param.image(128, fft=True, decorrelate=True)

input_name = model_keras.input.name
model_path = os.path.join(WORK_DIR, 'model_tf.pb')
model_lucid = Mobilenet_lucid(model_path, None, input_name, image_shape, image_value_range)
model_lucid.load_graphdef()

layer_render = model_keras.layers[CONV_LAYER].output.name.split(':')[0]+':'+str(CONV_FILTER)
vis = render.render_vis(model_lucid, layer_render, param_f)

fig = plt.figure()
plt.imshow(vis[0][0][0, ])
plt.axis('off')
fig.savefig(os.path.join(WORK_DIR, 'vis_layer{}_filter{}.png'.format(CONV_LAYER, CONV_FILTER)))

[JaspervDalen]

Thank you for this example and for sharing your whole repo!