How to make my own channel spritemaps for my TensorFlow model？

[PaulChongPeng]

Does lucid support to make my own TensorFlow model visualizations now？And how to make my own channel spritemaps？ Thanks！

[ludwigschubert]

We will have both soon! There will be a notebook showing how to import a custom model, and a recipe for creating sprites as well.

[vijayvee]

Has this functionality for visualizing custom tensorflow models been added? Thanks!

[ludwigschubert]

@vijayvee we have! Unfortunately documentation is rough because of the many ways people store their models. Read through the linked notebook, and give feedback if the instructions don't work for you. :-)

[vijayvee]

Sure, I shall check it out. Thanks! :dancing_men:

[vijayvee]

@ludwigschubert I did go through the notebook, it's great! Although it looks like I'm stuck at a point where tensorflow computes an output size that goes negative.

I am trying to visualize kernels in my custom Resnetv2 model with a [224,224,3] input layer.

Right before the global average pooling in the end, where the actual spatial dimensions should be [7,7], the model building code in render_vis receives a tensor with spatial dimensions [5,5]. As a result, while lucid build the model, it errors out before applying an average pool with kernel size [7,7], since the input it receives is an [n,5,5,c] tensor.

Am I messing up with some default padding that's being added to the input before lucid builds the model? Have you encountered this sort of an error before with lucid?

Error:

InvalidArgumentError (see above for traceback): Computed output size would be negative: -1 [input_size: 5, effective_filter_size: 7, stride: 1] [[Node: import/cnn/average_pooling2d/AvgPool = AvgPoolT=DT_FLOAT, data_format="NHWC", ksize=[1, 7, 7, 1], padding="VALID", strides=[1, 1, 1, 1], _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

[ludwigschubert]

Are you sure your Resnetv2's input size is not 229x229x3?

We are trying to support various input sizes (one effort, for example, is this forget_xy function). However, if you're actually visualizing a kernel that's pretty deep into your architecture, the pooling can be set up in such a way that you do need to give it (at least) the input size it was built for.

[vijayvee]

Yes, I am sure my model takes an input of size [224,224,3]. I am able to render visualizations if I switch the size of my average pool from 7 to 5, although the rendered visualizations are not super-interpretable. I'm guessing the model being loaded is in some way different from the model stored in the checkpoint :thinking:

Anyway, I can dig deeper into that. Is there also a way to create those amazing sprite-maps of the sorts I see here for my custom model? If yes, could you please direct me to it?

[ludwigschubert]

No guarantees, but things you can try:

• Use a bigger input. At some point you should at least get a different error—in all cases I have seen pooling is set up in such a way as to downsample inputs by a constant factor, so a larger input will at some point cause the layer you're concerned about to have a width of 7 rather than just 5.
• Create a colab notebook that reproduces your error and we can take a look.

We have no code we can share for spritesheet creation yet. Conceptually, we created the spritesheets by visualizing each individual neuron/channel in the network—so no magic.

[ludwigschubert]

@vijayvee I realized I left an ambiguity in my wording earlier. Are you specifying an input size of 224x224 for lucid? That is, are you using param.image(224,…)? If you don't specify a custom parameterization, lucid defaults to 128x128. This behaviour could cause the error you are seeing.

[vijayvee]

@ludwigschubert THAT FIXED IT! :dancer: I guess it would be neat if you could emphasize somewhere early in the tutorial notebook about passing the custom model input size through param.

Also, I noticed that there could have been a little demo code for exporting a graphdef. I've added a couple comments in the same notebook with a link to some demo code. Thanks!

[JaspervDalen]

@ludwigschubert Is there already some code available to create spritemaps?

[vijayvee]

@JaspervDalen You could check this notebook for something close. This notebook will walk you through creating visualizations for all conv layers in a network, but you could swap this out with any (or all) layer(s) you want to visualize.

Stitching the resulting images into a spritemap should be trivial. Hope this helps.

[JaspervDalen]

@vijayvee My main problem is finding which layers have gradients and thus work when you put in a network. I want to be able to visualize every node in every layer that is able to be visualized but I can't find out how to find them. I hoped that it would have been in their sprite map notebook

[vijayvee]

What I get from your previous message is, your goal is to find out the layers through which gradients are passing. The easiest way to just do this would be to use tf.gradients(). Please clarify if I misunderstood your goal, thanks.

[JaspervDalen]

@vijayvee how would you use that? I only have a layer no cost function or something, I cannot do something like or do not understand correctly how tf.gradients works: all_layers = [node.name for node in eid.graph_def.node if tf.gradients(node)]

[hegman12]

@JaspervDalen if you are still interested, use below code to display the ops in the graph. From there you can filter the tensors/ops and then pass it to render_viz to visualize them

import lucid.modelzoo.util as util gd=util.load_graphdef(<your path to protocol buffer>) tf.import_graph_def(gd) graph=tf.get_default_graph() graph.get_operations()

This will display the operations and their type. You can pick those ops which says Conv2D which are for convolutions. You can also directly extracts nodes from gd and then check them out.

[waqarws]

@ludwigschubert THAT FIXED IT! 💃 I guess it would be neat if you could emphasize somewhere early in the tutorial notebook about passing the custom model input size through param.

Also, I noticed that there could have been a little demo code for exporting a graphdef. I've added a couple comments in the same notebook with a link to some demo code. Thanks!

I have a model with input 224x224 I am trying to visualize the logits using class_logits objective. I use (224, 224) as parameter to param.image(). It gives me the following error. What is going wrong.

InvalidArgumentError (see above for traceback): Input to reshape is a tensor with 256 values, but the requested shape requires a multiple of 1024 [[Node: import/Reshape = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:CPU:0"](import/max_pooling2d_3/MaxPool, import/Reshape/shape)]]

[tul-urte]

I've made myself a kit for working with Lucid on the many families of frozen graphs produced by my experiments (with SSD Mobilenet PPN).

https://github.com/brentcroft/lucid-spritemaps

I'm experimenting with varying transforms across layers (rotations kill very early layers), and intercepting bad visualizations (i.e. when they end up just grey) by detecting when loss recurs. There are crude implementations of both in the kit.