Major features are extended siamese network support, visualizing histograms and correlation matrices built from activations and weights, extended deconv options, added grayscale support, improved optimize_image and find_max_acts, improved user interface, refactored and fixed bugs

[arikpoz]

Features

- Siamese network support

• Added support for loading Siamese networks, presenting the inputs side-by-side, and properly calculate activations and gradients for visualization
• Supports two types of Caffe based Siamese network structures: using shared weights, and using the "batch trick", i.e. images run thru the network in batch and then split and concat at the end
• Supports two types of input formatting for Siamese networks:
  • Double inputs are concatenated chanellwise, i.e. 6 channels for two color images, 2 channels for two grayscale images
  • Double inputs are concatenated along width, i.e. the images are laid side-by-side, concatenated on the width axis
• Revised find_max_acts.py and max_tracker.py, and subsequently crop_max_patches.py to work properly on siamese network
  • Handle siamese case where two layers generates activations for same neuron
  • Avoid input duplications while searching for max input images
  • Handle output generation for siamese network with duplicate layers
• Changed optimizer to be able to optimize over a Siamese network
• Added support for camera mode in siamese network
• Added keyboard binding for changing the siamese view input mode showing only the first image, only the second image, or both

- Activation and Weights Histograms

• Added support for maximal input histograms for each neuron. The basic data is gathered during max tracker operation, but the actual histogram image generation can be done both in the offline script and during runtime
• Added support for weights histogram view

- Added Correlation Visualization

• Visualize activations correlation
• Visualize weights correlation

- Improved max tracker

• Added batch support to find_max_acts
• Added support for skipping already generated files

- Improved optimize_image.py

• Added support for running on multiple layers, and multiple channels in a layer
• Only save the relevant part of the image, according to the receptive field
• Added batch support
• Added support for skipping files for units which were already generated
• Changed naming convention of output files
• Added support for loading pattern images in new format (directly from disk, instead of having a sprite image)

- Other models

• Identify MaxOut activation structures: double channels, split, elementwise-max
• Added support for grayscale models
• Updated setting files for caffenet, googlenet and squeezenet

- Deconv

• Added support for switching between running "Zeiler & Fergus" deconv, and "guided backprop" deconv
• Added softmax raw view option, and histogram view option
• Added display of grad values when we are in backprop mode

- User interface

• Added support for loading offline visualization in the format which max tracker generates
• Added option to show maximal input images as patterns
• Added option to show first or all images per neuron
• Added option to show score of maximal images
• Added show of groundtruth input label in status line
• Show current selected score in last layers
• Added 'B' key binding to back cycle backprop options in the UI
• Added option to overlay activation/gradient over original image
• Added color maps options
• Added option to overlay colormap with input image
• Added option to select deconv target layer
• Made control pane height be determined automatically
• Added layer size and receptive field to status line
• Added limitation in load_image_per_unit() to avoid hanging for loading >1000 images
• Added proof of concept for web based UI, currenly read only
• Added model name to window title
• Added input file names to second status line

Updated settings parameters

• Added caffe_net_raw_scale - network raw scale
• Added caffe_net_image_dims - network image dims
• Added is_siamese - is the network loaded a siamese network?
• Added convert_label_fb - function to apply on input label, e.g. convert probability into 0/1 categories
• Added layers_for_max_tracker - list of labels used to use in MaxTracker
• Added normalize_layer_name_for_max_tracker_fn - function to normalize layer name used in NetMaxTracker
• Added siamese_layer_to_index_of_saved_image_fn - function which selects an image from a siamese input pair
• Added find_max_acts_output_file - location for max tracker output file
• Added control_pane_height - control pane height
• Added input_scale and transpose to settings
• Renamed caffevis_unit_jpg_dir to caffevis_outputs_dir
• Removed find_max_acts_output_file setting - file name is deduced automatically
• Replaced max_tracker_output_dir with caffevis_outputs_dir
• Removed optimize_image_output_prefix
• Removed siamese_network_format, replaced with layer based information where needed
• Removed caffevis_score_layers setting, value is now deduce automatically
• Deduce automatic default value for caffe_net_channel_swap from network proto
• Changed default value for show_maximal_score - default is now on
• Added code to support replacing of max_tracker_layers_list with automatic data from network definition

Refactored

• Updated function find_max_acts to work with settings file
• Moved mean loading to separate function, to be used in find_max_acts
• Moved functions get_files_from_directory, get_files_from_image_list and get_files_from_siamese_image_list into misc.py
• Made max_tracker more generic
• Disabled generation of some of the outsputs by default
• Moved caffe dependency to local function in caffevis_helper.py
• Moved caffe_misc module to be able to use it from both max_tracker and optimizer
• Moved load_imagenet_mean from loaders into caffevis_helper
• Clean up code around: range_scale, net_channel_swap
• Removed unused hardcoded functions: hardcoded_get, load_labels, load_trained_net
• Removed now empty file loaders.py
• Refactor mean setting
• Extracted new method: _add_label_or_score_overlay to simplified code structure
• Extracted new function _display_pane_based_on_zoom_mode to organize the code
• Refactored function _draw_layer_pane
• Organized uses of mkdir_p
• Organized the help screen
• Added user settings file (settings_user.py.example) which contains model_to_load and gpu settings
• Added settings_model_selector.py
• Added caffevis_clear_negative_activations which is False by default to allow viewing max input images with negative activations
• Made get_receptive_field implementation lazy, to speed up loading of big networks (resnet, googlenet)
• Moved code for calculating histograms to end of processing max tracker
• Refactor loading method of network settings file
• Use RegionComputer.convert_region_dag in get_max_data_extent() and compute_data_layer_focus_area()
• Renamed is_conv with is_spatial
• Added function to handle missing force_backwards and to upgrade network definition to latest
• Replaced max_tracker_layers_to_output, layers_for_max_tracker and optimize_image_push_layers with max_tracker_layers_to_output
• Added code to set batch size to 1 in deepvis ui
• Added helper scripts to run image generation in one line
• Removed use of check_force_backward_true

Bug fixes

• Fixed bug in backprop image generation
• Fixed bug in image generation of multiple layers at once
• Added conversion of layer name to top name in max tracker search, this fix bugs related to blob-name/layer-name confusion
• Fixed crash when trying to deconv or backprop from invalid layers
• Fixed so that current deconv shows only selected image deconvolution
• Fixed bugs with layer-name/blob-name confusion in optimizer
• Fixed some bugs in offline image loading
• Fixed presentation bug of negative images
• Fixed crash when max tracker working on FC layers
• Fixed backprop/deconv for layers with top name different from layer name
• Fixed image loading in max tracker
• Fixed bug in optimize_image when data_mean is channelwise instead of image mean
• Fixed bug in loading images patterns
• Fixed bug in loading non square images
• Added empty implementation to avoid crash when patterns view with maximal input images runs with settings.caffevis_unit_jpg_dir_folder_format == 'original_combined_single_image'"
• Fixed bug when cache folder doesn't exist
• Fixed bug when moving between layers and siamese mode is first or second, we can't move into the other modes
• Fixed bug when data_mean is empty
• Fixed bug related to use of maxout activation
• Fixed a bug in optimizer when loading data_mean as tuple
• Fixed bug related to missing layer name to top name conversion
• Fixed camera support in grayscale model
• Fixed bug in image_misc resize_without_fit and resize_to_fit