[Question] using saliency.py

[michiyosony]

@rizkiarm Thanks for open-sourcing this implementation! It looks very interesting.

I've been trying out the pre-trained model in /evaluation and have successfully used predict.py. When I try to run saliency.py, however, I get this error:

Traceback (most recent call last):
  File "saliency.py", line 9, in <module>
    from vis.visualization import visualize_saliency
ImportError: No module named vis.visualization

I've been looking for package named vis on the internet with no success. Can you clarify what this dependency is and where to find it?

[michiyosony]

Ah! This gave me the hint I needed. pip install keras-vis got me past that error; is that the correct package?

When I run python saliency.py ./models/weights368.h5 ./samples/id2_vcd_swwp2s.mpg

I get farther--now the output looks like this:

Using TensorFlow backend.

Loading data from disk...
Data loaded.

W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
Traceback (most recent call last):
  File "saliency.py", line 65, in <module>
    video, result = predict(sys.argv[1], sys.argv[2])
  File "saliency.py", line 59, in predict
    heatmap = visualize_saliency(lipnet.model, layer_idx, range(0,28), video.data)
  File "/Users/michiyosony/tensorflow/lib/python2.7/site-packages/vis/visualization/saliency.py", line 125, in visualize_saliency
    return visualize_saliency_with_losses(model.input, losses, seed_input, grad_modifier)
  File "/Users/michiyosony/tensorflow/lib/python2.7/site-packages/vis/visualization/saliency.py", line 72, in visualize_saliency_with_losses
    opt = Optimizer(input_tensor, losses, norm_grads=False)
  File "/Users/michiyosony/tensorflow/lib/python2.7/site-packages/vis/optimizer.py", line 58, in __init__
    self.loss_functions + [overall_loss, grads, self.wrt_tensor])
  File "/Users/michiyosony/tensorflow/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 2095, in function
    return Function(inputs, outputs, updates=updates)
  File "/Users/michiyosony/tensorflow/lib/python2.7/site-packages/keras/backend/tensorflow_backend.py", line 2049, in __init__
    with tf.control_dependencies(self.outputs):
  File "/Users/michiyosony/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3583, in control_dependencies
    return get_default_graph().control_dependencies(control_inputs)
  File "/Users/michiyosony/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3314, in control_dependencies
    c = self.as_graph_element(c)
  File "/Users/michiyosony/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2405, in as_graph_element
    return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
  File "/Users/michiyosony/tensorflow/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2494, in _as_graph_element_locked
    % (type(obj).__name__, types_str))
TypeError: Can not convert a list into a Tensor or Operation.

This seems like a separate issue--unless it's a symptom of installing the wrong package.

[rizkiarm]

Hi @michiyosony, saliency visualization is still in progress. You can contribute to it if you want :)

[michiyosony]

@rizkiarm Got it, thanks :) I am new to python and machine learning, but I will not hesitate to contribute if I happen to do something contribution-worthy.

[kawseribn]

@michiyosony can you please tell me how you successfully used predict.py ? I am stuck on this for long time.