Closed sebastian-lapuschkin closed 6 years ago
albermax
commented
6 years ago Did you also update TF in the same instance? I go the same problems whenever I used TF for 3.5 and 2.7 with the same codebase. I had to reinstall TF each time before switching. I fear there is some issue with cached code.
Would please try to update and reinstall TF?
sebastian-lapuschkin
commented
6 years ago Reinstalling TF (via pip) did not help. I still get the following when running either example script:
python3 all_methods.py
Using TensorFlow backend.
2018-06-25 19:38:43.946589: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-06-25 19:38:44.149400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: TITAN Xp major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:04:00.0
totalMemory: 11.91GiB freeMemory: 11.74GiB
2018-06-25 19:38:44.241733: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 1 with properties:
name: Quadro M2000 major: 5 minor: 2 memoryClockRate(GHz): 1.1625
pciBusID: 0000:03:00.0
totalMemory: 3.93GiB freeMemory: 2.70GiB
2018-06-25 19:38:44.241777: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1420] Ignoring visible gpu device (device: 1, name: Quadro M2000, pci bus id: 0000:03:00.0, compute capability: 5.2) with Cuda multiprocessor count: 6. The minimum required count is 8. You can adjust this requirement with the env var TF_MIN_GPU_MULTIPROCESSOR_COUNT.
2018-06-25 19:38:44.241785: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-06-25 19:38:44.474333: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-06-25 19:38:44.474367: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0 1
2018-06-25 19:38:44.474374: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N N
2018-06-25 19:38:44.474378: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 1: N N
2018-06-25 19:38:44.474650: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11370 MB memory) -> physical GPU (device: 0, name: TITAN Xp, pci bus id: 0000:04:00.0, compute capability: 6.1)
A local file was found, but it seems to be incomplete or outdated because the sha256 file hash does not match the original value of 847a05e1cb2d689d56170cc1998a2d042d5624c6e58f19ac916b5d70555c41ad0 so we will re-download the data.
Downloading data from https://www.dropbox.com/s/15lip81fzvbgkaa/vgg16_pattern_type_relu_tf_dim_ordering_tf_kernels.npz?dl=1
553385984/553380098 [==============================] - 18s 0us/step
Image 0: Input Traceback (most recent call last):
File "all_methods.py", line 139, in <module>
a = analyzer.analyze(image if is_input_analyzer else x)
File "/home/lapuschkin/Desktop/innvestigate-lrp-wip/innvestigate/innvestigate/analyzer/base.py", line 400, in analyze
ret = self._analyzer_model.predict_on_batch(X)
File "/home/lapuschkin/.local/lib/python3.5/site-packages/keras/engine/training.py", line 1281, in predict_on_batch
outputs = self.predict_function(ins)
File "/home/lapuschkin/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2661, in __call__
return self._call(inputs)
File "/home/lapuschkin/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2630, in _call
session)
File "/home/lapuschkin/.local/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 2582, in _make_callable
callable_fn = session._make_callable_from_options(callable_opts)
File "/home/lapuschkin/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1480, in _make_callable_from_options
return BaseSession._Callable(self, callable_options)
File "/home/lapuschkin/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1441, in __init__
session._session, options_ptr, status)
File "/home/lapuschkin/.local/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 519, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: input_1:0 is both fed and fetched.
Exception ignored in: <bound method BaseSession._Callable.__del__ of <tensorflow.python.client.session.BaseSession._Callable object at 0x7f4fe24b5f28>>
Traceback (most recent call last):
File "/home/lapuschkin/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1464, in __del__
self._session._session, self._handle, status)
File "/home/lapuschkin/.local/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 519, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: No such callable handle: 436957728I forgot to push the fix for this. Sorry about that. Now it should work.
stuff is broken!
immediately after installing keras 2.2.0 for working on develop, I noticed the pretrained model in the mnist notebook performed with only 10% accuracy instead of the expected 99.17%
Next, the code crashed. PFA the corresponding notebook, including the stack trace mnist_example.ipynb.zip
running python3 setup.py test reported every single test a failue, with the logs extending the line buffer of my terminal. An attempt to reproduce to upload a log here resulted in a not further specified segfault immediately.