tensorflow.python.framework.errors_impl.NotFoundError:

[abidmalikwaterloo]

I am able to run this benchmark using P100 with python 2.7, CUDA9, cudnn 7, and TF 1.6.

I am trying to run the same with K20 using the same installation configuration and gettingthe following error: [amalik@node04 tf_cnn_benchmarks]$ python tf_cnn_benchmarks.py --num_gpus=1 --batch_size=32 --model=resnet50 --variable_update=parameter_server Traceback (most recent call last): File "tf_cnn_benchmarks.py", line 27, in import benchmark_cnn File "/home/amalik/horovod/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py", line 47, in import data_utils File "/home/amalik/horovod/benchmarks/scripts/tf_cnn_benchmarks/data_utils.py", line 21, in from tensorflow.contrib.data.python.ops import batching File "/home/amalik/.local/lib/python2.7/site-packages/tensorflow/contrib/init.py", line 84, in from tensorflow.contrib.eager.python import tfe as eager File "/home/amalik/.local/lib/python2.7/site-packages/tensorflow/contrib/eager/python/tfe.py", line 78, in from tensorflow.contrib.eager.python.datasets import Iterator File "/home/amalik/.local/lib/python2.7/site-packages/tensorflow/contrib/eager/python/datasets.py", line 23, in from tensorflow.contrib.data.python.ops import prefetching_ops File "/home/amalik/.local/lib/python2.7/site-packages/tensorflow/contrib/data/python/ops/prefetching_ops.py", line 25, in resource_loader.get_path_to_datafile("../../_prefetching_ops.so")) File "/home/amalik/.local/lib/python2.7/site-packages/tensorflow/contrib/util/loader.py", line 55, in load_op_library ret = load_library.load_op_library(path) File "/home/amalik/.local/lib/python2.7/site-packages/tensorflow/python/framework/load_library.py", line 58, in load_op_library lib_handle = py_tf.TF_LoadLibrary(library_filename, status) File "/home/amalik/.local/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in exit c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.NotFoundError: /home/amalik/.local/lib/python2.7/site-packages/tensorflow/contrib/data/python/ops/../../_prefetching_ops.so: undefined symbol: _ZN6google8protobuf8internal9ArenaImpl15AllocateAlignedEm

I googled for the solutions and tried most of them but could not get rid of this error.

Thanks,

[abidmalikwaterloo]

Does it have to do with --config=monlithic? I was getting an error during my build for TF from the source code with --config=opt.

[reedwm]

I have compiled with --config=monolothic before and it worked fine.

What happens if you just run the line:

from tensorflow.contrib.data.python.ops import prefetching_ops

Also, try on TensorFlow 1.7.