chaowang15
commented
3 years ago prepare datas start MTCNN MTCNN detect WARNING:tensorflow:From /home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py:349: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.
W1224 16:22:07.315534 140424515024704 module_wrapper.py:139] From /home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py:349: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.
hello WARNING:tensorflow:From /home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py:356: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
W1224 16:22:07.335879 140424515024704 module_wrapper.py:139] From /home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py:356: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2020-12-24 16:22:07.336520: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2020-12-24 16:22:07.369774: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.370135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: GeForce GTX 1660 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.59 pciBusID: 0000:01:00.0 2020-12-24 16:22:07.370294: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-12-24 16:22:07.371003: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2020-12-24 16:22:07.380746: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2020-12-24 16:22:07.382767: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2020-12-24 16:22:07.383489: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2020-12-24 16:22:07.392925: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2020-12-24 16:22:07.394720: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-12-24 16:22:07.394814: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.395156: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.395458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2020-12-24 16:22:07.395700: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-12-24 16:22:07.400321: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599990000 Hz 2020-12-24 16:22:07.401293: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55777bf37240 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-12-24 16:22:07.401307: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2020-12-24 16:22:07.462119: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.462506: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55777bfc9c10 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2020-12-24 16:22:07.462522: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1660 Ti, Compute Capability 7.5 2020-12-24 16:22:07.462639: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.462913: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: GeForce GTX 1660 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.59 pciBusID: 0000:01:00.0 2020-12-24 16:22:07.462943: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-12-24 16:22:07.462954: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2020-12-24 16:22:07.462964: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2020-12-24 16:22:07.462974: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2020-12-24 16:22:07.462984: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2020-12-24 16:22:07.462994: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2020-12-24 16:22:07.463004: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-12-24 16:22:07.463039: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.463320: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.463576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2020-12-24 16:22:07.463598: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-12-24 16:22:07.464194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-12-24 16:22:07.464205: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2020-12-24 16:22:07.464210: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2020-12-24 16:22:07.464273: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.464565: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.464843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5071 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py:358: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.
W1224 16:22:07.466073 140424515024704 module_wrapper.py:139] From /home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py:358: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.
========================= 2020-12-24 16:22:07.789434: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-12-24 16:22:08.154596: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2020-12-24 16:22:08.165649: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR Traceback (most recent call last): File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found. (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node pnet/conv1/Conv2D}}]] (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node pnet/conv1/Conv2D}}]] [[pnet/prob1/_5]] 0 successful operations. 0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "run_data_preparation.py", line 342, in app.run(main) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/absl/app.py", line 303, in run _run_main(main, args) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main sys.exit(main(argv)) File "run_data_preparation.py", line 321, in main prepare_test_data_RGB(FLAGS.img_dir, FLAGS.out_dir) File "run_data_preparation.py", line 204, in prepare_test_data_RGB names_list = detect_face_with_mtcnn.detect_with_MTCNN(img_dir, mtcnn_dir, pb_path) File "/home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py", line 388, in detect_with_MTCNN img, minsize, pnet, rnet, onet, threshold, factor File "/home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py", line 89, in detect_face out = pnet(img_y) File "/home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py", line 45, in ("pnet/conv4-2/BiasAdd:0", "pnet/prob1:0"), feed_dict={"pnet/input:0": img} File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run run_metadata_ptr) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run run_metadata) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found. (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node pnet/conv1/Conv2D (defined at /home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]] (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[node pnet/conv1/Conv2D (defined at /home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]] [[pnet/prob1/_5]] 0 successful operations. 0 derived errors ignored.
Original stack trace for 'pnet/conv1/Conv2D': File "run_data_preparation.py", line 342, in app.run(main) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/absl/app.py", line 303, in run _run_main(main, args) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main sys.exit(main(argv)) File "run_data_preparation.py", line 321, in main prepare_test_data_RGB(FLAGS.img_dir, FLAGS.out_dir) File "run_data_preparation.py", line 204, in prepare_test_data_RGB names_list = detect_face_with_mtcnn.detect_with_MTCNN(img_dir, mtcnn_dir, pb_path) File "/home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py", line 354, in detect_with_MTCNN tf.import_graph_def(graph_def, name="") File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func return func(*args, kwargs) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/importer.py", line 405, in import_graph_def producer_op_list=producer_op_list) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/importer.py", line 517, in _import_graph_def_internal _ProcessNewOps(graph) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/importer.py", line 243, in _ProcessNewOps for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3561, in _add_new_tf_operations for c_op in c_api_util.new_tf_operations(self) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3561, in for c_op in c_api_util.new_tf_operations(self) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3451, in _create_op_from_tf_operation ret = Operation(c_op, self) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init** self._traceback = tf_stack.extract_stack()
data prepare failed
I got exactly the same error on my Linux system. Environment is also the same: tensorflow -gpu1.15 cuda 10.0 cudnn 7.5. Is there any solution now?
lith0613
haoxianzGit
YiChenCityU
ZhenyanSun
prepare datas start MTCNN MTCNN detect WARNING:tensorflow:From /home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py:349: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.
W1224 16:22:07.315534 140424515024704 module_wrapper.py:139] From /home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py:349: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.
hello WARNING:tensorflow:From /home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py:356: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
W1224 16:22:07.335879 140424515024704 module_wrapper.py:139] From /home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py:356: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2020-12-24 16:22:07.336520: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1 2020-12-24 16:22:07.369774: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.370135: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: GeForce GTX 1660 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.59 pciBusID: 0000:01:00.0 2020-12-24 16:22:07.370294: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-12-24 16:22:07.371003: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2020-12-24 16:22:07.380746: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2020-12-24 16:22:07.382767: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2020-12-24 16:22:07.383489: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2020-12-24 16:22:07.392925: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2020-12-24 16:22:07.394720: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-12-24 16:22:07.394814: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.395156: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.395458: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2020-12-24 16:22:07.395700: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-12-24 16:22:07.400321: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599990000 Hz 2020-12-24 16:22:07.401293: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55777bf37240 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-12-24 16:22:07.401307: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2020-12-24 16:22:07.462119: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.462506: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55777bfc9c10 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2020-12-24 16:22:07.462522: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce GTX 1660 Ti, Compute Capability 7.5 2020-12-24 16:22:07.462639: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.462913: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties: name: GeForce GTX 1660 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.59 pciBusID: 0000:01:00.0 2020-12-24 16:22:07.462943: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-12-24 16:22:07.462954: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0 2020-12-24 16:22:07.462964: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0 2020-12-24 16:22:07.462974: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0 2020-12-24 16:22:07.462984: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0 2020-12-24 16:22:07.462994: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0 2020-12-24 16:22:07.463004: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-12-24 16:22:07.463039: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.463320: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.463576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0 2020-12-24 16:22:07.463598: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0 2020-12-24 16:22:07.464194: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-12-24 16:22:07.464205: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0 2020-12-24 16:22:07.464210: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N 2020-12-24 16:22:07.464273: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.464565: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-12-24 16:22:07.464843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5071 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5) WARNING:tensorflow:From /home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py:358: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.
W1224 16:22:07.466073 140424515024704 module_wrapper.py:139] From /home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py:358: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.
========================= 2020-12-24 16:22:07.789434: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7 2020-12-24 16:22:08.154596: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR 2020-12-24 16:22:08.165649: E tensorflow/stream_executor/cuda/cuda_dnn.cc:329] Could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR Traceback (most recent call last): File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found. (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node pnet/conv1/Conv2D}}]] (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above. [[{{node pnet/conv1/Conv2D}}]] [[pnet/prob1/_5]] 0 successful operations. 0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "run_data_preparation.py", line 342, in
app.run(main)
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "run_data_preparation.py", line 321, in main
prepare_test_data_RGB(FLAGS.img_dir, FLAGS.out_dir)
File "run_data_preparation.py", line 204, in prepare_test_data_RGB
names_list = detect_face_with_mtcnn.detect_with_MTCNN(img_dir, mtcnn_dir, pb_path)
File "/home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py", line 388, in detect_with_MTCNN
img, minsize, pnet, rnet, onet, threshold, factor
File "/home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py", line 89, in detect_face
out = pnet(img_y)
File "/home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py", line 45, in
("pnet/conv4-2/BiasAdd:0", "pnet/prob1:0"), feed_dict={"pnet/input:0": img}
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
(0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node pnet/conv1/Conv2D (defined at /home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
(1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
[[node pnet/conv1/Conv2D (defined at /home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[pnet/prob1/_5]]
0 successful operations.
0 derived errors ignored.
Original stack trace for 'pnet/conv1/Conv2D': File "run_data_preparation.py", line 342, in
app.run(main)
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "run_data_preparation.py", line 321, in main
prepare_test_data_RGB(FLAGS.img_dir, FLAGS.out_dir)
File "run_data_preparation.py", line 204, in prepare_test_data_RGB
names_list = detect_face_with_mtcnn.detect_with_MTCNN(img_dir, mtcnn_dir, pb_path)
File "/home/cc/code/hifi3dface/data_prepare/detect_face_with_mtcnn.py", line 354, in detect_with_MTCNN
tf.import_graph_def(graph_def, name="")
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/importer.py", line 405, in import_graph_def
producer_op_list=producer_op_list)
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/importer.py", line 517, in _import_graph_def_internal
_ProcessNewOps(graph)
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/importer.py", line 243, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3561, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3561, in
for c_op in c_api_util.new_tf_operations(self)
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3451, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File "/home/cc/anaconda3/envs/tf1.15/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()
data prepare failed