search

Shirhe-Lyh / Xception

A PyTorch implementation of Xception
18 stars 4 forks source link

Not found: No algorithm worked![[node xception_71/entry_flow/conv1_1/Conv2D #2

Open xhding1997 opened 3 years ago

xhding1997 commented 3 years ago

I'm tring to transform xception_71 from tf to torch. And these bugs occur. I'm using tf==2.4.1, and I changed these in xception_test.py. import tensorflow as tf slim = tf.contrib.slim to import tensorflow.compat.v1 as tf tf.disable_v2_behavior() import tf_slim as slim

bie@bie:~/Xception$ python3 xception_test.py 2021-03-25 16:16:55.958131: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 WARNING:tensorflow:From /home/bie/.local/lib/python3.8/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term /home/bie/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer_v1.py:1719: UserWarning: layer.apply is deprecated and will be removed in a future version. Please use layer.__call__ method instead. warnings.warn('layer.apply is deprecated and ' 2021-03-25 16:16:58.729343: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-03-25 16:16:58.729832: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2021-03-25 16:16:58.764587: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-03-25 16:16:58.764909: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:06:00.0 name: GeForce GTX 1650 SUPER computeCapability: 7.5 coreClock: 1.725GHz coreCount: 20 deviceMemorySize: 3.81GiB deviceMemoryBandwidth: 178.84GiB/s 2021-03-25 16:16:58.764935: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2021-03-25 16:16:58.766278: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2021-03-25 16:16:58.766321: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11 2021-03-25 16:16:58.766904: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2021-03-25 16:16:58.767038: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2021-03-25 16:16:58.768410: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2021-03-25 16:16:58.768753: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2021-03-25 16:16:58.768835: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2021-03-25 16:16:58.768920: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-03-25 16:16:58.769256: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-03-25 16:16:58.769522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0 2021-03-25 16:16:58.769738: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2021-03-25 16:16:58.770041: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set 2021-03-25 16:16:58.770121: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-03-25 16:16:58.770402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: pciBusID: 0000:06:00.0 name: GeForce GTX 1650 SUPER computeCapability: 7.5 coreClock: 1.725GHz coreCount: 20 deviceMemorySize: 3.81GiB deviceMemoryBandwidth: 178.84GiB/s 2021-03-25 16:16:58.770433: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2021-03-25 16:16:58.770456: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2021-03-25 16:16:58.770478: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11 2021-03-25 16:16:58.770499: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2021-03-25 16:16:58.770521: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2021-03-25 16:16:58.770542: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10 2021-03-25 16:16:58.770563: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2021-03-25 16:16:58.770584: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2021-03-25 16:16:58.770634: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-03-25 16:16:58.770938: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-03-25 16:16:58.771198: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0 2021-03-25 16:16:58.771230: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2021-03-25 16:17:00.686896: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix: 2021-03-25 16:17:00.686932: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0 2021-03-25 16:17:00.686940: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N 2021-03-25 16:17:00.687136: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-03-25 16:17:00.687431: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-03-25 16:17:00.687689: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2021-03-25 16:17:00.687921: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2069 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1650 SUPER, pci bus id: 0000:06:00.0, compute capability: 7.5) 2021-03-25 16:17:00.725655: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes) 2021-03-25 16:17:00.781638: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3593305000 Hz 2021-03-25 16:17:02.248784: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2021-03-25 16:17:03.560464: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2021-03-25 16:17:03.661930: E tensorflow/stream_executor/cuda/cuda_blas.cc:226] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED 2021-03-25 16:17:03.662060: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at conv_ops.cc:1106 : Not found: No algorithm worked! Traceback (most recent call last): File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1375, in _do_call return fn(*args) File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1359, in _run_fn return self._call_tf_sessionrun(options, feed_dict, fetch_list, File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1451, in _call_tf_sessionrun return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict, tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found. (0) Not found: No algorithm worked! [[{{node xception_71/entry_flow/conv1_1/Conv2D}}]] [[ArgMax/_1467]] (1) Not found: No algorithm worked! [[{{node xception_71/entry_flow/conv1_1/Conv2D}}]] 0 successful operations. 0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "xception_test.py", line 108, in logits, labels = sess.run([predictions, classes], File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 967, in run result = self._run(None, fetches, feed_dict, options_ptr, File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1190, in _run results = self._do_run(handle, final_targets, final_fetches, File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1368, in _do_run return self._do_call(_run_fn, feeds, fetches, targets, options, File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/client/session.py", line 1394, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found. (0) Not found: No algorithm worked! [[node xception_71/entry_flow/conv1_1/Conv2D (defined at /home/bie/.local/lib/python3.8/site-packages/tf_slim/layers/layers.py:1089) ]] [[ArgMax/_1467]] (1) Not found: No algorithm worked! [[node xception_71/entry_flow/conv1_1/Conv2D (defined at /home/bie/.local/lib/python3.8/site-packages/tf_slim/layers/layers.py:1089) ]] 0 successful operations. 0 derived errors ignored.

Errors may have originated from an input operation. Input Source operations connected to node xception_71/entry_flow/conv1_1/Conv2D: xception_71/entry_flow/conv1_1/weights/read (defined at /home/bie/.local/lib/python3.8/site-packages/tf_slim/ops/variables.py:244) xception_71/Pad (defined at /home/bie/.local/lib/python3.8/site-packages/tf_slim/nets/resnet_utils.py:141)

Input Source operations connected to node xception_71/entry_flow/conv1_1/Conv2D: xception_71/entry_flow/conv1_1/weights/read (defined at /home/bie/.local/lib/python3.8/site-packages/tf_slim/ops/variables.py:244) xception_71/Pad (defined at /home/bie/.local/lib/python3.8/site-packages/tf_slim/nets/resnet_utils.py:141)

Original stack trace for 'xception_71/entry_flow/conv1_1/Conv2D': File "xception_test.py", line 90, in net, end_points = xception_tf.xception_71(inputs, num_classes=1001, File "/home/bie/Xception/original_tf/xception.py", line 733, in xception_71 return xception(inputs, File "/home/bie/Xception/original_tf/xception.py", line 463, in xception net = resnet_utils.conv2d_same(net, 32, 3, stride=2, File "/home/bie/.local/lib/python3.8/site-packages/tf_slim/nets/resnet_utils.py", line 143, in conv2d_same return layers_lib.conv2d( File "/home/bie/.local/lib/python3.8/site-packages/tf_slim/ops/arg_scope.py", line 184, in func_with_args return func(*args, current_args) File "/home/bie/.local/lib/python3.8/site-packages/tf_slim/layers/layers.py", line 1171, in convolution2d return convolution( File "/home/bie/.local/lib/python3.8/site-packages/tf_slim/ops/arg_scope.py", line 184, in func_with_args return func(*args, *current_args) File "/home/bie/.local/lib/python3.8/site-packages/tf_slim/layers/layers.py", line 1089, in convolution outputs = layer.apply(inputs) File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 1722, in apply return self.call(inputs, args, kwargs) File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/keras/legacy_tf_layers/base.py", line 551, in call outputs = super(Layer, self).call(inputs, *args, kwargs) File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer_v1.py", line 786, in call outputs = call_fn(cast_inputs, *args, *kwargs) File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 667, in wrapper return converted_call(f, args, kwargs, options=options) File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 396, in converted_call return _call_unconverted(f, args, kwargs, options) File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py", line 478, in _call_unconverted return f(args, kwargs) File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/keras/layers/convolutional.py", line 248, in call outputs = self._convolution_op(inputs, self.kernel) File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/util/dispatch.py", line 201, in wrapper return target(*args, **kwargs) File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/ops/nn_ops.py", line 1013, in convolution_v2 return convolution_internal( File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/ops/nn_ops.py", line 1143, in convolution_internal return op( File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/ops/nn_ops.py", line 2597, in _conv2d_expanded_batch return gen_nn_ops.conv2d( File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/ops/gen_nnops.py", line 969, in conv2d , _, _op, _outputs = _op_def_library._apply_op_helper( File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/framework/op_def_library.py", line 748, in _apply_op_helper op = g._create_op_internal(op_type_name, inputs, dtypes=None, File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 3528, in _create_op_internal ret = Operation( File "/home/bie/.local/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 1990, in init self._traceback = tf_stack.extract_stack()

xhding1997 commented 3 years ago

I slove it by adding these to xception_test.py.

from tensorflow.compat.v1 import ConfigProto
from tensorflow.compat.v1 import InteractiveSession
config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)