TomHeaven
commented
4 years ago Try release of v2.2. Here is my test result:
python3 tf_cnn_benchmarks.py --num_gpus=1 --batch_size=64 --model=resnet50 --variable_update=parameter_server --xla_compile=True
2020-05-31 20:59:59.454636: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.10.0.dylib
WARNING: Logging before flag parsing goes to stderr.
W0531 21:00:02.565242 140735744557952 deprecation.py:323] From /usr/local/lib/python3.7/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
2020-05-31 21:00:02.632062: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.dylib
2020-05-31 21:00:02.639004: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:943] OS X does not support NUMA - returning NUMA node zero
2020-05-31 21:00:02.639154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1544] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: TITAN V computeCapability: 7.0
coreClock: 1.455GHz coreCount: 80 deviceMemorySize: 12.00GiB deviceMemoryBandwidth: 607.97GiB/s
2020-05-31 21:00:02.639501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.10.0.dylib
2020-05-31 21:00:02.660028: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.10.0.dylib
2020-05-31 21:00:02.679666: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.10.0.dylib
2020-05-31 21:00:02.680897: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.10.0.dylib
2020-05-31 21:00:02.744991: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.10.0.dylib
2020-05-31 21:00:02.765552: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.10.0.dylib
2020-05-31 21:00:02.787741: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.7.dylib
2020-05-31 21:00:02.787873: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:943] OS X does not support NUMA - returning NUMA node zero
2020-05-31 21:00:02.788137: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:943] OS X does not support NUMA - returning NUMA node zero
2020-05-31 21:00:02.788250: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1686] Adding visible gpu devices: 0
2020-05-31 21:00:03.547819: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1085] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-31 21:00:03.547838: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1091] 0
2020-05-31 21:00:03.547842: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1104] 0: N
2020-05-31 21:00:03.548349: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:943] OS X does not support NUMA - returning NUMA node zero
2020-05-31 21:00:03.548599: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:943] OS X does not support NUMA - returning NUMA node zero
2020-05-31 21:00:03.548862: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:943] OS X does not support NUMA - returning NUMA node zero
2020-05-31 21:00:03.549003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1230] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5444 MB memory) -> physical GPU (device: 0, name: TITAN V, pci bus id: 0000:01:00.0, compute capability: 7.0)
2020-05-31 21:00:03.557051: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fcf6fc35ad0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-31 21:00:03.557072: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): TITAN V, Compute Capability 7.0
2020-05-31 21:00:03.568181: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fcf76a14ea0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-31 21:00:03.568199: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
TensorFlow: 2.2
Model: resnet50
Dataset: imagenet (synthetic)
Mode: training
SingleSess: False
Batch size: 64 global
64 per device
Num batches: 100
Num epochs: 0.00
Devices: ['/gpu:0']
NUMA bind: False
Data format: NCHW
Optimizer: sgd
Variables: parameter_server
==========
Generating training model
W0531 21:00:03.581869 140735744557952 deprecation.py:323] From /Volumes/Data/libraries/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:134: conv2d (from tensorflow.python.keras.legacy_tf_layers.convolutional) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.keras.layers.Conv2D` instead.
W0531 21:00:03.586750 140735744557952 deprecation.py:323] From /usr/local/lib/python3.7/site-packages/tensorflow/python/keras/legacy_tf_layers/convolutional.py:424: Layer.apply (from tensorflow.python.keras.engine.base_layer_v1) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
W0531 21:00:03.621324 140735744557952 deprecation.py:323] From /Volumes/Data/libraries/benchmarks/scripts/tf_cnn_benchmarks/convnet_builder.py:266: max_pooling2d (from tensorflow.python.keras.legacy_tf_layers.pooling) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.MaxPooling2D instead.
Initializing graph
W0531 21:00:06.072226 140735744557952 deprecation.py:323] From /Volumes/Data/libraries/benchmarks/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2268: Supervisor.__init__ (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2020-05-31 21:00:06.395814: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:943] OS X does not support NUMA - returning NUMA node zero
2020-05-31 21:00:06.396008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1544] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: TITAN V computeCapability: 7.0
coreClock: 1.455GHz coreCount: 80 deviceMemorySize: 12.00GiB deviceMemoryBandwidth: 607.97GiB/s
2020-05-31 21:00:06.396435: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.10.0.dylib
2020-05-31 21:00:06.396737: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.10.0.dylib
2020-05-31 21:00:06.397049: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.10.0.dylib
2020-05-31 21:00:06.397340: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.10.0.dylib
2020-05-31 21:00:06.397631: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.10.0.dylib
2020-05-31 21:00:06.397920: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.10.0.dylib
2020-05-31 21:00:06.398186: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.7.dylib
2020-05-31 21:00:06.398295: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:943] OS X does not support NUMA - returning NUMA node zero
2020-05-31 21:00:06.398555: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:943] OS X does not support NUMA - returning NUMA node zero
2020-05-31 21:00:06.398670: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1686] Adding visible gpu devices: 0
2020-05-31 21:00:06.398693: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1085] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-31 21:00:06.398698: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1091] 0
2020-05-31 21:00:06.398702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1104] 0: N
2020-05-31 21:00:06.398846: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:943] OS X does not support NUMA - returning NUMA node zero
2020-05-31 21:00:06.399059: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:943] OS X does not support NUMA - returning NUMA node zero
2020-05-31 21:00:06.399178: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1230] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5444 MB memory) -> physical GPU (device: 0, name: TITAN V, pci bus id: 0000:01:00.0, compute capability: 7.0)
I0531 21:00:07.086311 140735744557952 session_manager.py:505] Running local_init_op.
I0531 21:00:07.158162 140735744557952 session_manager.py:508] Done running local_init_op.
Running warm up
2020-05-31 21:00:08.948844: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.10.0.dylib
2020-05-31 21:00:09.161863: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.7.dylib
2020-05-31 21:00:22.647080: I tensorflow/compiler/jit/xla_compilation_cache.cc:314] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
Done warm up
Step Img/sec total_loss
1 images/sec: 354.3 +/- 0.0 (jitter = 0.0) 7.608
10 images/sec: 350.1 +/- 1.0 (jitter = 4.2) 7.849
20 images/sec: 350.6 +/- 0.6 (jitter = 3.9) 8.013
30 images/sec: 350.5 +/- 0.6 (jitter = 3.8) 7.939
40 images/sec: 347.0 +/- 1.7 (jitter = 5.0) 8.135
50 images/sec: 347.9 +/- 1.4 (jitter = 4.4) 8.051
60 images/sec: 348.7 +/- 1.2 (jitter = 3.1) 7.786
70 images/sec: 349.4 +/- 1.0 (jitter = 1.7) 7.857
80 images/sec: 349.9 +/- 0.9 (jitter = 1.4) 8.007
90 images/sec: 350.3 +/- 0.8 (jitter = 1.2) 7.842
100 images/sec: 349.2 +/- 0.9 (jitter = 1.6) 8.088
----------------------------------------------------------------
total images/sec: 349.07
----------------------------------------------------------------
yrwy
Hi Tom, 又来骚扰你了，我印象中是1.3之后就无法使用XLA了，一旦使用就出现报错，无法找到设备。现在tf2.2默认开启了XLA的编译选项。我跑TF官方的benchmark会报错，因为TF在macOS下的GPU支持已经被放弃了，只能来骚扰你了，看下你有没有碰过这种情况。
具体如下： https://github.com/tensorflow/benchmarks 下载 benchmarks-cnn_tf_v2.1_compatible
python tf_cnn_benchmarks.py --num_gpus=1 --batch_size=64 --model=resnet50 --variable_update=parameter_server --xla_compile=True2020-05-30 15:31:34.302936: W tensorflow/core/framework/op_kernel.cc:1753] OP_REQUIRES failed at xla_ops.cc:368 : Not found: could not find registered platform with id: 0x141f70298 This error might be occurring with the use of xla.compile. If it is not necessary that every Op be compiled with XLA, an alternative is to use auto_jit with OptimizerOptions.global_jit_level = ON_2 or the environment variable TF_XLA_FLAGS="tf_xla_auto_jit=2" which will attempt to use xla to compile as much of the graph as the compiler is able to. INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.NotFoundError'>, 2 root error(s) found. (0) Not found: could not find registered platform with id: 0x141f70298 This error might be occurring with the use of xla.compile. If it is not necessary that every Op be compiled with XLA, an alternative is to use auto_jit with OptimizerOptions.global_jit_level = ON_2 or the environment variable TF_XLA_FLAGS="tf_xla_auto_jit=2" which will attempt to use xla to compile as much of the graph as the compiler is able to. [[{{node tower_0/v/cluster}}]] [[main_fetch_group/_566]] (1) Not found: could not find registered platform with id: 0x141f70298 This error might be occurring with the use of xla.compile. If it is not necessary that every Op be compiled with XLA, an alternative is to use auto_jit with OptimizerOptions.global_jit_level = ON_2 or the environment variable TF_XLA_FLAGS="tf_xla_auto_jit=2" which will attempt to use xla to compile as much of the graph as the compiler is able to. [[{{node tower_0/v/cluster}}]] 0 successful operations. 0 derived errors ignored.去掉 --xla_compile=True 这个参数是可以跑的起来的。