search

peterlee0127 / tensorflow-nvJetson

TensorFlow for NVIDIA Jetson, also include patch and script for building.
https://tfjetson.peterlee.app
205 stars 61 forks source link

test_tftrt.py does not seem to work with your 1.9 whl install? #26

Closed mw46d closed 6 years ago

mw46d commented 6 years ago

No, quite sure, yet, what is wrong:-(

ubuntu@tegra-ubuntu:~$ python3 test_tftrt.py 
2018-07-20 04:04:32.686105: F tensorflow/core/framework/op.cc:55] Non-OK-status: RegisterAlreadyLocked(op_data_factory) status: Already exists: Op with name _ScopedAllocator
Aborted (core dumped)

The gpu.py test program seems to be happy after a reboot now:

ubuntu@tegra-ubuntu:~$ python3 gpu.py
2018-07-20 04:08:28.954407: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:864] ARM64 does not support NUMA - returning NUMA node zero
2018-07-20 04:08:28.954549: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties: 
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.3005
pciBusID: 0000:00:00.0
totalMemory: 7.67GiB freeMemory: 5.75GiB
2018-07-20 04:08:28.954645: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2018-07-20 04:08:30.430963: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-20 04:08:30.431039: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958]      0 
2018-07-20 04:08:30.431067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   N 
2018-07-20 04:08:30.431276: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5387 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2
2018-07-20 04:08:30.433324: I tensorflow/core/common_runtime/direct_session.cc:288] Device mapping:
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2

MatMul: (MatMul): /job:localhost/replica:0/task:0/device:GPU:0
2018-07-20 04:08:30.434435: I tensorflow/core/common_runtime/placer.cc:886] MatMul: (MatMul)/job:localhost/replica:0/task:0/device:GPU:0
a: (Const): /job:localhost/replica:0/task:0/device:GPU:0
2018-07-20 04:08:30.434494: I tensorflow/core/common_runtime/placer.cc:886] a: (Const)/job:localhost/replica:0/task:0/device:GPU:0
b: (Const): /job:localhost/replica:0/task:0/device:GPU:0
2018-07-20 04:08:30.434526: I tensorflow/core/common_runtime/placer.cc:886] b: (Const)/job:localhost/replica:0/task:0/device:GPU:0
[[22. 28.]
 [49. 64.]]
mw46d commented 6 years ago

https://github.com/tensorflow/tensorflow/issues/20139 seems to describe the same issue for PC+GPU based setups?!

peterlee0127 commented 6 years ago

I had this issue too. ➜ ~ python3 test_tftrt.py 2018-07-20 13:37:45.340142: F tensorflow/core/framework/op.cc:55] Non-OK-status: RegisterAlreadyLocked(op_data_factory) status: Already exists: Op with name _ScopedAllocator [1] 6507 abort (core dumped) python3 test_tftrt.py

peterlee0127 commented 6 years ago

This issue already fixed in TensorFlow 1.10.0-rc0

2018-07-27 23:43:57.045232: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:864] ARM64 does not support NUMA - returning NUMA node zero
2018-07-27 23:43:57.045418: I tensorflow/core/grappler/devices.cc:51] Number of eligible GPUs (core count >= 8): 0
2018-07-27 23:43:57.055995: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope '', converted to graph
2018-07-27 23:43:57.071515: E tensorflow/contrib/tensorrt/convert/convert_graph.cc:720]  Can't find a GPU device to work with. Please instantiate a session to initialize devices
2018-07-27 23:43:57.071605: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:844] Can't identify the cuda device. Running on device 0
2018-07-27 23:44:02.423167: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1404] Found device 0 with properties:
name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.3005
pciBusID: 0000:00:00.0
totalMemory: 7.67GiB freeMemory: 4.13GiB
2018-07-27 23:44:02.423274: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-07-27 23:44:02.423325: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-27 23:44:02.423355: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0
2018-07-27 23:44:02.423382: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N
2018-07-27 23:44:02.423524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3927 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
2018-07-27 23:44:02.423772: E tensorflow/core/common_runtime/gpu/gpu_device.cc:228] Illegal GPUOptions.experimental.num_dev_to_dev_copy_streams=0 set to 1 instead.
2018-07-27 23:44:03.318290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-07-27 23:44:03.318420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-27 23:44:03.318450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0
2018-07-27 23:44:03.318476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N
2018-07-27 23:44:03.318573: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3927 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
2018-07-27 23:44:03.371300: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-07-27 23:44:03.371441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-27 23:44:03.371471: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0
2018-07-27 23:44:03.371502: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N
2018-07-27 23:44:03.371635: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3927 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
2018-07-27 23:44:03.417122: I tensorflow/core/grappler/devices.cc:51] Number of eligible GPUs (core count >= 8): 0
2018-07-27 23:44:03.422701: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope '', converted to graph
2018-07-27 23:44:03.423407: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:724] Can't determine the device, constructing an allocator at device 0
2018-07-27 23:44:03.586827: I tensorflow/core/grappler/devices.cc:51] Number of eligible GPUs (core count >= 8): 0
2018-07-27 23:44:03.593035: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2853] Segment @scope '', converted to graph
2018-07-27 23:44:03.593860: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:724] Can't determine the device, constructing an allocator at device 0
2018-07-27 23:44:03.676688: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-07-27 23:44:03.676832: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-27 23:44:03.676861: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0
2018-07-27 23:44:03.676887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N
2018-07-27 23:44:03.676987: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3927 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
2018-07-27 23:44:03.732965: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-07-27 23:44:03.733098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-27 23:44:03.733127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0
2018-07-27 23:44:03.733155: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N
2018-07-27 23:44:03.733284: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3927 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
2018-07-27 23:44:03.755073: I tensorflow/contrib/tensorrt/kernels/trt_engine_op.cc:567] Starting calibration thread on device 0, Calibration Resource @ 0x7f10000f70
2018-07-27 23:44:03.755366: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Int8 support requested on hardware without native Int8 support, performance will be negatively affected.
2018-07-27 23:44:03.885254: I tensorflow/contrib/tensorrt/convert/convert_graph.cc:153] Starting Calib Conversion
2018-07-27 23:44:03.885552: W tensorflow/contrib/tensorrt/convert/convert_graph.cc:159] Construction of static int8 engine is not implemented yet!. Dynamic engine will be constructed
2018-07-27 23:44:05.099111: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-07-27 23:44:05.099212: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-27 23:44:05.099244: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0
2018-07-27 23:44:05.099266: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N
2018-07-27 23:44:05.099360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3927 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2)
2018-07-27 23:44:05.112864: I tensorflow/contrib/tensorrt/kernels/trt_engine_op.cc:491] import/my_trt_op_0 Constructing a new engine with batch size 100
2018-07-27 23:44:05.113169: W tensorflow/contrib/tensorrt/log/trt_logger.cc:34] DefaultLogger Int8 support requested on hardware without native Int8 support, performance will be negatively affected.
Pass