Tensorflow hangs

[srivatsankrishnan]

Hi,

I used yout script to install tensorflow on Nvidia Jetson TX2. It hangs when i try to run gpu.py. Here is the stack trace: /usr/local/lib/python2.7/dist-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters 2018-06-12 01:22:24.660209: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:865] ARM64 does not support NUMA - returning NUMA node zero 2018-06-12 01:22:24.660419: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties: name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.3005 pciBusID: 0000:00:00.0 totalMemory: 7.67GiB freeMemory: 4.19GiB 2018-06-12 01:22:24.660468: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0

Have you seen this issue before?

Thanks Sri

[peterlee0127]

Do you have these environment setting ?

# Setting in .bashrc or .zshrc or other bash
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH

#sudo apt-get install libcupti-doc
export LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH

Sometime I ran TensorFlow program will hang too. Reboot may help.

[alejandroandreu]

@srivatsankrishnan , I've had this issue before. It doesn't hang and it's not an error, it just takes a long time to start up. For me, it was about 10 minutes. Are you running Tensorflow in a Docker container?

Anyhow, I compiled my own Tensorflow 1.8 wheel file inside my Jetson TX2 without TensorRT support, and with GDR and VERBS support. You can find my wheel file here (for Python 2.7).

[srivatsankrishnan]

@alejandroandreu Yeah it works now. You are right it takes really long to start up and after that point, it runs fine! Thanks, @peterlee0127 @alejandroandreu for your prompt reply! I will close this issue now!

-Sri