GPU sync failed - Githubissues

When I run on TX2, it report an error

nvidia@tegra-ubuntu:~/Downloads/alpr-unconstrained-master$ bash run.sh -i samples/test/ -o tem/output -c tmp/output/results.csv layer filters size input output 0 conv 32 3 x 3 / 1 416 x 416 x 3 -> 416 x 416 x 32 0.299 BFLOPs 1 max 2 x 2 / 2 416 x 416 x 32 -> 208 x 208 x 32 2 conv 64 3 x 3 / 1 208 x 208 x 32 -> 208 x 208 x 64 1.595 BFLOPs 3 max 2 x 2 / 2 208 x 208 x 64 -> 104 x 104 x 64 4 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BFLOPs 5 conv 64 1 x 1 / 1 104 x 104 x 128 -> 104 x 104 x 64 0.177 BFLOPs 6 conv 128 3 x 3 / 1 104 x 104 x 64 -> 104 x 104 x 128 1.595 BFLOPs 7 max 2 x 2 / 2 104 x 104 x 128 -> 52 x 52 x 128 8 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs 9 conv 128 1 x 1 / 1 52 x 52 x 256 -> 52 x 52 x 128 0.177 BFLOPs 10 conv 256 3 x 3 / 1 52 x 52 x 128 -> 52 x 52 x 256 1.595 BFLOPs 11 max 2 x 2 / 2 52 x 52 x 256 -> 26 x 26 x 256 12 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs 13 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs 14 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs 15 conv 256 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 256 0.177 BFLOPs 16 conv 512 3 x 3 / 1 26 x 26 x 256 -> 26 x 26 x 512 1.595 BFLOPs 17 max 2 x 2 / 2 26 x 26 x 512 -> 13 x 13 x 512 18 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs 19 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs 20 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs 21 conv 512 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 512 0.177 BFLOPs 22 conv 1024 3 x 3 / 1 13 x 13 x 512 -> 13 x 13 x1024 1.595 BFLOPs 23 conv 1024 3 x 3 / 1 13 x 13 x1024 -> 13 x 13 x1024 3.190 BFLOPs 24 conv 1024 3 x 3 / 1 13 x 13 x1024 -> 13 x 13 x1024 3.190 BFLOPs 25 route 16 26 conv 64 1 x 1 / 1 26 x 26 x 512 -> 26 x 26 x 64 0.044 BFLOPs 27 reorg / 2 26 x 26 x 64 -> 13 x 13 x 256 28 route 27 24 29 conv 1024 3 x 3 / 1 13 x 13 x1280 -> 13 x 13 x1024 3.987 BFLOPs 30 conv 125 1 x 1 / 1 13 x 13 x1024 -> 13 x 13 x 125 0.043 BFLOPs 31 detection mask_scale: Using default '1.000000' Loading weights from data/vehicle-detector/yolo-voc.weights...Done! Searching for vehicles using YOLO... Scanning samples/test/03009.jpg 2 cars found Scanning samples/test/03016.jpg 1 cars found Scanning samples/test/03025.jpg 1 cars found Scanning samples/test/03033.jpg 1 cars found Scanning samples/test/03057.jpg 1 cars found Scanning samples/test/03058.jpg 2 cars found Scanning samples/test/03066.jpg 3 cars found Scanning samples/test/03071.jpg 1 cars found Using TensorFlow backend. 2019-03-06 08:28:30.372763: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:865] ARM64 does not support NUMA - returning NUMA node zero 2019-03-06 08:28:30.372933: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties: name: NVIDIA Tegra X2 major: 6 minor: 2 memoryClockRate(GHz): 1.3005 pciBusID: 0000:00:00.0 totalMemory: 7.67GiB freeMemory: 5.29GiB 2019-03-06 08:28:30.372994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0 2019-03-06 08:28:33.551594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-03-06 08:28:33.551754: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0 2019-03-06 08:28:33.551804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N 2019-03-06 08:28:33.552022: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4303 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X2, pci bus id: 0000:00:00.0, compute capability: 6.2) 2019-03-06 08:28:33.782724: E tensorflow/stream_executor/cuda/cuda_driver.cc:1110] could not synchronize on CUDA context: CUDA_ERROR_UNKNOWN :: Begin stack trace perftools::gputools::cuda::CUDADriver::SynchronizeContext(perftools::gputools::cuda::CudaContext) perftools::gputools::StreamExecutor::SynchronizeAllActivity() tensorflow::GPUUtil::SyncAll(tensorflow::Device) End stack trace

Traceback (most recent call last): File "license-plate-detection.py", line 28, in wpod_net = load_model(wpod_net_path) File "/home/nvidia/Downloads/alpr-unconstrained-master/src/keras_utils.py", line 35, in load_model model = model_from_json(model_json, custom_objects=custom_objects) File "/usr/local/lib/python2.7/dist-packages/keras/engine/saving.py", line 369, in model_from_json return deserialize(config, custom_objects=custom_objects) File "/usr/local/lib/python2.7/dist-packages/keras/layers/init.py", line 55, in deserialize printable_module_name='layer') File "/usr/local/lib/python2.7/dist-packages/keras/utils/generic_utils.py", line 145, in deserialize_keras_object list(custom_objects.items()))) File "/usr/local/lib/python2.7/dist-packages/keras/engine/network.py", line 1046, in from_config process_node(layer, node_data) File "/usr/local/lib/python2.7/dist-packages/keras/engine/network.py", line 1003, in process_node layer(input_tensors[0], kwargs) File "/usr/local/lib/python2.7/dist-packages/keras/engine/base_layer.py", line 460, in call output = self.call(inputs, kwargs) File "/usr/local/lib/python2.7/dist-packages/keras/layers/normalization.py", line 183, in call epsilon=self.epsilon) File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 1835, in normalize_batch_in_training if not _has_nchw_support() and list(reduction_axes) == [0, 2, 3]: File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 289, in _has_nchw_support gpus_available = len(_get_available_gpus()) > 0 File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 275, in _get_available_gpus _LOCAL_DEVICES = get_session().list_devices() File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 196, in get_session [tf.is_variable_initialized(v) for v in candidate_vars]) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 900, in run run_metadata_ptr) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1135, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1316, in _do_run run_metadata) File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1335, in _do_call raise type(e)(node_def, op, message) InternalError: GPU sync failed nvidia@tegra-ubuntu:~/Downloads/alpr-unconstrained-master$

how can I do ?

sergiomsilva / alpr-unconstrained

GPU sync failed #52