run validate.py has a error: failed to allocate xxx from device: CUDA_ERROR_OUT_OF_MEMORY

[Depblu]

log info: python validate.py --n_proc 2 --exp_dir ../../data/experiments/NGSIM-gail/ --params_filename itr_1000.npz --random_seed 42 I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally 50 vehicles with H = 200

W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations. W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations. I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate (GHz) 1.8095 pciBusID 0000:83:00.0 Total memory: 7.93GiB Free memory: 7.70GiB W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one is currently active; existing: 0x641f8c0 I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0 with properties: name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate (GHz) 1.8095 pciBusID 0000:83:00.0 Total memory: 7.93GiB Free memory: 7.70GiB W tensorflow/stream_executor/cuda/cuda_driver.cc:590] creating context when one is currently active; existing: 0x730cd40 I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1 with properties: name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate (GHz) 1.8095 pciBusID 0000:84:00.0 Total memory: 7.93GiB Free memory: 7.70GiB I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 1 with properties: name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate (GHz) 1.8095 pciBusID 0000:84:00.0 Total memory: 7.93GiB Free memory: 7.70GiB I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 1 I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0 1 I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 1: Y Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0: Y Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 1: Y Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:83:00.0) I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:83:00.0) I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 1080, pci bus id: 0000:84:00.0) I tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 1080, pci bus id: 0000:84:00.0) E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 7.32G (7857173248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 6.58G (7071455744 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 5.93G (6364310016 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 5.33G (5727878656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 4.80G (5155090432 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 4.32G (4639581184 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 3.89G (4175622912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 3.50G (3758060544 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 3.15G (3382254336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 2.83G (3044028928 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 2.55G (2739625984 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 2.30G (2465663232 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 2.07G (2219096832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.86G (1997187072 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.67G (1797468416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.51G (1617721600 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.36G (1455949568 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.22G (1310354688 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1.10G (1179319296 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 1012.22M (1061387520 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 911.00M (955248896 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 819.90M (859724032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 737.91M (773751808 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 664.12M (696376832 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 597.71M (626739200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 537.93M (564065280 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 484.14M (507658752 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 435.73M (456892928 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY E tensorflow/stream_executor/cuda/cuda_driver.cc:1002] failed to allocate 392.15M (411203840 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY pid: 0 traj: 0 / 1031E tensorflow/stream_executor/cuda/cuda_blas.cc:372] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED W tensorflow/stream_executor/stream.cc:1390] attempting to perform BLAS operation using StreamExecutor without BLAS support E tensorflow/stream_executor/cuda/cuda_blas.cc:372] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED W tensorflow/stream_executor/stream.cc:1390] attempting to perform BLAS operation using StreamExecutor without BLAS support Traceback (most recent call last): File "validate.py", line 381, in random_seed=run_args.random_seed File "validate.py", line 266, in collect random_seed=random_seed File "validate.py", line 188, in parallel_collect_trajectories [res.get() for res in results] File "validate.py", line 188, in [res.get() for res in results] File "/opt/conda/envs/rllab3/lib/python3.5/multiprocessing/pool.py", line 644, in get raise self._value multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x7f23f5742710>'. Reason: 'PicklingError("Can't pickle <class 'module'>: attribute lookup module on builtins failed",)'

signal (15): Terminated in expression starting at no file:0 do_futex_wait.constprop.3 at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line) __new_sem_wait_slow.constprop.0 at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line) sem_wait at /lib/x86_64-linux-gnu/libpthread.so.0 (unknown line) semlock_acquire at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/_multiprocessing/semaphore.c:310

signal (15): Terminated in expression starting at no file:0 PyCFunction_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/methodobject.c:98 call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4758 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092 function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 PyObject_CallFunctionObjArgs at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2446 PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3160 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092 function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) _ZN4llvm15ScalarEvolution16isKnownPredicateENS_7CmpInst9PredicateEPKNS4SCEVES5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 _ZN4llvm15ScalarEvolution25isKnownViaSimpleReasoningENS_7CmpInst9PredicateEPKNS4SCEVES5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) _ZN4llvm15ScalarEvolution27isImpliedCondOperandsHelperENS_7CmpInst9PredicateEPKNS_4SCEVES5_S5S5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) _ZN4llvm15ScalarEvolution21isImpliedCondOperandsENS_7CmpInst9PredicateEPKNS_4SCEVES5_S5S5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) slot_tp_init at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:6314 type_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:905 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4989 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4785 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092 function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 _ZN4llvm15ScalarEvolution13isImpliedCondENS_7CmpInst9PredicateEPKNS_4SCEVES5_S2_S5S5 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) _ZN4llvm15ScalarEvolution27isLoopBackedgeGuardedByCondEPKNS_4LoopENS_7CmpInst9PredicateEPKNS4SCEVES8 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) _ZN4llvm15ScalarEvolution17getSignExtendExprEPKNS_4SCEVEPNS_4TypeEj at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) slot_tp_init at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:6314 _ZN4llvm17simplifyUsersOfIVEPNS_7PHINodeEPNS_15ScalarEvolutionEPNS_13DominatorTreeEPNS_8LoopInfoERNS_15SmallVectorImplINS_14WeakTrackingVHEEERNS_12SCEVExpanderEPNS_9IVVisitorE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) type_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:905 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4989 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4785 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092 PyEval_EvalCode at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:777 run_mod at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:991 [inlined] PyRun_FileExFlags at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:944 PyRun_SimpleFileExFlags at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:411 run_file at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/main.c:328 [inlined] Py_Main at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/main.c:779 main at python (unknown line) _ZN12_GLOBALN_114IndVarSimplify17simplifyAndExtendEPN4llvm4LoopERNS1_12SCEVExpanderEPNS1_8LoopInfoE.constprop.426 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) _ZN12_GLOBAL__N_114IndVarSimplify3runEPN4llvm4LoopE.part.412 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) unknown function (ip: 0x400998) unknown function (ip: 0xffffffffffffffff) Allocations: 112352957 (Pool: 112338583; Big: 14374); GC: 238

signal (11): Segmentation fault in expression starting at no file:0 PyErr_Fetch at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/errors.c:328 PyObject_ClearWeakRefs at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/weakrefobject.c:909 _ZN12_GLOBAL__N_124IndVarSimplifyLegacyPass9runOnLoopEPN4llvm4LoopERNS1_13LPPassManagerE.part.413 at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) _ZN4llvm13LPPassManager13runOnFunctionERNS_8FunctionE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) _ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) _ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) _ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE at /root/code/julia-1.1.0/bin/../lib/julia/libLLVM-6.0.so (unknown line) array_dealloc at /home/ilan/minonda/conda-bld/numpy-1.121486268766587/work/numpy-1.12.0/numpy/core/src/multiarray/arrayobject.c:384 pydecref at /root/.julia/packages/PyCall/ttONZ/src/PyCall.jl:121 pydecref at /root/.julia/packages/PyCall/ttONZ/src/PyCall.jl:126 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219 jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1571 [inlined] run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:232 jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:318 jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:346 jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:249 jl_exit at /buildworker/worker/package_linux64/build/src/jl_uv.c:592 jl_exit_thread0_cb at /buildworker/worker/package_linux64/build/src/signals-unix.c:344 unknown function (ip: 0xffffffffffffffff) Allocations: 112353419 (Pool: 112339045; Big: 14374); GC: 238 operator() at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:335 E tensorflow/stream_executor/cuda/cuda_blas.cc:372] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED W tensorflow/stream_executor/stream.cc:1390] attempting to perform BLAS operation using StreamExecutor without BLAS support addModule at /buildworker/worker/package_linux64/build/usr/include/llvm/ExecutionEngine/Orc/IRCompileLayer.h:57 [inlined] addModule at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:454 jl_add_to_ee at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:679 [inlined] jl_finalize_function at /buildworker/worker/package_linux64/build/src/jitlayers.cpp:687 E tensorflow/stream_executor/cuda/cuda_blas.cc:372] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED W tensorflow/stream_executor/stream.cc:1390] attempting to perform BLAS operation using StreamExecutor without BLAS support getAddressForFunction at /buildworker/worker/package_linux64/build/src/codegen.cpp:1281 jl_generate_fptr at /buildworker/worker/package_linux64/build/src/codegen.cpp:1388 jl_compile_method_internal at /buildworker/worker/package_linux64/build/src/gf.c:1856 [inlined] jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1863 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219 deepcopy_internal at ./deepcopy.jl:78 [inlined] deepcopy at ./deepcopy.jl:28 [inlined] get_features at /home/lius/tmp/github/ngsim_env/julia/src/ngsim_env.jl:218

reset#22 at /home/lius/tmp/github/ngsim_env/julia/src/ngsim_env.jl:125

unknown function (ip: 0x7f23cbe7e1a4) jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1864 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219

reset at ./none:0 [inlined]

reset#2 at /home/lius/tmp/github/ngsim_env/julia/src/env.jl:19 [inlined]

reset at ./none:0

unknown function (ip: 0x7f23cbe7dcb9) jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1864 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219 f_kw_closure at /root/.julia/packages/PyCall/ttONZ/src/callback.jl:36 jl_fptr_trampoline at /buildworker/worker/package_linux64/build/src/gf.c:1864 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219 jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1571 [inlined] jl_fapply at /buildworker/worker/package_linux64/build/src/builtins.c:556 jl_fapply_latest at /buildworker/worker/package_linux64/build/src/builtins.c:594 _pyjlwrap_call at /root/.julia/packages/PyCall/ttONZ/src/callback.jl:37 unknown function (ip: 0x7f23cbe1c443) jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219 pyjlwrap_call at /root/.julia/packages/PyCall/ttONZ/src/callback.jl:49 unknown function (ip: 0x7f23cbe1bd76) jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219 unknown function (ip: 0x7f23cbddbafc) unknown function (ip: 0x7f23cbddb9ce) PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092 function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092 function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092 function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092 function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092 function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 ext_do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:5087 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3328 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092 function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 slot_tp_init at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:6314 type_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:905 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4989 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4785 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4856 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092 function_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/funcobject.c:627 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 method_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/classobject.c:330 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 slot_tp_init at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:6314 type_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/typeobject.c:905 PyObject_Call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/abstract.c:2166 do_call at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4989 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4785 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 fast_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4866 [inlined] call_function at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4783 [inlined] PyEval_EvalFrameEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:3289 _PyEval_EvalCodeWithName at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4071 PyEval_EvalCodeEx at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:4092 PyEval_EvalCode at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/ceval.c:777 run_mod at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:991 [inlined] PyRun_FileExFlags at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:944 PyRun_SimpleFileExFlags at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/pythonrun.c:411 run_file at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/main.c:328 [inlined] Py_Main at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Modules/main.c:779 main at python (unknown line) __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line) unknown function (ip: 0x400998) unknown function (ip: 0xffffffffffffffff) Allocations: 112471597 (Pool: 112457178; Big: 14419); GC: 238

signal (11): Segmentation fault in expression starting at no file:0 PyErr_Fetch at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Python/errors.c:328 PyObject_ClearWeakRefs at /home/ilan/minonda/conda-bld/python_1502735045308/work/Python-3.5.4/Objects/weakrefobject.c:909 array_dealloc at /home/ilan/minonda/conda-bld/numpy-1.121486268766587/work/numpy-1.12.0/numpy/core/src/multiarray/arrayobject.c:384 pydecref at /root/.julia/packages/PyCall/ttONZ/src/PyCall.jl:121 pydecref at /root/.julia/packages/PyCall/ttONZ/src/PyCall.jl:126 jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2219 jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1571 [inlined] run_finalizer at /buildworker/worker/package_linux64/build/src/gc.c:232 jl_gc_run_finalizers_in_list at /buildworker/worker/package_linux64/build/src/gc.c:318 jl_gc_run_all_finalizers at /buildworker/worker/package_linux64/build/src/gc.c:346 jl_atexit_hook at /buildworker/worker/package_linux64/build/src/init.c:249 jl_exit at /buildworker/worker/package_linux64/build/src/jl_uv.c:592 jl_exit_thread0_cb at /buildworker/worker/package_linux64/build/src/signals-unix.c:344 unknown function (ip: 0xffffffffffffffff) Allocations: 112472059 (Pool: 112457640; Big: 14419); GC: 238

[mengyuest]

I encountered a similar issue (CUDA OUT OF MEMORY). One simple solution is just to uninstalled the tensorflow-gpu and reinstalled the tensorflow (so they will not use CUDA). There should be a more elegant way to handle this.