About GPU memory leak

[xurong1981]

Does anyone encounter the following error on "CUDA_ERROR_ILLEGAL_ADDRESS" ? I have changed multiprocessing to single process, but the same problem happened.

My GPU is GeForce RTX 2080 8GB (driver: 440.33.01), and tensorflow: 1.12.0 cuda: 9.0 cudnn: 7.5.0

Training command is like this way: CUDA_VISIBLE_DEVICES=0 python main.py \ --model_name=model_roerich \ --batch_size=1 \ --phase=train \ --image_size=768 \ --lr=0.0002 \ --dsr=0.8 \ --ptcd=./data/Places2/data_large \ --ptad=./data/artist/nicholas-roerich

Finally, the error message are: tensorflow::CurrentStackTrace() stream_executor::cuda::CUDADriver::SynchronizeContext(stream_executor::cuda::CudaContext) stream_executor::StreamExecutor::SynchronizeAllActivity() tensorflow::GPUUtil::SyncAll(tensorflow::Device) tensorflow::BaseGPUDevice::Sync()

Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop(int)
std::_Function_handler<void (), tensorflow::thread::EigenEnvironment::CreateThread(std::function<void ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&)

clone

End stack trace

2020-01-26 00:10:03.045956: E tensorflow/stream_executor/event.cc:34] error destroying CUDA event in context 0x5402c10: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered Traceback (most recent call last): File "/home/username/.local/share/virtualenvs/adaptive-style-transfer-PbxNnQ9W/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call return fn(*args) File "/home/username/.local/share/virtualenvs/adaptive-style-transfer-PbxNnQ9W/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/username/.local/share/virtualenvs/adaptive-style-transfer-PbxNnQ9W/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InternalError: GPU sync failed