Closed minhnhatvt closed 2 years ago
just my $0.02, regarding crash during very long job i find that tuning a bit CUDA does help
plus it makes it simple to monitor GPU memory since allocation is on request instead of preallocate all
(cuda allocated memory still never gets deallocated until cuda exits to avoid memory fragmentation)
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2' # reduce tf logging
os.environ['CUDA_CACHE_DISABLE'] = '0'
os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true'
os.environ['TF_GPU_THREAD_MODE'] = 'gpu_private'
os.environ['TF_USE_CUDNN_BATCHNORM_SPATIAL_PERSISTENT'] = '1'
os.environ['TF_NEED_HDFS'] = '0'
os.environ['TF_NEED_GCP'] = '0'
os.environ['TF_CUDA_COMPUTE_CAPABILITIES'] = '8.0,8.6'
os.environ['TF_ENABLE_XLA 1'] = '1'
Hi, thank you for the excellent work. I would like to ask some questions:
demo_video_batched.py
(metrabs_eff2l_y4, batch size=16, and the image size is 1920x1080) but the program crashed after predicting roughly 2000 frames. Maybe there exists some memory leak inside the detector? (My specs is Nvidia 2080Ti 11GB with 48GB of RAM)