So I setup everything, and when I run the training/001_start_train.sh, I get the out of memory error:
Check failed: error == cudaSuccess (2 vs. 0) out of memory
I have 6GB on my GPU, and constantly watching its memory-usage. I even tried setting the batch-size to 1 for both training and testing. None of these helped and I still receive this error.
Since I get passed the network initialization and the initial memory size check, my guess is something is going on in the IMAGE_SEG_DATA layer.
Are you loading all the data into the GPU memory at the very beginning to reduce disk access?
So I setup everything, and when I run the training/001_start_train.sh, I get the out of memory error: Check failed: error == cudaSuccess (2 vs. 0) out of memory
I have 6GB on my GPU, and constantly watching its memory-usage. I even tried setting the batch-size to 1 for both training and testing. None of these helped and I still receive this error.
Since I get passed the network initialization and the initial memory size check, my guess is something is going on in the IMAGE_SEG_DATA layer. Are you loading all the data into the GPU memory at the very beginning to reduce disk access?