BobMcDear
commented
2 years ago Hello Yujiang,
Thank you for the kind words, I am glad to hear you find this project helpful.
Yes, you are right, the code maintains the intermediate values of the neural network, however, that is not the problem. The segmentation fault stems from not freeing inputs & outputs of the layers after every epoch, as you mention, which may be fixed via running the following at the end of each epoch.
for (int i=0; i<seq.layers.size(); i++){
Module *layer = seq.layers[i];
cudaFree(layer->out);
}I was aware of this issue, but this program's purpose was teaching the basics of CUDA, thus no emphasis was placed on time or memory handling.
This is also how PyTorch functions: The outputs of the modules are kept for performing backpropagation, but they are not retained for the entire duration of training and are replaced at every pass.
You are correct in your understanding of cudaMallocManaged.
Does that answer your questions?
EricWu23
I feel that at least there are two more place I free weird.
Here, I think we should pass Sequential_GPU Object by reference, otherwise, it will create a new copy of the seq that was created in the main() of main.cu.
Hi Borna @BobMcDear , Thanks for the amazing project and detailed documentation. I tried to run your project on my computer. While all the unit tests went well, when I try to run the main.cu in GPU folder, I did get the segmentation fault. I reduced the network to only linear layer and relu. It passed without fault. However, the training on GPU is much slower than the CPU version.
Here is my suspicion, the most confusing part of the code to me is the memory management in the sequential_forward_gpu() function. there, after getting the output size of each layer by "sz_out = layer->sz_out;" , we use cudaMallocManaged(&curr_out, sz_out*sizeof(float)); to allocate memory, however, I didn't see anywhere this memory being allocated inside for loop got released by cudaFree().
My understanding is that you have to let those memory allocated stay there because you need to store all outputs at each layer during forward propagation, which will be later on used in backward propagation to compute gradients. However, this does raise my concern on the potential memory leakage. As we have more layers, it will just allocate more memory and it allocate memory every time we call layer->update() inside sequetial_update_gpu(). What makes it worse is that, when we are looping through 100 epoch, for every new iteration, it will allocate new memory. Question 1: Am I misunderstanding something? this is just my suspicion. Let me know if you had a different design idea. Question 2: Also, do you know in general how pytorch or something similar manage the temporary memory needed to backward propagation. It seems to me now that this could be a nightmare for deep NN. Question 3: I kind of understand cudaMallocManaged gives you ability to allocate a chunk of memory that can be accessed from host and device using the same pointer. And perhaps the memory copy on/off device is handled by itself. Is my understanding correct? Thanks.
Some additional notes after debugging, if I change the n_hidden to be n_in/10;, I can get deeper in terms of epoch. Meaning instead of fail at the first epoch, I would get segmentation fault on the 4th or 5th epoch.