search

NeuromatchAcademy / course-content-dl

NMA deep learning course
https://deeplearning.neuromatch.io/
Creative Commons Attribution 4.0 International
731 stars 265 forks source link

FIX: Fixing memory limit exceeding in em_synapses.ipynb #955

Closed MobinNesari81 closed 6 days ago

MobinNesari81 commented 1 week ago

Hello. I hope you are doing well. In this commit, I've solved the Google Colab Memory Limit Exceeding error by dividing the incoming package into smaller data chunks. I've tested both in Colab and my computer, and all things work very well. Please let me know if there are any other issues or ambiguities in my code or explanation. I've also didn't add my name at the top of the file for notebook tester because I don't know if I should or not. Sincerely, M. Nesari

GodPascal commented 1 week ago

How are you saving the notebook? What environment are you using? Are you clearing the cells and kernel before saving? I cannot view the diff on GH

I encountered the memory problem as well (using Colab).

By checking https://github.com/NeuromatchAcademy/course-content-dl/pull/955.patch you can see the diff (in an uglier way). I think the most important change Mobin has made is freeing the memory allocated to the downloaded file by changing r = requests.get(url, allow_redirects=True, stream=True) to with requests.get(url, stream=True) as r:.

iamzoltan commented 1 week ago

That sounds like a reasonable change. Please run the notebooks locally and make that small change. Then push your results. Dont forget to clear all cells and restart your kernel before saving. Otherwise we are including a lot of extra information that is unnecessary and makes it hard to review your changes.

MobinNesari81 commented 1 week ago

Thank you, @GodPascal, for your explanation. As he said, I declared a stream for downloading data to prevent a memory limit from exceeding the error. I just talked with Marius, and he said he would upload dataset split files on OSF. When he sends me download links, I will update this file run it as you said and then resend it for you, @iamzoltan. Is that OK?

iamzoltan commented 1 week ago

Sounds good to me.

MobinNesari81 commented 1 week ago

Dear @iamzoltan, I updated the notebook and push it here. I've also tested it and added new links. The notebook works perfectly. Please let me know if there is any issue to solve.

MobinNesari81 commented 6 days ago

Oh yes my bad. I am removing them right now