search

marco-c / autowebcompat

Automatically detect web compatibility issues
Mozilla Public License 2.0
34 stars 41 forks source link

Fix for out of memory error issue #280 #281

Closed sdv4 closed 5 years ago

sdv4 commented 5 years ago

The problem was fixed by removing the creation of an array containing a vector for every image file in utils.py/get_ImageDataGenerator. The code was there in preparation for a call to fit() for the ImageGenerator object. Upon inspecting the documentation for this method, it turns out that this call was not necessary for the image processing we are doing.

I tested this on Colab for a few epochs on vgg16 using imagenet weights.

marco-c commented 5 years ago

I wonder why this wasn't a problem in @sagarvijaygupta case (he was using Colab too).

Upon inspecting the documentation for this method, it turns out that this call was not necessary for the image processing we are doing.

It isn't necessary for the image processing we are doing right now, but could be needed in the future if we try to do some. Anyway, we can bring it back if needed.

sdv4 commented 5 years ago

I am going to open a new pull request that with a clean commit history and with the changes @marco-c specified above.

sdv4 commented 5 years ago

I wonder why this wasn't a problem in @sagarvijaygupta case (he was using Colab too).

Was he using Colab when the dataset was much smaller, before all of that additional labelling was done back in Oct 2018?

marco-c commented 5 years ago

Was he using Colab when the dataset was much smaller, before all of that additional labelling was done back in Oct 2018?

Yes, I think so.

I am going to open a new pull request that with a clean commit history and with the changes @marco-c specified above.

It's better not to close a PR and open a new one in cases like this (well, basically in all cases :P). Otherwise you lose the history of discussion.

sdv4 commented 5 years ago

It's better not to close a PR and open a new one in cases like this (well, basically in all cases :P). Otherwise you lose the history of discussion.

Oh ok, that makes sense. Thanks for letting me know; I will make sure to follow this procedure in the future!