search

koriavinash1 / DigitalHistoPath

MIT License
23 stars 9 forks source link

Memory error while running points_extractor.py #18

Open haranrk opened 3 years ago

haranrk commented 3 years ago

After converting the binary mask images into the pyramidal format, and start running the script of "points_extractor", I am facing a memory error when using deep copy. Before throwing this error, it only save the coordinates of the patches extracted from the first training svs file. Can you please advise me if the problem is with my machine or not? I am using 32GB RAM. Thank you in advance.

Originally posted by @heba9004 in https://github.com/koriavinash1/DigitalHistoPath/issues/16#issuecomment-753734411

heba9004 commented 3 years ago

Thank you for your continuous support. Below is the error message. The error raised in calling RandomUniformSample function from extract_tumor_patches_from_wsi function. It succeed in extracting 2000 normal patch coordinates (2000 lines in the generated text file) from the first svs image before the error. memory error

haranrk commented 3 years ago

I have pushed a hotfix, @heba9004 kindly let me know if that resolves the memory errors.

The error seemed to have originated when the script attempted to unnecessarily make a copy of the sample image.

heba9004 commented 3 years ago

Thank you for your fix. After it , it succeed in extracting the normal and tumor coordinates for the first svs image successfully and the normal patches of the second image with a total of 6000 lines in generated text file, then pycharm unexpectedly crashes. I am now inspecting the problem that caused this behavior, it gave another memory error that I am trying to understand but I posted since you asked so. Thank you again for your interest. memory error2

haranrk commented 3 years ago

Ah damn, this script has not been optimised for memory. I'll try to find places where memory can be optimised like I had done in the previous fix.

Please feel free to look through the script and find places where you can optimised memory. A pull request from you would be most welcome.