search

evmer / perlego-downloader

Download books from Perlego.com in PDF format
MIT License
106 stars 52 forks source link

How to solve 'Memory error'? #42

Open hunching51 opened 1 year ago

hunching51 commented 1 year ago

Hi all! It seems that my computer does not have enough RAM to process.

I could only download around 600 pages of pdf and then my cmd would shows a memory error.

Is it possible to indicate which page to start? So that it would be possible for me to download it in two parts.

I am using a Ryzen 5 3600 and 16GB ram.

evmer commented 1 year ago

Hey, thanks for reporting this to me. Consider that the script is old & inefficient and it needs a refactoring.

Memory errors are caused by the fact that every page is stored in the RAM twice (during the render in the virtual browser first, and then during the PDFs merging) and that's a pretty useless process. We can partially solve the problem reusing the same puppeteer browser page instead to open a new one every time (see https://github.com/evmer/perlego-downloader/blob/main/downloader.py#L194) or simply closing it when the render/save is done.

I'll work on it in the next few days! :)