search

monsieurvideo / get-flash-videos

Download or play videos from various Flash-based video hosting sites, without having to use the Flash player.
Apache License 2.0
241 stars 68 forks source link

HLSDownloader.pm consumes too much memory #197

Closed pcwalden closed 8 years ago

pcwalden commented 8 years ago

I have been trying to download http://www.pbs.org/video/2365556519/ which is a 3.5 hour 720p show. The HLSDownloader monotonically grows in memory usage until it maxes out my computer's 2GB memory and 4GB swap and is killed by the kernel. The get_flash_video program reaches about 5.8 GB in size before death.

Is there anyway to prevent this situation? I note that the ts file is ~50% saved before death occurs. So there does not seem to be any reason to retain the download image in memory.

karjonas commented 8 years ago

I have also noticed this. The problem is the $browser object store all downloaded files internally. The proper way is to store into a temp file like suggested at http://search.cpan.org/dist/WWW-Mechanize/lib/WWW/Mechanize/Cookbook.pod#Fetch_a_page_into_a_file

pcwalden commented 8 years ago

The stumbling block to using the above example, is that you want the segment appended to the existing file, not to overwrite it.

karjonas commented 8 years ago

Yeah, I think we are bound to use a temp file and then append that to our final file, I have used http://perldoc.perl.org/File/Temp.html before. It is not super elegant but it works.

pcwalden commented 8 years ago

I tried just adding $browser->get($url, ":content_file" => ">>$filename_ts"); but the ">>" prefix did not process as the open() function would have.

So I have a patch working that saves the segment to a re-used temp file and then copies it to the end of the TS file.

pcwalden commented 8 years ago

Fixed in pull request #198