search

vifreefly / kimuraframework

Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
MIT License
1.01k stars 155 forks source link

How to download files in parallel? #4

Closed caecity43 closed 6 years ago

caecity43 commented 6 years ago

I use in_parallel method to request page, but download file is not support in pipelines.

Is there any good advice?

Thank you.

vifreefly commented 6 years ago

in_parallel is for web pages. If you want to download files in parallel use peach https://github.com/bruceadams/pmap + Down https://github.com/janko-m/down . 

array_of_files_urls.peach(10) do |url|
  Down.download(url, destination: "/path/to/destination_folder")
end
caecity43 commented 6 years ago

@vifreefly Thank you, You made my day. 😉

vifreefly commented 6 years ago

@caecity43 No problem :)