I have tried to download 1187 image url (stored in log text file) and download them in my local machine , most of the file are downloaded but i get TimeOut Error for some of them. (the total image file might be around 866 file, because some of the url are duplicate. i would appreciate if someone with modifying my script source code show me , how can i find a fix or workaround for the timeout issue and get download all the files fine. i uploaded the stored text file and the script here, so you can test it.
if you want to test my situation , you need to download above attached file and put it in the same folder of script file and run the script.
here is the source code :
var fs = require('fs'),
request = require('request');
var path = require('path');
var lineReader = require('readline').createInterface({
input: require('fs').createReadStream('log.txt')
});
lineReader.on('line', function (line) {
console.log('Line from file:', line);
var img_fileName = path.basename(line);
var file = fs.createWriteStream(img_fileName,{ flags: 'w'});
var options = {
url: line,
port: 80,
setTimeout: 0,
setNoDelay: true,
setSocketKeepAlive: true
};
request(options, function(err,response,body) {
if (err)
console.log(err);
}).pipe(file);;
});
i get this error in console log on some of the files but most of them download fine and those file that get timeout when download them , store as a empty file.
I have tried to download 1187 image url (stored in log text file) and download them in my local machine , most of the file are downloaded but i get TimeOut Error for some of them. (the total image file might be around 866 file, because some of the url are duplicate. i would appreciate if someone with modifying my script source code show me , how can i find a fix or workaround for the timeout issue and get download all the files fine. i uploaded the stored text file and the script here, so you can test it.
click here to download stored url log.txt
if you want to test my situation , you need to download above attached file and put it in the same folder of script file and run the script.
here is the source code :
i get this error in console log on some of the files but most of them download fine and those file that get timeout when download them , store as a empty file.
P.S :
npm -v : 3.5.3 ######### node -v : 5.3.0