How to download files from auto generated URLs?

[korobochkin]

I need to download some files from site. Links to files I create automaticly in Gruntfile.js. My links looks like: http://site.com/images/D83CDF81.png

Where D83CDF81 generated after some calculation in downloadEmoji method. My Gruntfile.js:

module.exports = function (grunt) {
    var link = '';
    /* Link to file. Example of link:
     * http://site.com/images/D83CDF81.png
     */

    var name = ''; // Required name of file
    /* Name of file. Example:
     * D83CDF81.png
     */

    var downloadEmoji = function () {
        /* Generating links anf file name here
         * and set this values to 'link' and 'name' vars
         */

        /* I have multiple files (over 500)
         * I use forEach loop for generate links to all files
         * and run grunt task here.
         * How to pass my 'link' and 'name' vars to grunt-curl?
         */
        grunt.task.run (
            'curl'
        );
    };

    grunt.initConfig (
        pkg: grunt.file.readJSON ('package.json'),

        curl: {
            'emoji/<%= code %>.png' : '<%= link %>'
            // ???
            // 'images/D83CDF81.png' : 'http://site.com/images/D83CDF81.png'
        }
    );
    grunt.loadNpmTasks ('grunt-curl');
    grunt.registerTask ('downloademoji', 'Download emoji', downloadEmoji);
}

[twolfson]

initConfig is one way to set up grunt but you can also alter the config later on via grunt.config.set/get. An example would be:

var downloadEmoji = function () {
  // Generate names and such
  var emojiFilenameMap = {
    // dest: url of 500 emojis
  };

  // Set up `curl` config
  grunt.config.set('curl', emojiFilenameMap);

  // Run our curl task
  grunt.task.run('curl');
};

grunt.initConfig({
  pkg: grunt.file.readJSON ('package.json')
  // no `curl` setting here
});

http://gruntjs.com/api/grunt.config#accessing-config-data

Alternatively, we can use an array instead of an object via the curl-dir task:

// Define array of URLs to download from
var emojiUrls = [
  'http://site.com/abcdef.png',
  'http://site.com/bbcdef.png'
];

// Set up `curl-dir` to download all images to same directory
grunt.config.set('curl-dir', {
  src: emojiUrls,
  dest: 'emoji/'
});

// Run our `curl-dir` task
grunt.task.run('curl-dir');

https://github.com/twolfson/grunt-curl#curl-dir

[korobochkin]

Thank you! But some of files not available on server (404 returned). I set force to true before run curl:

grunt.config.set ('curl', emojiFilenameMap);
grunt.option ('force', true);
grunt.task.run ('curl');

Maybe curl have other ways to disable failure on 404 error? How to catch this event and skip going in my emojiFilenameMap and download next files?

[twolfson]

grunt is a build tool and usually leveraged from the command line. As a result, the ways of dealing with errors are very limited; ignore them or die. If you are looking for handling failed downloads, a custom script will probably be much more helpful.

// Load in dependencies
var fs = require('fs');
var request = require('request');

// Open a stream to the URL
var req = request('http://site.com/abcdef.png');
req.on('error', function handleError (err) {
  // TODO: Handle error (e.g. ECONNREFUSED)
});
req.on('response', function handleResponse (res) {
  // If the status code is not a 200, handle it
  if (res.statusCode !== 200) {
    // TODO: Handle non-200 status code
  // Otherwise, pipe the contents into a file
  } else {
    var handle = fs.createWriteStream('emojis/abcdef.png');
    res.pipe(handle);
  }
});

http://nodejs.org/api/fs.html https://github.com/mikeal/request