Help! I don't use Edge to scrape web information

[jiangwest]

Hi, I want to use edge.js to scrape the web information, e.g., ACM DL webpage.

• First, I use c# to build a simple GUI interface,
• then I hope to utilize Node.js, esp. cheerio module, to scrape web information.
• Next, edge.js act as a channel to deliver information between Node.js and C#
• But I fail. Why?

return function (data, callback) {    

    var http = require('http');
    var cheerio = require("cheerio");
    var fs = require('fs');

    //data is the target web site URL
    var acmWeb=data;
    var pageData;

    http.get(acmWeb, function(res){
        res.setEncoding('utf8');
        res.on('data',function(chunk){
            pageData += chunk; //collect web info
        });

        res.on('end', function(){
            //to process data herre
            var $ = cheerio.load(pageData);

            $('meta').each(function(i, e) {  //filter
                //'\r\n' -- return and newline,
               //open a new file to save the result
                fs.appendFile('message.txt', $(e).attr('content') + '\r\n', function (err) {
                    if (err) throw err;  

                    console.log('It\'s saved!');
                });
            });
        });        
    });

   // it will return the processing data
    callback(null, pageData);
}

[tjanczuk]

I think you meant to call the callback function only after the HTTP request has completed?

[Davevb]

In my script I also use 'http' to call an url. But it's never called. Any thoughts?