segment-boneyard / nightmare

A high-level browser automation library.
https://open.segment.com
19.54k stars 1.08k forks source link

Using nightmare in a loop and scraping multiples webpages at once #1585

Open druv5319 opened 4 years ago

druv5319 commented 4 years ago

Here is my code snippet:

const SGLink = nightmare
                       .goto('https://www.stadiumgoods.com/catalogsearch/result/?q='+ json.Products[i].styleId)
                       .wait('.item')
                       .evaluate(() => document.querySelector('.item').innerHTML)
                       .end()
                       .then(response => {
                           const $ = cheerio.load(response);
                           console.log($('a').attr('href'));
                           return $('a').attr('href');
                       }).catch(err => {
                           console.log(err);
                       }); 
                       console.log(SGLink); 
                       shoe.resellLinks.stadiumGoods = SGLink;

This snippet is within a for loop that searches this website for every styleID and grabs the link to the first product. However, this process itself takes about 30 seconds to just grab one link. Is there any way of making this faster?

The snippet works fine outside of the loop when it runs once but once insert it within the loop, it's not able to grab any links for some reason. Here is my error code:

{ Error: Nothing responds to "goto"
    at EventEmitter.<anonymous> (/Users/dhruv/Sneaks-API/node_modules/nightmare/lib/ipc.js:138:19)
    at emitMany (events.js:147:13)
    at EventEmitter.emit (events.js:224:7)
    at process.<anonymous> (/Users/dhruv/Sneaks-API/node_modules/nightmare/lib/ipc.js:49:10)
    at emitTwo (events.js:126:13)
    at process.emit (events.js:214:7)
    at emit (internal/child_process.js:772:12)
    at _combinedTickCallback (internal/process/next_tick.js:141:11)
    at process._tickCallback (internal/process/next_tick.js:180:9) code: -1 }