Web Scraper Question

[nrc0004]

We are building a web scraper and it works the first time you deploy it, but then the node server crashes and it cant be used again. Below is our code and the error we get:

const express = require("express")
const cheerio = require("cheerio")
const request = require("request")
const fs  = require('fs')

const app = express()

app.get('/', function (req, res){

  url = 'https://ucr.fbi.gov/crime-in-the-u.s/2015/crime-in-the-u.s.-2015/tables/table-1'

  request(url, function(error, response, html){
    if(!error){
      var $ = cheerio.load(html)
      var population
      var json = { population: ""}

      $('#cell51').filter(function(){
        var data = $(this)
        population = data.html()

        json.population = population;
        res.send(population)
      })
    }
  })
})

app.listen('4000')
console.log('Scraping faces on 4k')

exports = module.exports = app

---

Error:

[nodemon] starting node index.js Scraping faces on 4k _http_outgoing.js:367 throw new Error('Can\'t set headers after they are sent.'); ^

Error: Can't set headers after they are sent. at ServerResponse.setHeader (_http_outgoing.js:367:11) at ServerResponse.header (/Users/natashaclark/wdi/Lab/sweet-libs/node_modules/express/lib/response.js:725:10) at ServerResponse.send (/Users/natashaclark/wdi/Lab/sweet-libs/node_modules/express/lib/response.js:170:12) at Object. (/Users/natashaclark/wdi/Lab/sweet-libs/index.js:24:13) at testFn (/Users/natashaclark/wdi/Lab/sweet-libs/node_modules/cheerio/lib/api/traversing.js:320:22) at /Users/natashaclark/wdi/Lab/sweet-libs/node_modules/lodash.filter/index.js:869:9 at /Users/natashaclark/wdi/Lab/sweet-libs/node_modules/lodash.filter/index.js:1255:11

[amaseda]

Hm, is the .filter callback executing multiple times? If so, running res.send( ) might be what's causing that error.

[nrc0004]

How would I check if it were firing multiple times? When I try to comment out res.send, it still works in the browser but the server still crashes.

On Mar 23, 2017, at 3:26 PM, Adrian Maseda notifications@github.com wrote:

Hm, is the .filter callback executing multiple times? If so, running res.send( ) might be what's causing that error.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ga-wdi-exercises/sweet-libs/issues/88#issuecomment-288833848, or mute the thread https://github.com/notifications/unsubscribe-auth/AXqfma3cDaNTePFRDrlzkMAmx4u5LVzmks5rosdigaJpZM4Mm_KL.

[nrc0004]

Hey Adrian,

Isaac got it, apparently we needed to switch res.send to res.write.

On Mar 23, 2017, at 3:26 PM, Adrian Maseda notifications@github.com wrote:

Hm, is the .filter callback executing multiple times? If so, running res.send( ) might be what's causing that error.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ga-wdi-exercises/sweet-libs/issues/88#issuecomment-288833848, or mute the thread https://github.com/notifications/unsubscribe-auth/AXqfma3cDaNTePFRDrlzkMAmx4u5LVzmks5rosdigaJpZM4Mm_KL.

[amaseda]

Oh word, awesome!