nelsonic / github-scraper

🕷 🕸 crawl GitHub web pages for insights we can't GET from the API ... 💡
425 stars 96 forks source link

Scraping of issues list is broken? #107

Open gianlucascoccia opened 4 years ago

gianlucascoccia commented 4 years ago

I'm trying to use the scraper to get all the issues in a bunch of repositories. However, the scraper appears to behave differently from what it is described in the documentation. I have copied the example code in a file:

var gs  = require('github-scraper');
var url = '/dwyl/tudo/issues';
gs(url, function (err, data) {
  console.log(data); // use the data how ever you like
});

and executed it. My output appears to return repository stats:

- - - GitHub Scraper >> /dwyl/tudo/issues/ >> repo  - - -  
{
  url: '/dwyl/tudo/issues/',
  type: 'repo',
  description: '',
  website: '',
  tags: '',
  watchers: 27,
  stars: 52,
  forks: 8,
  commits: NaN,
  branches: NaN,
  releases: 34,
  langs: []
}

instead of issues list as expected (output given in the example):

{ entries:
   [
     {
       url: '/dwyl/tudo/issues/46',
       title: 'discuss components',
       created: '2015-07-21T15:34:22Z',
       author: 'benjaminlees',
       comments: 3,
       assignee: 'izaakrogan',
       milestone: 'I don\'t know what I\'m doing',
       labels: [ 'enhancement', 'help wanted', 'question' ]
     },
     {
       url: '/dwyl/tudo/issues/45',
       title: 'Create riot components from HTML structure files',
       created: '2015-07-21T15:24:58Z',
       author: 'msmichellegar',
       comments: 2,
       assignee: 'msmichellegar',
       labels: [ 'question' ]
     }
  ], // truncated for brevity
  open: 30,
  closed: 20,
  next: '/dwyl/tudo/issues?page=2&q=is%3Aissue+is%3Aopen',
  url: '/dwyl/tudo/issues'
}