search

hatchways / team-cherry

1 stars 1 forks source link

Add Twitter scraper #27

Closed powerseed closed 4 years ago

powerseed commented 4 years ago

Please issue npm run scrapeTwitter to test, which will print the output onto the console.

  1. Used library twitter and the Twitter Premium search APIs to scrape Twitter, which will return tweets with the wanted company name posted within the previous 30 days (the uplimit for free use).
  2. Since tweets don't have title, we select a part of content to be the title, which is from 4 words ahead of the company name to 4 words after it.
  3. Since tweets don't have thumbnail picture, this field of each Twitter mention is set to be the Twitter icon;
  4. Because of the limit imposed by Twitter, at most 100 tweets can be obtained in each scraping.
  5. Used "favorite count" of each tweet as the value of its popularity attribute.
  6. Retweets are ignored.