Route for article scrape - Githubissues

alltrust / B-Article-Scraper

Scrape multiple online articles all at once. By inputting the articles you want scraped into the input field, B-Article-Scraper will save you time from manually navigating to each article and finding what you need.

1 stars 0 forks source link

Route for article scrape #4

Open alltrust opened 2 years ago

alltrust commented 2 years ago

Summary

Create route for user to be able to scrape the articles contents

Expected Behavior

User can input a single or a list of article URLs which returns the articles contents.

Acceptance Criteria

use cheerio and axios to select the article content from any given article URL.
have an article interface model with properties of title, article, article date, and summary,
create an all articles model containing properties of the URLs, Date scraped, all scraped content, reference to the User, all summarized articles (with the shape of the article interface)

alltrust commented 2 years ago

Additional Criteria

use a cloudscraper package to be able to bypass cloudflare for website
organize scraped data into an array of content that is more aesthetic and easy-to-read to client.

alltrust commented 1 year ago

acceptance criteria has been met.
using cloudscaper to bypass most websites with cloudflare.
update: Raw Articles has an Article subdoc and is stored :