search

codelucas / newspaper

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
https://goo.gl/VX41yK
MIT License
14.09k stars 2.11k forks source link

Getting Older News Articles #973

Open PaulKMandal opened 1 year ago

PaulKMandal commented 1 year ago

Hello, seven years ago this was posted: https://github.com/codelucas/newspaper/issues/245

I have a problem that requires me to scrape a large corpus of titles from 2013-2019 from various news sources. Ideally I would like to scrape 10 articles per date through this date range. The issue that I have is that newspaper only pulls the latest results. Does anyone have any insights on how to achieve this? Thanks!

banagale commented 1 year ago

Paul, out of curiosity, can you share why you’re trying to use this package instead of scrapy?

johnbumgarner commented 1 year ago

Paul, can you share an example of what you are trying to do?

PaulKMandal commented 1 year ago

I apologize for the delay.

Paul, out of curiosity, can you share why you’re trying to use this package instead of scrapy?

Because NewsPaper has very robust functionality for scraping News Articles.

Paul, can you share an example of what you are trying to do?

Ideally, I want to be able to specify a date and scrape the news articles from a certain date. I wrote an implementation that pulls articles from Archive.org. My implementation is available here and it works as intended, but Archive.org can be slow and often times out.

johnbumgarner commented 1 year ago

So you want to search the wayback archives. I wrote an example on this in my overview document for NewsPaper. If you provide me some more details I will add another example on searching a resource (website) for random articles for a certain date(s).

FYI the wayback archives are always slow for scraping. There might be a way to gain some performance, but that would require testing.