search

codelucas / newspaper

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:
https://goo.gl/VX41yK
MIT License
14.11k stars 2.11k forks source link

Getting older articles #245

Open tehnar opened 8 years ago

tehnar commented 8 years ago

Hello, is there any way to get more articles from a particular website? I get only the latest (amount and beginning date depends on site) news, not all of them. Caching is disabled, but it doesn't help.

yprez commented 8 years ago

@tehnar can you provide a specific example?

tehnar commented 8 years ago

@yprez For example, http://blog.jetbrains.com/ruby/ Newspaper thinks that there are only 94 articles while the real amount is much larger. The latest article downloaded by newspaper is dated autumn 2015.

Also, there is an additional trouble with this website: if I try some other blog (for example, http://blog.jetbrains.com/pycharm/), I'll get 0 articles. I've managed to fix it by manually deleting contents of ~/.newspaper/feed_category_cache, but it's a strange hack.

yprez commented 8 years ago

Did you try disabling cache? e.g. newspaper.build('http://blog.jetbrains.com/ruby/', memoize_articles=False)

I'm getting 127 articles from http://blog.jetbrains.com/ruby/, not sure if it's all of them or not. I think you're getting 0 on the 2nd result because the articles are cached by the domain...

tehnar commented 8 years ago

@yprez I'm still getting only 95 articles (I disabled caching and removed ~/.newspaper_scraper). There are 32 pages (the last page http://blog.jetbrains.com/ruby/page/32/), 10 articles on each, 320 articles in total, the first article is published in 2010, while I'm getting only articles published in 2015-2016

tehnar commented 8 years ago

Also, publish dates are not extracted properly for all the articles. For example, publish date for this article http://blog.jetbrains.com/ruby/2016/05/rubymine-2016-1-1-security-update/ is not recognized while for this article http://blog.jetbrains.com/ruby/2015/12/20-years-of-ruby/ it's extracted properly. It looks like a bug.

yprez commented 8 years ago

Funny, but the bug with parsing the date is actually in the 2nd article where it succeeds...

It parses the date from the URL (I couldn't find any meta date attributes in these articles), so 2015/12/20 part is parsed as a date, and the result is wrong too (20/05/2015 instead of the 22nd)

tehnar commented 8 years ago

@yprez Oh, really funny bug :) How about adding some more regexps for different date formats? I suppose that not so many sites represent date as year/month/day. Actually, I think that most of websites put the year as the last part of publish date (but I might be wrong). And what about a low amount of downloaded articles? Is that a king of bug, or what?

yprez commented 8 years ago

There was a ticket about trying to find dates with regex within the article - #168, and a closed pull-request somewhere. Can't really get it from the URL if it only has the year and month.

Regarding the amount of articles, newspaper combines several strategies to get a list of articles, e.g. links from the page, categories, rss feeds, etc. It doesn't go over the pages of paginated results... The first page http://blog.jetbrains.com/ruby/ only has articles from January 2016, same about the rss feed at: http://blog.jetbrains.com/ruby/feed/

To get all the articles you would need to paginate over all result pages, I don't think that's currently possible with newspaper...

tehnar commented 8 years ago

Ok, I've got it. Thank for the clarification.