FeLoe
commented
4 years ago I'm not even a reviewer here, so completely ignore my comment if you want :) Also I do not think this needs to be added now, maybe later. The paywall_na variable is not related to not having text but to having something in the HTML that only appears if there is a paywall (for welt.de it for example the paywall has "data-testid = "Offer-Card"). This way we can still see whether no text is related to video content or to a paywall.
damian0604
annekroon
The following scrapers are adjusted, working, and ready for production:
myinca.rssscrapers.diewelt(save=False), myinca.rssscrapers.aachenerzeitung(save=False), myinca.rssscrapers.rheinischepost(save=False), myinca.rssscrapers.stuttgarterzeitung(save=False), myinca.rssscrapers.dertagesspiegel(save=False), myinca.rssscrapers.diesueddeutsche(save=False)
these scrapers can be moved to 'master'
We did not include 'paywall_na = TRUE' as sometimes there are also videos or even short news updates without body text as well. This also differs per newspaper. Die Welt, for example, has quite a lot of video content compared to some of the other newspapers. For the Stuttgarter Zeitung, their RSS feed also features a city blog that the text-scraping doesn't work for either (it only features local news for the city, so it's not relevant for my purposes anyway. But still...). So the way I see it using "paywall_na" =TRUE would be misleading in too many cases to be of value here.