Parser for jp.dk

[ndarville]

• [ ] Ensure all articles (and their sections) are captured

  • [ ] Title
  • [x] Date
  • [x] Byline
  • [ ] Body

  $ python parsers/test_parser.py jp.JPParser http://jyllands-posten.dk/politik/ECE6804079/hoer-loekkes-tale-fra-folkemoedet-her/

• [x] Working URL schemes

  $ python parsers/test_parser.py jp.JPParser

• [ ] Working DOM scraper

[ndarville]

from baseparser import BaseParser
from BeautifulSoup import BeautifulSoup

class JPParser(BaseParser):
    feeder_pat =   '^http://jyllands-posten.dk/(indland|politik|international)/.'
    feeder_pages = ['http://jyllands-posten.dk/seneste/']

    def _parse(self, html):
        """Retrieve and serve the required fields to create an entry."""
        soup = BeautifulSoup(html,
            convertEntities=BeautifulSoup.HTML_ENTITIES,
            fromEncoding='utf-8')

        self.meta = soup.findAll('meta')
        self.title = soup.find('h1').getText()
        self.date = soup.find('div', 'date').getText().lstrip().capitalize()
        self.byline = soup.find('a', 'artByline').getText().title()
        summary = soup.find('div', 'artDescription').getText()
        content = soup.find('div', 'content-block').getText()
        self.body =  '\n'.join([summary, content])  # Remove div.relArt