search

alan-turing-institute / misinformation-crawler

Web crawler to collect snapshots of articles to web archive
MIT License
5 stars 2 forks source link

Crash when interpreting article from breitbart #325

Closed jemrobinson closed 5 years ago

jemrobinson commented 5 years ago 
2019-07-17 15:24:12     INFO: Searching for an article at: https://www.breitbart.com/politics/2019/06/11/donald-trump-in-iowa-immigration-the-defining-issue-of-2020/
Traceback (most recent call last):
  File "populate_article_db.py", line 55, in <module>
  File "populate_article_db.py", line 51, in main

  File "/Users/jrobinson/Projects/misinformation/misinformation-crawler/misinformation/warc/warc_parser.py", line 131, in process_webpages
    article = extract_article(response, config, entry, self.content_digests, self.node_indexes)
  File "/Users/jrobinson/Projects/misinformation/misinformation-crawler/misinformation/extractors/extract_article.py", line 41, in extract_article
    default_readability_article = simple_json_from_html_string(page_html, content_digests, node_indexes, use_readability=False)
  File "/Users/jrobinson/Projects/misinformation/misinformation-crawler/ReadabiliPy/readabilipy/simple_json.py", line 34, in simple_json_from_html_string
    "content": str(simple_tree_from_html_string(html))
  File "/Users/jrobinson/Projects/misinformation/misinformation-crawler/ReadabiliPy/readabilipy/simple_tree.py", line 42, in simple_tree_from_html_string
    insert_paragraph_breaks(soup)
  File "/Users/jrobinson/Projects/misinformation/misinformation-crawler/ReadabiliPy/readabilipy/simplifiers/html.py", line 199, in insert_paragraph_breaks
    parent_element.string.replace_with(text_fragments[0])
AttributeError: 'NoneType' object has no attribute 'replace_with'
jemrobinson commented 5 years ago

Fixed in ReadabiliPy (https://github.com/alan-turing-institute/ReadabiliPy/pull/78)