Data4Democracy / internal-displacement

Studying news events and internal displacement.
43 stars 27 forks source link

Deal with datetime issue #143

Open georgerichardson opened 7 years ago

georgerichardson commented 7 years ago

Sometimes no publication date is available and a blank string is returned. However the db model expects a date time.

Possible fix in scraper.Scraper.html_article:

if not isinstance(a.publish_date, datetime.datetime):
    article_pub_date = datetime.datetime.now()
else:
    article_pub_date = a.publish_date
WanderingStar commented 7 years ago

Does it make more sense to use None/NULL? It seems misleading to say that the publication date of an article is when the scraper ran.

domingohui commented 7 years ago

I think None makes more sense. If there had to be a date, a fallback could be the latest date mentioned in the article ( or the latest Report).

simonb83 commented 7 years ago

Yeah agreed None makes more sense now that I think about it.

georgerichardson commented 7 years ago

Yep!