Make keepValidPages a bit smarter

lintool / warcbase

Warcbase is an open-source platform for managing analyzing web archives

http://warcbase.org/

161 stars 47 forks source link

Closed lintool closed 8 years ago

lintool commented 8 years ago

keepValidPages currently just does this:

rdd.discardDate(null).keepMimeTypes(Set("text/html"))

Can we make it a bit smarter? For example:

If the file extension is .html or .html, keep the page anyway (even if the MIME type is wrong).
Throw out robots.txt (for some reason they have MIME type text/html)

aliceranzhou commented 8 years ago

marking as done