search

ukwa / shine

Prototype SOLR-powered web archive exploration UI.
https://github.com/ukwa/shine/wiki
Apache License 2.0
42 stars 7 forks source link

Removal of "phantom" pre-1996 resources #44

Open peterwebster opened 9 years ago

peterwebster commented 9 years ago

The fact that there is the long tail of eccentrically-dated results wasn't obvious in AADDA, but is made very obvious in the Shine ngram. We should consider excluding pre-1996 resources from Solr perhaps ?

peterwebster commented 9 years ago

See http://192.168.45.220/shine/graph with no query specified

anjackson commented 9 years ago

For now, I think Shine should exclude data prior to 1995 (facets and n-gram). Note that I say 1995 because the UKWA Open collection contains BL content from 1995. We'll follow up on the data quality problem internally.

peterwebster commented 9 years ago

Trac ticket http://tickets.wa.bl.uk/trac/ticket/2422 to look into data quality.

peterwebster commented 9 years ago

The trends graph now shows only 1996, and I can't find any search results earlier than that. So, @anjackson , I suppose we close this, unless you want to roll it back to include 1995 at this stage, in this version ?