search

arquivo / pwa-technologies

Arquivo.pt main goal is the preservation and access of web contents that are no longer available online. During the developing of the PWA IR (information retrieval) system we faced limitations in searching speed, quality of results, scalability and usability. To cope with this, we modified the archive-access project (http://archive-access.sourceforge.net/) to support our web archive IR requirements. Nutchwax, Nutch and Wayback’s code were adapted to meet the requirements. Several optimizations were added, such as simplifications in the way document versions are searched and several bottlenecks were resolved. The PWA search engine is a public service at http://archive.pt and a research platform for web archiving. As it predecessor Nutch, it runs over Hadoop clusters for distributed computing following the map-reduce paradigm. Its major features include fast full-text search, URL search, phrase search, faceted search (date, format, site), and sorting by relevance and date. The PWA search engine is highly scalable and its architecture is flexible enough to enable the deployment of different configurations to respond to the different needs. Currently, it serves an archive collection searchable by full-text with 180 million documents ranging between 1996 and 2010.
http://www.arquivo.pt
GNU General Public License v3.0
39 stars 7 forks source link

Version history on the Text Search API should allow sort from oldest to newest #916

Open amourao opened 4 years ago

amourao commented 4 years ago

What is the URL that originated the issue? https://arquivo.pt/textsearch?versionHistory=uc.pt

What happened? Results are sorted from newest to oldest

What should have happened? User should be able to sort from oldest to newest

danielbicho commented 4 years ago

Currently this textsearch?versionHistory doesn't make much sense, and should be removed.

  1. CDX API already does that, and offers more query options.
  2. Doesn't do anything related with the textsearch, it uses the cdx indexes. This doens't look like a textsearch search does it? Example

I think we should:

  1. offer an endpoint to search for URLs like /urlsearch (is basically a wrapper around cdx to have a query/response uniform with our others APIS)
  2. offer a /metadata endpoint where we give all available metadata about a resource <timestamp,url> (aggregate available information from the img indexes, text indexes and url indexes)
  3. all resources from the search apis can have a link rel to this metadata point

This will allow each API endpoint to have their own search domain, decoupled from the others, instead of having almost everything stuffed in the /textsearch API

amourao commented 3 years ago

Recheck when preparing new SolrCloud based index.